Data Engineer

Roles and Responsibilities :

• Design, develop, test, and deploy data pipelines using Airflow to extract insights from large datasets.
• Collaborate with cross-functional teams to identify business requirements and design solutions that meet those needs.
• Develop Python scripts using PySpark to process and analyze big data sets.
• Troubleshoot issues with existing data pipelines and implement improvements for better performance.
• Ensure data quality and integrity by implementing data validation and testing procedures.

Job Requirements :

• Strong understanding of Airflow workflow management system.
• Proficiency in Python programming language with experience working with PySpark for big data processing.
• Experience with designing and developing scalable data architectures.
• Strong problem-solving skills and attention to detail.
• Ability to work collaboratively in a team environment.
• Familiarity with data analysis and visualization tools is an added advantage.