Senior Data Engineer

Roles and Responsibilities :

• Design, develop, test, and maintain large-scale data pipelines using ETL processes to extract, transform, and load data from various sources into MySQL databases.
• Collaborate with cross-functional teams to identify business requirements and design solutions that meet those needs through API integration with PySpark.
• Develop complex SQL queries to optimize database performance and troubleshoot issues related to data ingestion.
• Implement scheduling tools such as Airflow or similar technologies to automate the execution of data pipelines.

Job Requirements :

• 6-9 years of experience in a Data Engineering role with expertise in building scalable data pipelines using ETL processes.
• Strong proficiency in Python programming language with experience working with APIs, particularly those built on top of APIs like Flask or Django.
• Experience working with relational databases such as MySQL and ability to write efficient SQL queries for querying large datasets.