Azure Data Engineer

Roles and Responsibilities :

• Design, develop, test, deploy and maintain large-scale data pipelines using Azure Data Factory (ADF) to extract, transform, load (ETL) data from various sources into Azure Databricks.
• Collaborate with cross-functional teams to understand business requirements and design scalable data warehousing solutions on Azure Synapse Analytics.
• Develop complex ETL processes using PySpark and Python scripts to migrate large volumes of structured and unstructured data between different systems.
• Ensure high availability, performance, security, and scalability of all deployed applications by monitoring logs, troubleshooting issues, and implementing fixes.

Job Requirements :

• 6-8 years of experience in Data Engineering with expertise in ADF, Azure Databricks, CI/CD pipeline management.
• Strong understanding of SQL programming language for querying relational databases like PostgreSQL or MySQL.
• Proficiency in developing complex ETL processes using PySpark and Python scripts.