Data Engineer

We are looking for an experienced Azure Data Engineer with strong hands-on expertise in Azure data services, Python, SQL, and Spark/PySpark to design, build, and maintain scalable, production-grade data pipelines and data platforms. The ideal candidate should have strong experience working with Azure Data Factory, Azure Data Lake Storage, Azure Databricks, SQL, and PySpark, along with a good understanding of modern cloud data architectures. You will work closely with Data Scientists, Data Analysts, and Engineering teams to build reliable data platforms and prepare high-quality datasets for analytics and downstream applications.

Responsibilities:

• Design, build, and maintain scalable ETL/ELT data pipelines using Azure Data Factory, Python, SQL, and Spark/PySpark.
• Develop and maintain data pipelines for ingesting data from multiple structured and unstructured sources.
• Build and manage scalable data solutions using Azure Data Lake Storage (ADLS).
• Develop and optimise data processing workloads using Azure Databricks and PySpark.
• Write complex and optimised SQL queries for data transformation and processing.
• Implement data integration workflows using Azure Data Factory (ADF).
• Work with cloud-based data lakes, data warehouses, and distributed data processing frameworks.
• Optimise data pipelines and Spark jobs for performance, reliability, scalability, and maintainability.
• Implement data quality checks, validation, testing, monitoring, and pipeline observability.
• Troubleshoot pipeline failures, data inconsistencies, and performance issues.
• Collaborate with Data Scientists, Data Analysts, and Engineering teams to prepare high-quality datasets.
• Implement security, governance, access control, and deployment best practices within Azure data environments.
• Participate in Agile development practices, code reviews, technical discussions, and production support.

Requirements:

• 5-10 years of experience in Data Engineering or a closely related role.
• Strong hands-on experience with Microsoft Azure data services.
• Strong hands-on experience with Azure Data Factory (ADF).
• Strong experience with Azure Data Lake Storage (ADLS/ADLS Gen2).
• Hands-on experience with Azure Databricks.
• Strong programming experience in Python.
• Strong hands-on experience with SQL.
• Hands-on experience with Apache Spark/PySpark for large-scale data processing.
• Strong experience building and maintaining production-grade ETL/ELT pipelines.
• Good understanding of data lakes, data warehouses, distributed data processing, and modern data architectures.
• Experience with data integration from multiple data sources.
• Understanding of data quality, testing, monitoring, security, and governance practices.
• Experience troubleshooting data pipeline failures and optimising data processing workloads.
• Strong problem-solving and communication skills.
• Ability to work effectively in an Agile engineering environment.