Azure Data Engineer

Role Summary

We are seeking a highly skilled Azure Data Engineer with a minimum of 5 years of experience in designing, developing, and maintaining modern cloud-based data platforms. The ideal candidate should have strong expertise in Azure Data Factory, Azure Databricks, Spark, SQL, Python, Data Warehousing, and DevOps practices. The role requires hands-on experience in building scalable data pipelines, implementing Lakehouse architectures, and delivering enterprise-grade data solutions.

Key Responsibilities

• Design and develop scalable data ingestion, transformation, and orchestration solutions on Azure.
• Build and maintain batch and streaming data pipelines using Azure Databricks and Azure Data Factory.
• Implement Lakehouse architecture using Delta Lake and Azure Data Lake Storage Gen2.
• Develop high-performance data transformation logic using Spark DataFrame APIs and Spark SQL.
• Optimise data processing workloads through caching, persistence, partitioning, and performance tuning techniques.
• Collaborate with architects, analysts, and business stakeholders to deliver data solutions.
• Implement CI/CD and source control best practices using Git and branching strategies.
• Support production deployments, troubleshooting, and operational excellence.

Mandatory Technical Skills

• Azure Data Factory (ADF): Orchestration, ingestion from on-premises and cloud sources, triggers, linked services, and Integration Runtime (IR).
• Azure Databricks: Data ingestion from ADLS Gen2, Delta Lake, CRUD operations, Time Travel, Optimisation, Merge operations, Streaming, and Auto Loader.
• Apache Spark: Data transformation using DataFrame APIs, Spark SQL, caching, persistence, and performance optimisation.
• Data Warehousing: Strong understanding of Slowly Changing Dimensions (SCD) and data loading strategies.
• SQL: Advanced SQL development and coding expertise. (Window Function/CTE)
• Python: Data handling using Pandas, Object-Oriented Programming concepts, Classes, and Objects.
• DevOps: Git usage, source code management, and branching strategy awareness.