Data Engineer (Azure)

We are seeking a highly skilled Azure Data Engineer with experience in designing, developing, and maintaining modern cloud-based data platforms. The ideal candidate should have strong expertise in Azure Data Factory, Azure Databricks, Spark, SQL, Python, data warehousing, and DevOps practices. The role requires hands-on experience in building scalable data pipelines, implementing Lakehouse architecture, and delivering enterprise-grade data solutions.

Responsibilities:

• Design and develop scalable data ingestion, transformation, and orchestration solutions on Azure.
• Build and maintain batch and streaming data pipelines using Azure Databricks and Azure Data Factory.
• Implement Lakehouse architecture using Delta Lake and Azure Data Lake Storage Gen2
• Develop high-performance data transformation logic using Spark DataFrame APIs and Spark SQL.
• Optimize data processing workloads through caching, persistence, partitioning, and performance tuning techniques.
• Collaborate with architects, analysts, and business stakeholders to deliver data solutions.
• Implement CI/CD and source control best practices using Git and branching strategies.
• Support production deployments, troubleshooting, and operational excellence.

Requirements:

• Strong problem-solving and analytical skills.
• Ability to perform coding assessments in Spark SQL, SQL, and Python.
• Experience working in Agile delivery environments.
• Excellent communication and stakeholder management skills.
• Ability to mentor junior engineers and drive technical excellence.

Mandatory Technical Skills:

• Azure Data Factory (ADF): Orchestration, ingestion from on-premises and cloud sources, triggers, linked services, and Integration Runtime (IR).
• Azure Databricks: Data ingestion from ADLS Gen2 Delta Lake, CRUD operations, time travel, optimization, merge operations, streaming, and Auto Loader.
• Apache Spark: Data transformation using DataFrame APIs, Spark SQL, caching, persistence, and performance optimization.
• Data Warehousing: Strong understanding of Slowly Changing Dimensions (SCD) and data loading strategies.
• SQL: Advanced SQL development and coding expertise. (Window Function/CTE)
• Python: Data handling using Pandas, object-oriented programming concepts, classes, and objects.
• DevOps: Git usage, source code management, and branching strategy awareness.

Good to Have Skills:

• ADF Data Flows.
• Dimensional Modelling (Facts and Dimensions).
• Real-time analytics and event-driven architecture.
• Exposure to Azure Synapse Analytics and related Azure data services.