Azure Data Engineer

Senior Azure Data Engineer  

Job Description

We are looking for a highly skilled Azure Data Engineer with strong hands-on experience in Azure Databricks, PySpark, Azure Data Factory (ADF), ADLS Gen2, Delta Live Tables (DLT), and Unity Catalog to build and maintain enterprise-scale data platforms and modern Lakehouse architectures.

The ideal candidate should have experience designing scalable ETL/ELT pipelines, implementing Medallion Architecture, optimizing Spark workloads, and driving data governance through Unity Catalog.

The ideal candidate should have experience designing scalable ETL/ELT pipelines, implementing Medallion Architecture, optimizing Spark workloads, and driving data governance through Unity Catalog.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines using Azure Data Factory and Azure Databricks.
• Build and implement Medallion Architecture (Bronze, Silver, Gold layers).
• Develop data transformation logic using PySpark and Spark SQL.
• Build and manage Delta Live Tables (DLT) pipelines.
• Implement data governance, security, and access controls using Unity Catalog.
• Design and manage data ingestion frameworks using ADLS Gen2 and Auto Loader.
• Optimize Databricks jobs and Spark workloads for performance and cost efficiency.
• Develop metadata-driven and reusable data pipelines.
• Implement monitoring, logging, and data quality checks.
• Work closely with business and technology teams to gather requirements and deliver data solutions.
• Support CI/CD deployment using Azure DevOps.
• Participate in production support, troubleshooting, and performance tuning activities.

Mandatory Skills

• Azure Databricks
• Azure Data Factory (ADF)
• PySpark
• SQL
• ADLS Gen2
• Delta Live Tables (DLT)
• Unity Catalog
• Delta Lake
• Medallion Architecture
• ETL / ELT Development

Good to Have

• Auto Loader
• Azure Synapse Analytics
• Microsoft Fabric
• Databricks Workflows
• Structured Streaming
• Azure Event Hub
• Kafka
• Azure DevOps CI/CD
• Data Quality Frameworks
• Azure Key Vault

Experience: 6.5 to 15 Years

Location: Bangalore