Azure Data Engineer

Senior Data Engineer Azure Databricks

Python | PySpark | Spark | Delta Lake | Unity Catalog | Azure

Job Overview

We are looking for an experienced Senior Data Engineer with strong hands-on expertise in Azure Databricks, PySpark, Apache Spark, Delta Lake and Azure data services. The role will focus on designing and developing scalable data engineering solutions, building batch and streaming pipelines, implementing Lakehouse architectures, and optimizing Spark workloads for performance, reliability and maintainability.

Key Responsibilities

• Design, develop and maintain scalable data pipelines using Azure Databricks, PySpark and Spark SQL. • Build and manage Lakehouse solutions using Bronze, Silver and Gold / Medallion architecture. • Develop robust batch and streaming ETL/ELT pipelines for structured and semi-structured data.
• Work extensively with Delta Lake, including MERGE/upserts, schema evolution, data quality and incremental processing.
• Implement and manage data governance using Unity Catalog, including catalogs, schemas, permissions and access controls.
• Develop and orchestrate Databricks notebooks, jobs and workflows for reliable data processing.
• Integrate Databricks with Azure services such as ADLS Gen2, Azure Data Factory, Synapse Analytics, Event Hubs and Service Bus.
• Optimize Spark workloads using appropriate partitioning, caching, broadcast joins, shuffle optimization and other performance-tuning techniques.
• Apply Spark fundamentals such as lazy evaluation, transformations/actions, joins, repartitioning and data-skew handling.
• Design data models and curated datasets for analytics, reporting and downstream applications. • Implement data quality checks, schema validation, monitoring, error handling and SLA-driven pipeline delivery. • Develop solutions using Python and SQL and follow clean, modular and maintainable coding practices. • Use Git, Azure DevOps and CI/CD practices for source control, deployment and release management.
• Collaborate with engineering, analytics and business teams to understand requirements and deliver reliable data products.

Required Technical Skills

Area

Technologies / Expertise

Programming

Python, SQL

Big Data

Apache Spark, PySpark, Spark SQL

Databricks

Azure Databricks, notebooks, jobs/workflows, performance optimization

Lakehouse

Delta Lake, Medallion architecture, incremental processing, MERGE, schema evolution

Governance

Unity Catalog, access control, data governance

Azure Data

ADLS Gen2, Azure Data Factory, Azure Synapse Analytics

Messaging / Integration

Azure Event Hubs, Azure Service Bus

DevOps

Git, Azure DevOps, CI/CD, Docker

Area

Technologies / Expertise

Databases

SQL Server, PostgreSQL, Oracle, Azure SQL

Spark & Databricks Expertise

• Strong understanding of Spark execution concepts, including lazy evaluation, transformations and actions. • Hands-on experience with joins, broadcast joins, shuffles, partitioning, repartitioning and caching. • Experience troubleshooting and optimizing slow or resource-intensive Spark jobs.
• Hands-on Delta Lake experience, including MERGE/upserts, schema evolution and transactional data processing. • Practical experience with Unity Catalog and governed access to data assets.
• Ability to implement reliable deduplication, incremental-load and latest-record processing patterns using PySpark. Good to Have
• Experience with Azure Functions, Azure Logic Apps, Azure Event Grid, Azure Monitor and Log Analytics. • Experience with Apache Kafka or other real-time streaming technologies.
• Experience with Power BI and REST API-based data integration.
• Exposure to AI/ML data pipelines, vector search or RAG-related data preparation.
• Financial services, banking, insurance or risk-management domain experience.
• Databricks Data Engineer Associate / Professional certification or equivalent practical expertise. Qualifications & Experience
• 8+ years of overall experience in software/data engineering.
• Strong hands-on experience in Azure Databricks and PySpark-based data engineering.
• Strong Python and SQL development skills.
• Demonstrated experience designing and supporting production-grade data pipelines.
• Strong analytical, troubleshooting and problem-solving skills.
• Good communication and ability to explain technical approaches clearly.