Lead Azure Data Engineer

Experience: 5 - 12+ Years
Location: Bangalore / Chennai / Hyderabad / Pune
Work Model: Hybrid
Hands-on Coding: 60 - 70%

Role OverviewWe are looking for a highly hands-on Data Engineer Lead with strong expertise in Python, PySpark, Apache Spark, Azure Databricks and Azure Data Engineering.Although this is a Lead-level position, the role is strongly hands-on, with approximately 6070% of the role focused on coding and technical implementation.The ideal candidate should combine architect-level design capabilities with strong hands-on Python and data engineering skills.Key

Responsibilities

• Design and develop scalable data ingestion, transformation and validation frameworks using Python, PySpark and Databricks.
• Develop reusable and metadata/configuration-driven data engineering frameworks.
• Design end-to-end solutions using Azure Databricks, ADLS, ADF, Azure SQL/Synapse and Delta Lake.
• Develop robust ETL/ELT pipelines supporting batch, streaming and incremental processing.
• Define architecture for data quality, validation, reconciliation, error handling, quarantine, replay and observability.
• Design solutions for CDC, idempotency, schema evolution, SCD and incremental processing.
• Develop reusable Python frameworks for ingestion, transformation and validation.
• Troubleshoot complex production issues and drive root-cause resolution.
• Optimize Spark/PySpark workloads involving partitioning, joins, shuffles, data skew and query execution.
• Establish Python coding standards, framework patterns, testing practices and CI/CD standards.
• Conduct technical design and code reviews.
• Provide technical leadership to data engineering teams.
• Translate business and functional requirements into scalable technical solutions.
• Collaborate with architects, product teams, business stakeholders and engineering teams.

Mandatory Technical Skills

• Python Advanced / Hands-on
• PySpark Strong
• Apache Spark Strong
• Azure Databricks
• Advanced SQL
• ETL/ELT
• Data ingestion and transformation framework development
• Azure Data Factory
• ADLS
• Azure SQL / Synapse
• Delta Lake
• Spark performance optimization
• Data quality and validation
• Production debugging and troubleshooting
• Git / CI-CD

Lead-Level ExpectationsThe candidate must be able to:

• Code independently and remain hands-on for 6070% of the role.
• Design reusable Python-based data engineering frameworks.
• Make architecture-level technical decisions.
• Explain technical trade-offs behind design decisions.
• Review code and identify performance/design issues.
• Troubleshoot complex production failures.
• Mentor engineers on Python, PySpark, Spark and Databricks.
• Own technical delivery from design through production implementation.

Critical Screening Areas
AreaExpected LevelPythonAdvancedPython Framework DevelopmentMandatoryPySparkStrongApache SparkStrongDatabricksStrong Hands-onSQLAdvancedAzureStrong Hands-onETL FrameworksStrongSpark OptimizationCriticalProduction DebuggingCriticalArchitectureStrongData ModellingStrongTechnical LeadershipStrongGood to Have

• CDC / Kafka
• Unity Catalog
• Data Lakehouse / Medallion Architecture
• Oracle
• Azure Synapse
• Data archival / retention / purge
• Insurance / BFSI domain experience
• Enterprise-scale data platform experience

Mandatory Candidate CriteriaThe candidate must have personally developed Python frameworks for data ingestion, transformation, validation or similar data engineering use cases.Candidates who have only:

• Managed a team,
• Performed production monitoring,
• Used existing frameworks,
• Worked mainly on SQL,
• Used Databricks only for notebook execution,without strong hands-on Python framework development experience should not be considered.

Ideal Pipeline Architecture ExposureCandidates should be able to explain architectures such as:Source ADF/Ingestion ADLS Databricks/PySpark Delta SQL/Synapse Downstream ConsumptionThey should also be able to explain how they handled incremental loads, CDC, data quality, failures, reconciliation and performance optimization.