Azure Databricks Architect / Lead

We are looking for an experienced Azure Databricks Lead / Architect to drive the design, architecture, migration, implementation, and optimization of enterprise-scale data platforms on Azure.

Key Responsibilities

• Design and architect scalable Azure Databricks solutions for enterprise data platforms.
  • Lead Databricks migration and implementation projects from on-premise or legacy platforms to Azure.
  • Define Lakehouse architecture, data pipelines, ingestion, transformation, and processing frameworks.
  • Provide technical leadership for Databricks clusters, performance tuning, scalability, and cost optimization.
  • Work closely with Azure services such as ADLS Gen2, Azure Data Factory, Azure Synapse, Azure SQL, Key Vault, and Azure DevOps.
  • Design and implement secure, reliable, and highly available data solutions.
  • Establish best practices around Delta Lake, Unity Catalog, CI/CD, governance, and data security.
  • Lead technical discussions with stakeholders and guide development teams.
  • Troubleshoot complex performance, integration, and data engineering challenges.

Must-Have Skills

Strong hands-on experience with Azure Databricks
Expertise in Databricks architecture, migration, and implementation
Strong knowledge of PySpark / Spark / Python / SQL
Experience with Delta Lake and Unity Catalog
Strong understanding of ADLS Gen2 and Azure Data Factory
Experience in performance tuning and cluster optimization
Experience designing enterprise-scale data platforms
Strong understanding of Azure cloud architecture and security
Experience leading data engineering / architecture teams

Good to Have

Databricks certification
Experience with Azure DevOps and CI/CD
Experience with Terraform / Infrastructure as Code
Knowledge of data governance and security frameworks
Experience handling large-scale data migration projects