Azure Databricks Migration Architect

Work Experience: 10+ Years (Including 4+ years of designing and implementing Azure Databricks solutions)

• Experience in data engineering, data architecture, cloud architecture, or enterprise analytics.
• Hands-on experience of designing and implementing Azure Databricks solutions; proven experience leading enterprise data-platform migration.
• Strong proficiency in Azure Databricks, Azure Data Factory, Apache Spark, PySpark, SQL, and Delta Lake.
• Experience with Azure Data Lake Storage Gen2, Azure Synapse Analytics, Azure SQL, and related Azure services.
• Strong understanding of Lakehouse architecture, Medallion architecture, analytical data modeling, and data integration.
• Experience with Spark performance tuning, partitioning, caching, data-skew management, cluster sizing, and Delta Lake optimization.
• Experience with Unity Catalog or comparable enterprise data-governance solutions.
• Familiarity with Git, CI/CD, automated testing, Azure DevOps or GitHub Actions, and infrastructure as code using Terraform or Bicep.
• Ability to translate business requirements into technical architecture and clearly explain technology options, risks, costs, and trade-offs.
• Strong communication, stakeholder-management, problem-solving, estimation, and technical-leadership skills.
• Experience migrating from Azure Synapse, SQL Server, Oracle, Teradata, Hadoop, Informatica, or SSIS to Azure Databricks.
• Knowledge of Databricks Workflows, Auto Loader, Databricks SQL, Lakeflow Declarative Pipelines, MLflow, Event Hubs, or Kafka.
• Exposure to Power BI, AI/ML platforms, MLOps, FinOps, AWS, or GCP.
• Databricks Data Engineer Professional, Azure Solutions Architect Expert, or Azure Data Engineer certification.

Qualifications: Bachelor s or master s degree in Computer Science, Information Technology, or a related discipline.

• Design secure, scalable, highly available, and cost-effective Azure Databricks Lakehouse solutions.
• Assess existing data warehouses, data lakes, ETL platforms, data sources, and reporting workloads.
• Define migration strategies, target-state architecture, implementation roadmaps, effort estimates, cutover plans, and rollback approaches.
• Lead end-to-end migrations from legacy or cloud data platforms to Azure Databricks.
• Build scalable batch and streaming ETL/ELT pipelines using Azure Data Factory, Apache Spark, PySpark, SQL, and Delta Lake.
• Integrate data from databases, APIs, applications, files, cloud storage, and streaming platforms.
• Design data warehouses and data lakes for analytics, reporting, AI, and machine-learning workloads.
• Implement Medallion architecture, data modeling, data-quality checks, reconciliation, lineage, and metadata management.
• Optimize Spark jobs, Delta tables, clusters, SQL workloads, storage, and cloud costs.
• Design Azure networking and security using VNets, private endpoints, managed identities, service principals, Key Vault, RBAC, and encryption.
• Implement governance and fine-grained access controls using Databricks Unity Catalog.
• Establish CI/CD, automated testing, version control, infrastructure-as-code, monitoring, and operational best practices.