Azure Databricks Data Engineer

Role & responsibilities

• Design and develop scalable data pipelines and ETL/ELT processes using Azure Databricks.
• Develop data transformation and processing solutions using PySpark, Spark SQL, and Python.
• Build and maintain data pipelines using Azure Data Factory (ADF).
• Work with Azure storage and data services such as Azure Data Lake Storage (ADLS Gen2) and Azure SQL.
• Implement data ingestion from structured and unstructured data sources.
• Develop optimized Spark jobs and troubleshoot performance issues.
• Design data models and implement data transformation logic according to business requirements.
• Work with Delta Lake for reliable and scalable data storage and processing.
• Implement data quality, validation, error handling, and monitoring mechanisms.
• Manage Databricks workflows, jobs, clusters, notebooks, and associated configurations.
• Implement CI/CD practices for Databricks and data engineering solutions using Azure DevOps/Git.
• Collaborate with data architects, analysts, developers, and business stakeholders.
• Ensure solutions meet security, performance, scalability, and governance requirements.

Mandatory Skills

• Strong experience in Azure Databricks.
• Strong hands-on experience with PySpark and Python.
• Good knowledge of Apache Spark and Spark SQL.
• Experience with Azure Data Factory.
• Experience with Azure Data Lake Storage (ADLS Gen2).
• Strong understanding of Delta Lake and data lake architecture.
• Good understanding of ETL/ELT concepts and data warehousing.
• Experience with SQL and relational databases.
• Knowledge of Git and Azure DevOps.
• Strong troubleshooting and performance-tuning skills.

Good-to-Have Skills

• Experience with Azure Synapse Analytics.
• Knowledge of Azure Event Hubs or other streaming technologies.
• Experience with Databricks Unity Catalog and data governance.
• Knowledge of real-time/streaming data processing.
• Experience with CI/CD and infrastructure automation.
• Knowledge of dimensional data modeling and Medallion Architecture.
• Experience working in Agile/Scrum environments.

Preferred candidate profile