Azure Data Engineer

Role & responsibilities

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure Databricks and Azure Data Factory.
• Develop robust ETL/ELT solutions for ingesting data from multiple structured and unstructured sources.
• Build data transformation and processing solutions using Databricks, PySpark, Spark SQL, and SQL.
• Implement batch and incremental data processing frameworks for large-volume datasets.
• Work with Delta Lake to build reliable and scalable lakehouse data solutions.
• Design and implement data pipelines following Bronze, Silver, and Gold/Medallion architecture principles.
• Create and manage ADF pipelines, datasets, linked services, triggers, parameters, and integration runtimes.
• Develop complex SQL queries, stored procedures, views, and data transformation logic.
• Perform data cleansing, validation, reconciliation, and quality checks across data pipelines.
• Optimize Databricks jobs, Spark workloads, SQL queries, and data processing for performance and cost efficiency.
• Integrate Azure Databricks and ADF with ADLS Gen2, Azure Synapse, and other Azure services.
• Develop workflow and application integrations using Azure Logic Apps, where required.
• Troubleshoot and resolve data pipeline failures, performance issues, and production incidents.
• Implement appropriate security, access control, monitoring, and governance practices for Azure data solutions.
• Work with DevOps teams to implement CI/CD pipelines and automated deployment processes.
• Collaborate with business analysts, data architects, developers, and other stakeholders to understand requirements and deliver end-to-end data solutions.
• Prepare and maintain technical documentation covering data flows, pipeline architecture, transformations, and operational processes.

Technical Requirements

Must Have

• Strong hands-on experience with Azure Databricks.
• Strong experience in Azure Data Factory.
• Advanced knowledge of SQL and relational databases.
• Experience with PySpark / Apache Spark for large-scale data processing.
• Good understanding of ETL/ELT concepts and data engineering best practices.
• Experience working with Azure Data Lake Storage Gen2.
• Experience developing and supporting production-grade data pipelines.
• Strong understanding of data transformation, data validation, and performance optimization.

Good to Have

• Experience with Azure Synapse Analytics.
• Hands-on experience with Delta Lake and Delta tables.
• Knowledge of Medallion/Lakehouse architecture.
• Experience with Azure Logic Apps.
• Knowledge of Unity Catalog and data governance.
• Experience with Azure DevOps, Git, and CI/CD.
• Knowledge of data warehousing concepts and dimensional data modeling.
• Experience with monitoring and troubleshooting Azure data workloads.

Education

• Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field.
• Relevant Azure/Databricks certifications will be an added advantage.

Behavioral & Soft Skills

• Strong analytical and problem-solving skills.
• Excellent communication and stakeholder-management skills.
• Ability to work independently as well as collaboratively in a team environment.
• Strong attention to detail and commitment to data quality.
• Ability to understand business requirements and translate them into scalable technical solutions.
• Willingness to learn and adapt to emerging Azure and data engineering technologies.

Preferred candidate profile