Azure Data Engineer

Azure Data Engineer

Role Overview

An Azure Data Engineer will design, build, and support enterprise-scale cloud data platforms. This role requires hands-on expertise in Azure, Databricks, Spark, Lakehouse architecture, and modern data pipelines, along with the ability to collaborate with engineers, business stakeholders, and DevOps teams to deliver secure, reliable, analytics-ready data solutions in an Agile environment.

Key Responsibilities

• Architect and deliver scalable data lakehouse solutions using Azure Data Lake Storage, Azure Databricks, Delta Lake, Unity Catalog, and Medallion architecture.
• Build curated data products, data marts, and analytics-ready datasets for analysts, data scientists, and enterprise reporting teams.
• Develop reliable batch and near-real-time pipelines using Databricks, Apache Spark, PySpark, SQL, Azure Data Factory, Synapse, and related Azure services.
• Implement ingestion frameworks for structured, semi-structured, and unstructured data while ensuring data quality, lineage, validation, observability, and reusable patterns.
• Apply data modeling, partitioning, schema evolution, performance tuning, cost optimization, error handling, and operational readiness practices.
• Implement CI/CD for data workloads using Azure DevOps or GitHub Actions, including automated testing, deployment, release governance, and secure engineering practices.
• Use Entra ID, RBAC, secrets management, Key Vault, Managed Identity, and governance controls to protect data platforms.
• Leverage GitHub Copilot, Microsoft Copilot, Databricks Assistant, and AI-powered workflows to accelerate SQL/PySpark development, ETL delivery, test data generation, documentation, troubleshooting, and productivity.

Candidate Qualifications

• Expert-level experience with Azure Databricks, Azure Synapse Analytics, Azure Data Factory, PySpark, Spark SQL, and Azure Data Lake Storage Gen2.
• Strong knowledge of Delta Lake, Lakehouse architecture, Unity Catalog, data governance, Medallion architecture, CDC, SCD, data quality frameworks, reconciliation, and lineage.
• Advanced SQL expertise across Oracle, SQL Server, PostgreSQL, and modern cloud data platforms.
• Hands-on experience with Informatica PowerCenter is preferred, especially for migration programs.
• Experience with Git, GitHub, Azure DevOps, CI/CD, automated deployment, secure coding, monitoring, alerting, and operational dashboards.
• Strong understanding of Azure security services including Key Vault, Managed Identity, Entra ID, and role-based access controls.

Tech Stack

• Data Engineering: Azure Databricks, Azure Data Factory, Azure Synapse, ADLS Gen2, Python, PySpark, SQL, Spark SQL, ETL/ELT, orchestration, metadata management, and data modeling.
• Lakehouse: Delta Lake, Bronze/Silver/Gold layers, schema evolution, SCD, reusable ingestion frameworks, and performance tuning.
• DevOps & Quality: Git, GitHub, Azure DevOps, CI/CD, automated testing, release pipelines, SonarQube, secure coding, Azure Monitor, Application Insights, logs, alerts, and pipeline health checks.