Azure Data Engineer

Role & responsibilities

• 5-10 years of data engineering experience, including 3+ years of hands-on Databricks experience.
• Hands-on experience with Azure Databricks, including notebooks, workflows, jobs, and pipeline development.
• Strong proficiency in PySpark, Spark SQL, Python, and SQL.
• Experience with Delta Lake, Delta Live Tables (DLT), Auto Loader, Lakeflow, and Structured Streaming.
• Strong understanding of Medallion Lakehouse architecture, ETL/ELT pipeline engineering, data ingestion, data quality, monitoring, logging, and performance optimization.
• Experience with Unity Catalog, RBAC, data governance, lineage, and access control.
• Ability to optimize Spark workloads using partitioning, caching, cluster sizing, and query tuning.
• Good understanding of security, identity and access management, compliance, and governance.
• Experience with Azure Data Factory (ADF), Integration Runtime, ADLS Gen2, Airflow, Azure SQL, Key Vault, Logic Apps, Event Grid, Azure VMs, Bicep, VNet, and Private Endpoints.
• Experience with Azure DevOps, Git, CI/CD, and environment management.
• Good communication and collaboration skills.
• Good-to-have exposure to GenAI workloads on Databricks.