AI Data Engineer (Azure)

Key Skills: LLM, Azure Databricks, SQL, AI Engineering, Python, Azure Data lake, ETL, Azure ADF, GitHub, DBT, Kafka, Spark

Roles and Responsibilities:

• Design and build scalable ETL/ELT pipelines for batch and analytical workloads on Azure.
• Develop and optimize data processing workflows in Azure Databricks using Python, SQL, and Spark.
• Implement lakehouse data models using Delta Lake and medallion architecture patterns.
• Integrate governed data sources and manage access controls, lineage, and cataloging across the platform.
• Support AI data preparation workflows for embeddings, vector search, and RAG-oriented datasets.

Skills Required:

• 4+ years of relevant experience in data engineering, Azure data platforms, and production pipeline delivery.
• Strong hands-on experience with Azure Data Factory, Azure Databricks, Python, SQL, and Spark SQL/PySpark.
• Experience building production data platforms on ADLS Gen2 and working with Delta Lake and lakehouse patterns.
• Ability to handle large-scale data processing with columnar and row-based formats such as Parquet, Avro, and JSON.
• Experience with data governance, lineage, and access control tools such as Unity Catalog and Microsoft Purview.

Good to Have:

• Experience with Spark, Kafka, GitHub, dbt, Structured Streaming, Azure Event Hubs, Debezium, Great Expectations, Delta Live Tables, Terraform, Bicep, Apache Iceberg, Delta UniForm, Azure DevOps, GitHub Actions, Key Vault, and Managed Identity.

Education: Bachelor's degree in Computer Science, Information Technology, Engineering, or a related discipline.