Azure Data Engineer

Role Overview

We are looking for a contract Data Engineer to support enterprise BI & Analytics deliveries within IDO. The role will design, build, and operate scalable data pipelines and curated datasets on Microsoft Azure, with strong hands-on delivery in Databricks (Spark/Delta). Exposure to AI engineering / GenAI enablement is a strong plus for future roadmap initiatives.

Key Responsibilities

• Build and maintain data pipelines on Azure (ingestion, transformation, validation, and publishing) using Databricks and Azure services.
• Develop curated, analytics-ready datasets for BI/reporting and downstream consumers.
• Good understanding of FACT and dimensional modelling.
• Implement incremental load patterns, backfills, and handling of late-arriving data to minimize full reloads.
• Ensure strong data quality, reconciliation checks, and pipeline monitoring/alerting.
• Optimize performance and cost (Spark tuning, partitioning strategies, file formats).
• Maintain documentation (source-to-target mappings, runbooks) and support releases/production issues in an Agile model.

Must-Have Skills (Azure Data Engineering + Databricks)

• 38 years of hands-on Data Engineering experience.
• Strong experience with Azure data ecosystem and Databricks:
• Spark (PySpark), Delta Lake (MERGE/upserts, schema evolution), Lakehouse patterns
• Strong SQL and Python (production-grade coding).
• Orchestration experience with Azure Data Factory and/or Databricks Workflows.
• Practical experience implementing incremental processing and robust operational pipelines (retry, idempotency, monitoring).