Data Engineer

Job Summary

We are seeking a Data Engineer with hands-on expertise in Azure Databricks, Azure Data Factory and Azure Data Lake Storage. The role will build scalable, governed data pipelines and Lakehouse data products for BI, analytics, machine learning and AI-enabled self-service. It will also enable Databricks Genie through trusted datasets, business metrics and semantic definitions.

Key Responsibilities

• Data Pipelines & Integration: Design, develop and optimise batch and streaming pipelines to integrate structured, semi-structured and unstructured data into the Databricks Lakehouse.
• Lakehouse Data Products: Build reusable, business-ready datasets using Delta Lake and medallion architecture for dashboards, analytics, machine learning and AI use cases.
• Databricks Genie Enablement: Configure and improve Genie Agents using trusted datasets, business terminology, metrics, instructions, sample queries and verified answers.
• Semantic Models & Metrics: Translate business definitions and KPIs into consistent dimensions, measures and reusable metric views.
• Data Quality & Observability: Implement automated quality checks and monitor pipeline health, data freshness, schema changes and failures.
• Governance & Security: Use Unity Catalog for metadata, lineage, role-based access and governed data sharing, in line with data privacy and security policies.
• Performance & Cost: Optimise Spark workloads, Delta tables, clusters and queries for scalability, reliability and cost efficiency.
• DataOps & Support: Apply Git, testing, code reviews and CI/CD practices; monitor production solutions, resolve incidents and maintain technical documentation.
• Stakeholder Collaboration & Adoption: Partner with business teams, analysts, data scientists and platform teams to deliver scalable solutions and improve adoption of Genie and analytics products.

Required Skills & Qualifications

• Bachelor's degree in Computer Science, Information Technology, Engineering, MCA or a related discipline.
• Hands-on data engineering experience, preferably with Azure Databricks.
• Experience converting business requirements and KPI definitions into governed data models and production-grade data products.
• Strong analytical, problem-solving, documentation and stakeholder-management skills.
• Exposure to manufacturing, supply chain, sales, procurement, finance or agriculture-related datasets is desirable.
• Technical Skills.
  • Azure Databricks, Delta Lake, Azure Data Factory, Azure Data Lake Storage and Lakeflow or equivalent orchestration.
  • Databricks Genie Agents, semantic modelling, metric views and Unity Catalog.
  • Advanced SQL, Python and PySpark, with Spark performance optimisation.
  • Data quality, observability, batch and streaming architectures; Kafka or Event Hubs exposure is preferred.
  • Git, automated testing, CI/CD and deployment management.
  • Power BI, Tableau or Databricks AI/BI dashboards; MLflow exposure is desirable.
  • Databricks Data Engineer certification is preferred.

Work Experience

• 2-4 years.

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.