Data Engineer

Role Overview

We are looking for a highly skilled and hands-on Data Engineer with 4+ years of experience in building scalable, production-grade data pipelines. The ideal candidate must have strong expertise in Python, SQL, DBT, and orchestration frameworks, along with experience in distributed processing and cloud platforms.

Key Responsibilities

• Design, build, and maintain scalable ETL/ELT pipelines.
• Develop production-grade Python code (modular, reusable, with proper logging & exception handling).
• Write optimised and complex SQL queries (joins, window functions, performance tuning).
• Implement Change Data Capture (CDC) and incremental/idempotent pipeline design.
• Develop transformation logic using DBT (data modelling, incremental models, testing).
• Build and manage workflows using Apache Airflow or equivalent orchestration tool (mandatory).
• Work with distributed data processing frameworks such as Apache Spark / Databricks.
• Perform data modelling (fact/dimension tables, star & snowflake schemas).
• Monitor pipeline performance and handle production issues.

Mandatory Technical Skills

• Strong Python (production-level development)
• Advanced SQL (complex joins, window functions, optimisation)
• Orchestration Apache Airflow or any enterprise scheduler (mandatory)
• Apache Spark / Databricks
• ETL / ELT frameworks
• Change Data Capture (CDC)
• Idempotent and incremental pipeline design
• Experience with at least one Cloud platform (AWS / Azure / GCP)
• Object storage (S3 / ADLS)
• Git version control