Lead Data Engineer

Role & responsibilities

JD:

Apache Spark strong hands-on expertise in large-scale distributed data processing; Scala – advanced programming proficiency for big data engineering

Big Data Ecosystem – strong understanding of distributed computing and data platform components; Data Lake Architecture – experience designing scalable modern data lake solutions

Apache Iceberg – working knowledge of open table formats for large-scale analytics; Parquet – experience with columnar storage formats

Hive Metastore – metadata management experience; ETL/ELT Pipeline Development – ability to build reliable and scalable data pipelines

Data Modelling & Data Warehousing – strong foundation in analytical data structures; Performance Optimisation – tuning data processing jobs and storage usage

Data Quality Frameworks – ensuring accuracy, reliability, and governance of data; CI/CD Practices – deployment automation and engineering best practices

Python & SQL Optimisation – good-to-have skills for scripting and query performance; Cloud Data Platforms – good-to-have experience with scalable cloud-based data platforms

Stakeholder Collaboration – working with Product Managers, business teams, Data Scientists, and Engineering teams Communication & Problem Solving – ability to translate business needs into data solutions