Lead Data Engineer

We are looking for a Lead Data Engineer who can own the design and development of scalable data platforms and pipelines while continuing to remain hands-on technically. The ideal candidate should have experience working in a product/technology company, building data engineering solutions at scale, and taking ownership of architecture, technical decisions, production reliability, and mentoring engineers.

Responsibilities:

• Own the architecture and end-to-end design of scalable batch and real-time data pipelines.
• Design and develop high-performance data platforms using Python, SQL, and Spark/PySpark.
• Build and optimize ETL/ELT pipelines, data ingestion frameworks, and transformation workflows.
• Work with Kafka / Spark Streaming / other streaming technologies for near-real-time data processing.
• Design data platforms using cloud technologies such as AWS / GCP / Azure.
• Work with technologies such as Airflow, Databricks, Snowflake, BigQuery, Delta Lake/Iceberg, or equivalent.
• Define data models, lakehouse/data warehouse architecture, partitioning, performance, and cost-optimization strategies.
• Own production data pipelines, including monitoring, data quality, observability, troubleshooting, and reliability.
• Participate in architecture/design reviews and make technical decisions for scalability and maintainability.
• Mentor and guide 3-8 engineers, conduct code/design reviews, and establish engineering best practices.
• Work closely with product, engineering, analytics, and other business stakeholders to translate requirements into scalable technical solutions.
• Drive migration, modernization, and optimization of existing data platforms where required.

Requirements:

• Experience: 7.5-10 years.
• Industry preference: Product/SaaS/Technology companies.

Must-Have Skills:

• Core: Python + SQL + Spark/PySpark.
• Data Engineering: ETL/ELT, Data Pipelines, Data Modelling, Data Warehousing/Lakehouse.
• Distributed Processing: Spark/PySpark + preferably Kafka/Streaming.
• Cloud: Strong experience in at least one of AWS, GCP, or Azure.
• Orchestration: Airflow / Cloud Composer / ADF / equivalent.
• Architecture: Data platform architecture, scalability, performance optimization, data quality.
• Leadership: Technical leadership, mentoring, architecture/design reviews, and stakeholder management.