Lead Data Engineer

We are seeking a highly experienced and motivated Lead Data Engineer to join our dynamic data platform team in our Gurugram office to drive the design, development, and optimization of scalable data platforms and real-time data processing solutions. The ideal candidate will have extensive experience building modern data pipelines on AWS Cloud, working with streaming technologies, expert-level proficiency in PySpark and SQL, and leading the implementation of enterprise-grade data engineering solutions. As a lead data engineer, you will work closely with data architects, data scientists, analytics teams, and business stakeholders to deliver reliable, scalable, and high-performance data ecosystems that enable data-driven decision-making.

Responsibilities:

• Design, develop, and maintain scalable batch and real-time data pipelines.
• Build and optimize cloud-native data solutions on AWS leveraging modern data engineering best practices.
• Develop and manage streaming data platforms using technologies such as MSK, Firehose, Spark Streaming, or Flink.
• Leverage Amazon S3, Amazon Redshift, or Amazon Aurora for efficient storage and querying of structured data.
• Design and manage Apache Iceberg-based data platforms with support for schema evolution, time travel, partition optimization, and ACID transactions.
• Lead data ingestion, transformation, integration, and orchestration processes across multiple data sources.
• Implement idempotent processing, CDC merge (upsert) strategies, and data reconciliation mechanisms to ensure consistency across batch and streaming pipelines.
• Define and enforce data governance, data quality, and metadata management standards across the data platform.
• Collaborate with cross-functional teams to understand business requirements and translate them into scalable technical solutions.
• Apply performance tuning and optimization techniques to improve pipeline efficiency, scalability, and cost-effectiveness in AWS environments.
• Support CI/CD implementation, infrastructure automation, monitoring, and production deployments.
• Troubleshoot and resolve complex data engineering challenges in production environments.
• Mentor junior and mid-level data engineers and establish engineering standards and best practices.

Requirements:

• Bachelor's or master's degree in computer science, information technology, engineering, or a related field.
• 7-10 years of hands-on experience in data engineering.
• Strong understanding of data engineering concepts, including data modeling, ETL/ELT, data warehousing, and distributed computing.
• Extensive experience working with AWS services such as AWS Glue, S3, Redshift, Athena, Lambda, Step Functions, and Amazon EMR.
• Strong hands-on experience with AWS data services, such as building and managing real-time and streaming data pipelines.
• Hands-on expertise with streaming frameworks such as Apache Kafka, AWS MSK, AWS Kinesis, Apache Spark Streaming, and Apache Flink.
• Strong programming skills in Python, PySpark, SQL, and experience with data processing frameworks.
• Experience working with large-scale structured and unstructured datasets.
• Strong knowledge of data lake and lakehouse architectures.
• Good to know about building AI-ready data platforms and understanding multi-agent frameworks and agentic AI architectures.
• Familiarity with Git, CI/CD pipelines, and DevOps practices.