Data Engineer - AI Labs

We are looking for a hands-on Data Engineer with strong experience in big data engineering and real-time streaming applications. Hands-on expertise in Apache Flink is mandatory for this role. The ideal candidate will have built and operated production-grade Flink applications, along with Spark Streaming pipelines, and solid fundamentals in distributed data systems and cloud-native data engineering on AWS. DE will work closely with data platform, analytics, and ML teams to design scalable, low-latency data pipelines that power real-time decision-making.

Responsibilities:

• Design, develop, and own real-time streaming applications using Apache Flink (mandatory, core deliverable).
• Tune and troubleshoot Flink and Spark jobs in production (checkpointing, state management, backpressure, resource optimisation).
• Build and maintain Spark Structured Streaming pipelines for real-time data processing.
• Develop and optimise batch and streaming ETL pipelines using PySpark and Python.
• Write and optimise SQL for data transformation, validation, and analytics consumption.
• Build and deploy data pipelines on AWS, using EMR, S3 Lambda, and Glue.
• Apply strong DSA fundamentals to deliver performant, scalable, production- grade code.
• Ensure data pipeline reliability through monitoring, schema validation, and data quality checks.
• Deliver pipelines that meet defined SLAs for latency, throughput, and data accuracy.

Requirements:

• Minimum Number of years - 3 to 5 years.
• Can manage within a complex ecosystem of technology, vendors and other stakeholders.
• Proficiency in Python and Java/Scala is a plus.
• Experience in working on agile projects that span multiple organisations and business units.
• Proficient in Spark/Pyspark, Apache Flink, Spark Streaming, Python.
• Good at SQL; dashboarding experience (Superset/Tableau/Power BI); graph database Neo4j.
• Good to have: Agentic AI.