Big Data Lead

Job Title: Big Data Lead

Experience Level: 8+ Years

Location: Pune, India

Job Summary

We are looking for a Lead Big Data Lead to design, scale, and modernize our high-throughput data platform. You will direct technical strategy, manage data pipelines, and bridge traditional big data architectures with modern Lakehouse patterns and Generative AI applications.

Key Responsibilities

• Platform & Lakehouse Architecture: Drive the architecture and migration from legacy Hadoop/Hive systems to modern Apache Iceberg formats for ACID compliance, schema evolution, and time-travel querying.
• Query & Compute Optimization: Build and tune high-performance batch and streaming pipelines using Apache Spark. Deploy Trino for sub-second, federated interactive SQL querying across heterogeneous storage layers.
• Orchestration & DevOps: Manage enterprise-wide DAG workflows using Apache Airflow. Deploy, auto-scale, and manage compute workloads on Kubernetes (K8s) clusters.
• GenAI Integrations: Architect ingestion and processing pipelines for unstructured data, handling vector embeddings and Retrieval-Augmented Generation (RAG) infrastructure.
• Team Leadership & Governance: Lead engineering teams through code reviews, technical design sessions, platform security enforcement, cost optimization, and SLA management.

Technical Skills & Qualifications

Category Requirements

Big Data Core Apache Spark (PySpark/Scala), Apache Hive, Hadoop/HDFS

Lakehouse & Querying Apache Iceberg, Trino (Presto), Complex SQL

Orchestration & Infra Apache Airflow, Kubernetes (K8s), Docker

GenAI Ecosystem Vector databases (Milvus/Pinecone/Qdrant), RAG ingestion pipelines, LLM data processing

Programming Python, Scala, Java