Big Data Lead

• Platform Lakehouse Architecture: Drive the architecture and migration from legacy Hadoop/Hive systems to modern Apache Iceberg formats for ACID compliance, schema evolution, and time-travel querying.
• Query Compute Optimization: Build and tune high-performance batch and streaming pipelines using Apache Spark. Deploy Trino for sub-second, federated interactive SQL querying across heterogeneous storage layers.
• Orchestration DevOps: Manage enterprise-wide DAG workflows using Apache Airflow. Deploy, auto-scale, and manage compute workloads on Kubernetes (K8s) clusters.
• GenAI Integrations: Architect ingestion and processing pipelines for unstructured data, handling vector embeddings and Retrieval-Augmented Generation (RAG) infrastructure.
• Team Leadership Governance: Lead engineering teams through code reviews, technical design sessions, platform security enforcement, cost optimization, and SLA management.

Category Requirements Big Data Core Apache Spark (PySpark/Scala), Apache Hive, Hadoop/HDFS Lakehouse Querying Apache Iceberg, Trino (Presto), Complex SQL Orchestration Infra Apache Airflow, Kubernetes (K8s), Docker GenAI Ecosystem Vector databases (Milvus/Pinecone/Qdrant), RAG ingestion pipelines, LLM data processing Programming Python, Scala, Java

Disclaimer: This job posting and Location has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.