Big Data Lead

Job Title: Big Data Lead

Location: Hyderabad (Offshore)

Experience: 8-10 Years

Shift Timings: Night Shift (5:00 PM 2:00 AM IST) / US EST Primetime

Role Overview:

We are looking for a Big Data Lead with 8-10 years of experience to design, scale, and manage our high-volume data pipelines and real-time streaming platforms. In this role, you will serve as the primary technical point of contact for US clients, lead engineering teams, and ensure data systems run reliably during US overlap hours.

Key Responsibilities

• Leadership & Strategy: Guide architecture standards and data governance practices. Mentor developers, train tech leads, and build a strong engineering culture.
• Pipeline Development: Build and optimize data processing applications using Hadoop, Spark, Kafka, and HBase. Manage batch/streaming performance, low-latency querying, and partition strategies.
• Client Engagement: Act as the technical anchor for client accounts. Translate technical performance and architecture plans into clear updates for stakeholders.
• Incident Management: Lead troubleshooting and root-cause analysis (RCA) during system outages or performance bottlenecks. Set up monitoring and automated recovery routines.

Required Technical & Professional Skills

• Experience: 8 to 10 years in software engineering and big data development, including experience leading client-facing delivery teams.

• Big Data Stack: Strong hands-on experience with Hadoop (HDFS, MapReduce, YARN), Apache Spark (Core, SQL, Streaming), Apache Kafka, and HBase.

• Architecture: Deep understanding of distributed systems, real-time message streaming, NoSQL schema design (HBase compaction/region tuning), and data lake design.

• Performance Tuning: Proven capability in tuning Spark jobs, memory allocation, execution plans, and Kafka partition throughput.

• Programming: Proficiency in Scala, Java, or Python, along with strong SQL skills.

• CI/CD & Orchestration: Hands-on experience with Git, CI/CD pipelines, and workflow managers like Airflow or Oozie.

• Communication: Clear verbal and written communication with the flexibility to work night shifts and overlap with US hours.

Preferred Certifications

• Databricks Certified Apache Spark Developer / Data Engineer Professional

• AWS Certified Data Engineer or GCP Professional Data Engineer