Hadoop / PySpark

Job DescriptionJoin a data-driven team where your work helps turn large-scale information into clear, actionable insights. In this role, youll collaborate with engineers, analysts, and stakeholders to build and optimize reliable big-data solutions that power reporting, analytics, and downstream applications. Youll get hands-on exposure to modern distributed processing, contribute to production-grade pipelines, and learn best practices for performance, quality, and governance. If you enjoy solving complex data challenges, working in an environment that values curiosity and teamwork, and delivering measurable impact through scalable engineering, this opportunity will help you grow your technical depth while making a real difference across projects and business outcomes.
Roles Responsibilities

Design, develop, and support scalable data processing workflows using Hadoop and PySpark for batch and large-volume processing.

Build and maintain data pipelines that ingest, transform, and validate data from multiple sources into curated datasets.

Optimize Spark jobs for performance (partitioning, caching, shuffle tuning) and improve overall pipeline efficiency and reliability.

Perform data quality checks, reconciliation, and root-cause analysis for pipeline failures or data anomalies.

Collaborate with cross-functional teams to understand requirements, translate them into technical solutions, and deliver within timelines.

Create clear technical documentation for workflows, data mappings, and operational runbooks.

Participate in code reviews, follow engineering best practices, and contribute to continuous improvement of standards and tooling.
Technical Requirementechnology->Big Data - Data Processing->PySpark,Technology->Big Data - Hadoop->Hadoop Administration->Hadoop
Additional Responsibility

25 years of experience in big data engineering or data processing roles.

Bachelors/Masters degree in Engineering, Computer Science, or equivalent (BTech/BE/MCA/MSc/MTech or related).

Hands-on experience with Hadoop ecosystem concepts (HDFS, distributed processing fundamentals).

Practical experience developing data transformations using PySpark.

Strong problem-solving skills with the ability to debug data and job execution issues in distributed environments.


Preferred Qualifications:

Experience building end-to-end ETL/ELT pipelines and managing dependencies across multiple data workflows.

Working knowledge of Spark optimization techniques and handling skewed/large datasets efficiently.

Familiarity with data modeling concepts and designing curated datasets for analytics and reporting use cases.

Exposure to production support practices such as monitoring, incident triage, and improving pipeline resiliency.

Strong communication skills to collaborate effectively with stakeholders and explain technical trade-offs clearly.


Good to have skills:


Hive, HBase, Kafka, Airflow, Scala
Educational RequirementMCA,MSc,MTech,Bachelor of Engineering,BTech
Preferred SkillsTechnology->Big Data - Hadoop->Hadoop Administration->Hadoop,Technology->Big Data - Data Processing->PySpark
Service LineData Analytics Unit