Pyspark

Job DescriptionJoin a fast-paced, collaborative team where data powers smarter decisions and better customer experiences. In this role, youll work hands-on with large-scale datasets to build reliable, high-performing data processing solutions using PySpark and Spark. Youll partner closely with engineers, analysts, and stakeholders to understand business needs, translate them into scalable pipelines, and continuously improve data quality and performance. If you enjoy solving complex data challenges, optimizing distributed workloads, and taking ownership from design to delivery, this is a great opportunity to grow your impact. Youll be encouraged to share ideas, learn from peers, and contribute to a culture that values clarity, craftsmanship, and continuous improvement.
Roles ResponsibilitiesKey Responsibilities

Design, develop, and maintain scalable batch data pipelines using PySpark and Apache Spark for large datasets.

Perform data ingestion, transformation, and enrichment while ensuring accuracy, completeness, and consistency of outputs.

Optimize Spark jobs for performance (partitioning, caching, joins, shuffles) and improve runtime efficiency and resource utilization.

Implement robust error handling, logging, and monitoring to ensure reliable pipeline execution and faster issue resolution.

Collaborate with cross-functional teams to gather requirements, define data contracts, and deliver well-documented solutions.

Conduct code reviews, follow engineering best practices, and contribute to reusable components and standards.

Troubleshoot production issues, perform root-cause analysis, and drive corrective and preventive actions.
Technical Requirement

Primary skills:Technology->Big Data - Data Processing->PySpark
Additional ResponsibilityMinimum Qualifications:

Bachelors degree (or equivalent) in Engineering/Computer Science/IT or related field.

35 years of experience in data engineering or big data development roles.

Strong hands-on experience with PySpark and Apache Spark for building data processing workflows.

Solid understanding of distributed data processing concepts and performance tuning fundamentals.

Ability to translate business requirements into technical implementations and deliver within timelines.
Preferred Qualifications:

Experience building end-to-end Spark applications including job orchestration, dependency management, and production support readiness.

Strong data transformation skills with a focus on data quality checks, reconciliation, and pipeline reliability.

Exposure to designing modular, reusable Spark components and maintaining clean, maintainable codebases.

Familiarity with structured and semi-structured data formats and efficient processing patterns in Spark.

Proven ability to collaborate effectively across teams, communicate clearly, and contribute to continuous improvement initiatives.
Good to have skills:
Spark SQL, Delta Lake, Databricks, Airflow, Hadoop
Educational RequirementMCA,MSc,MTech,Bachelor of Engineering,BTech
Preferred SkillsTechnology->Big Data - Data Processing->PySpark
Service LineData Analytics Unit