Pyspark Developer

Job Summary

We are seeking a skilled and motivated PySpark Engineer to design, develop, and optimize large-scale data processing solutions using PySpark and distributed computing technologies.

The ideal candidate will have strong hands-on experience in PySpark, Python, SQL, and ETL, along with strong analytical, debugging, and problem-solving skills.

The candidate should also have a strong interest in learning and adapting to new technologies as required by business and project needs.

Pre-Screening Questionnaire Mandatory

Candidates are required to complete the PySpark Engineer Pre-Screening Questionnaire as part of the application process.

Questionnaire Link:
https://forms.cloud.microsoft/r/JuFHXi1Dgz

Please complete the questionnaire after applying for this position.

Roles & Responsibilities

• Design, develop, and maintain scalable data processing applications using PySpark.
• Build and optimize ETL/ELT pipelines for structured and unstructured data.
• Process large datasets efficiently using Apache Spark.
• Integrate data from relational databases, cloud storage, APIs, and streaming platforms.
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Collaborate with Data Engineers, Data Scientists, Business Analysts, and Architects to deliver business solutions.
• Develop reusable frameworks and follow coding standards for data engineering projects.
• Perform data validation and data quality checks.
• Troubleshoot production issues and resolve performance bottlenecks.
• Monitor production jobs and ensure smooth execution.
• Participate in code reviews and maintain technical documentation.
• Work closely with cross-functional teams throughout the software development lifecycle.

Required Skills

• 47 years of experience in Data Engineering.
• Minimum 3 years of hands-on experience in PySpark development.
• Strong Python programming skills.
• Hands-on experience with Apache Spark, Spark SQL, Spark DataFrames, and RDDs.
• Strong SQL and ETL development experience.
• Good understanding of distributed computing concepts.
• Experience with AWS and/or Azure cloud platforms.
• Familiarity with Git version control.
• Experience with workflow orchestration tools such as Airflow or Azure Data Factory.
• Good understanding of Data Warehousing concepts and Dimensional Modeling.
• Excellent analytical, debugging, and problem-solving skills.
• Willingness to learn and adapt to new technologies as required is mandatory.

Preferred Skills

• Databricks
• Delta Lake
• Lakehouse Architecture
• Docker
• Kubernetes
• CI/CD Pipelines
• DevOps practices for Data Engineering

Qualification

Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related discipline.

Location

Hyderabad / Pune / Coimbatore / Bengaluru

Notice Period

Immediate to 15 days preferred