PySpark, Spark Developer

Roles Responsibilities
ob Title: PySpark / Spark Developer

Experience: 5-8Years
Design, develop, and maintain scalable data processing solutions using Apache Spark and PySpark.
Build and optimize ETL/ELT pipelines for large-scale data processing.
Develop Spark applications for batch and real-time data processing.
Analyze, transform, and load structured and unstructured data from multiple sources.
Tune Spark jobs for performance, scalability, and reliability.
Work with distributed computing frameworks and big data technologies.
Collaborate with Data Engineers, Data Architects, and Business Analysts to understand requirements.
Troubleshoot production issues and provide efficient solutions.
Implement data quality checks and monitoring mechanisms.
Follow coding standards, version control, and CI/CD best practices.


Technical Requirement
Required Skills
Strong experience in PySpark, Apache Spark, and Python.
Good understanding of Spark Core, Spark SQL, DataFrames, and RDDs.
Experience with Hive, HDFS, SQL, and data warehousing concepts.
Knowledge of Azure Databricks, AWS EMR, or Hadoop Ecosystem.
Experience with performance tuning and query optimization.
Hands-on experience with Git and Agile methodologies.
Strong analytical and problem-solving skills.


Educational Requirement
MCA,MSc,MTech,BCA,BSc,BTech


Preferred Skills
Technology->Big Data - Data Processing->PySpark,Technology->Big Data - Data Processing->Spark


Service Line
Data Analytics Unit