PySpark, Spark Developer

Key Responsibilities:

• ob Title PySpark Spark Developer
• Experience 5 8Years
• Design develop and maintain scalable data processing solutions using Apache Spark and PySpark
• Build and optimize ETL ELT pipelines for large scale data processing
• Develop Spark applications for batch and real time data processing
• Analyze transform and load structured and unstructured data from multiple sources
• Tune Spark jobs for performance scalability and reliability
• Work with distributed computing frameworks and big data technologies
• Collaborate with Data Engineers Data Architects and Business Analysts to understand requirements
• Troubleshoot production issues and provide efficient solutions
• Implement data quality checks and monitoring mechanisms
• Follow coding standards version control and CI CD best practices

Technical Requirements:

• Required Skills
• Strong experience in PySpark Apache Spark and Python
• Good understanding of Spark Core Spark SQL DataFrames and RDDs
• Experience with Hive HDFS SQL and data warehousing concepts
• Knowledge of Azure Databricks AWS EMR or Hadoop Ecosystem
• Experience with performance tuning and query optimization
• Hands on experience with Git and Agile methodologies
• Strong analytical and problem solving skills

Preferred Skills:

Technology->Big Data - Data Processing->PySpark,Technology->Big Data - Data Processing->Spark