Python,pyspark,GCP

Desired Competencies (Technical/Behavioral Competency)

Must-Have**

(Ideally should not be more than 6 - 8)

• Strong proficiency in Python programming.
• Hands-on experience with PySpark and Apache Spark.
• Knowledge of Big Data technologies (Hadoop, Hive, Kafka, etc.).
• Experience with SQL and relational/non-relational databases.
• Familiarity with distributed computing and parallel processing.
• Understanding data engineering best practices.
• Experience with REST APIs, JSON/XML, and data serialization.
• Exposure to cloud computing environments.

Good-to-Have

• 5+ years of experience in Python and PySpark development.
• Experience with data warehousing and data lakes.
• Knowledge of machine learning libraries (e.g., MLlib) is a plus.
• Strong problem-solving and debugging skills.
• Excellent communication and collaboration abilities.

SN

Responsibility of / Expectations from the Role

1

• Develop and maintain scalable data pipelines using Python and PySpark.
• Design and implement ETL (Extract, Transform, Load) processes.
• Optimize and troubleshoot existing PySpark applications for performance.
• Collaborate with cross-functional teams to understand data requirements.
• Write clean, efficient, and well-documented code.
• Conduct code reviews and participate in design discussions.
• Ensure data integrity and quality across the data lifecycle.
• Integrate with cloud platforms like AWS, Azure, or GCP.
• Implement data storage solutions and manage large-scale datasets.