Pyspark Developer

Roles and Responsibilities

• Design, develop, test, deploy, and maintain scalable data pipelines using PySpark on AWS EMR.
  
• Collaborate with cross-functional teams to gather requirements and deliver high-quality solutions.
  
• Troubleshoot complex issues related to Kafka messaging systems and Apache Airflow workflows.
  
• Develop automation scripts using Ansible or equivalent tools for deployment and management of big data applications.
  
• Ensure compliance with security standards by implementing access controls and encryption mechanisms.
  

Desired Candidate Profile

• 7-12 years of experience in software development with expertise in Python programming language.
  
• Strong understanding of Big Data technologies such as Hadoop, Spark, Kafka, etc. .
  
• Experience working with cloud-based platforms like AWS EMR or similar services.
  
• Proficiency in scripting languages like Shell Scripting (Bash) or Python for automating tasks.