Pyspark Data Engineer

Roles and Responsibilities

• Design, develop, test, deploy, and maintain large-scale data pipelines using PySpark.
  
• Collaborate with cross-functional teams to gather requirements and deliver high-quality solutions.
  
• Develop scalable and efficient ETL processes for big data processing using Hadoop ecosystem components (HDFS, MapReduce, YARN).
  
• Troubleshoot complex issues related to data pipeline failures or performance bottlenecks.
  
• Ensure compliance with security standards by implementing access controls on sensitive datasets.