AWS Databricks

Roles and Responsibilities :

• Design, develop, test, deploy and maintain data pipelines using AWS services such as S3, Lambda, Step Functions etc.
• Collaborate with cross-functional teams to gather requirements and design solutions for complex data processing needs.
• Develop ETL processes using PySpark on Delta Lake to extract insights from large datasets.
• Troubleshoot issues related to pipeline failures and optimize performance for improved efficiency.

Job Requirements :

• 5-15 years of experience in software development with expertise in AWS ecosystem (S3, Lambda).
• Strong understanding of SQL concepts and ability to write efficient queries.
• Experience working with big data technologies like Data Bricks or similar tools.
• Proficiency in developing ETL processes using PySpark on Delta Lake.