Data Engineer (AWS | Spark | Python)

About the Role

We're looking for an expert-level Data Engineer to design, build, and optimize scalable cloud data solutions for our Cloud Data Engineering team. You'll work with AWS, Apache Spark, and Python to deliver robust data pipelines and platforms that power analytics and business decisions, while championing automation and CI/CD best practices.

Key Responsibilities

• Design, develop, and maintain scalable batch and streaming data pipelines on AWS
• Build and optimize large-scale data processing workloads using Apache Spark (PySpark)
• Write clean, efficient, and well-tested Python code for data transformation and automation
• Implement and manage CI/CD pipelines for data engineering workflows
• Ensure data quality, reliability, security, and performance across data platforms
• Troubleshoot and tune pipeline performance and cost on cloud infrastructure
• Collaborate with data scientists, analysts, architects, and product teams to translate requirements into technical solutions
• Mentor junior engineers and contribute to code reviews, standards, and best practices

Required Skills & Qualifications

• Expert-level experience in data engineering, with a strong track record of delivering production-grade solutions
• Hands-on expertise with AWS services such as S3, Glue, EMR, Lambda, Redshift, Athena, and IAM
• Strong working knowledge of Apache Spark, including performance tuning and optimization
• Advanced proficiency in Python
• Experience building and maintaining CI/CD pipelines (e.g., Jenkins, GitHub Actions, GitLab CI, AWS CodePipeline)
• Solid understanding of data modeling, ETL/ELT patterns, and data warehousing concepts
• Strong SQL skills
• Excellent problem-solving, communication, and stakeholder management skills

Good to Have

• Experience with orchestration tools such as Airflow or Step Functions
• Familiarity with Infrastructure as Code (Terraform, CloudFormation)
• Exposure to Docker, Kubernetes, and containerized deployments
• Knowledge of Delta Lake, Iceberg, or Hudi
• AWS certifications (e.g., AWS Certified Data Analytics / Solutions Architect)