Senior Big Data Engineer (AWS | PySpark | EMR)

Senior Big Data Engineer (AWS | PySpark | EMR)

Location: Bangalore (Hybrid/Onsite)

Experience: 10 to 16Years

Relevant Experience: Minimum 8 Years in Data Engineering / Big Data

Job Description

We are looking for a highly skilled Senior Big Data Engineer with strong hands-on experience in AWS, PySpark/Scala, Apache Spark, EMR, and enterprise-scale data processing.

Candidates should have proven experience building and optimizing large-scale distributed data platforms and must be comfortable working with complex ETL ecosystems handling massive data volumes.

Mandatory Skills

AWS (Must Have)

• Strong hands-on experience with AWS ecosystem
• Expert knowledge of:
• AWS EMR
• S3
• Glue
• Redshift
• Athena
• Lambda
• Step Functions
• CloudWatch
• Experience designing and managing large-scale AWS data platforms

Big Data Technologies (Must Have)

• Apache Spark
• PySpark and/or Scala
• Hadoop Ecosystem
• Hive
• HDFS
• Spark SQL
• Distributed Data Processing

Data Engineering (Must Have)

• Design and development of scalable ETL/ELT pipelines
• Batch and large-scale data processing
• Data Lake and Data Warehouse solutions
• Performance tuning of Spark applications
• Data quality and validation frameworks

Orchestration (Must Have)

• Apache Airflow
• AWS Step Functions
• Workflow orchestration
• DAG design and optimization
• Job scheduling and dependency management
• State management for enterprise data pipelines

Required Experience

• 8+ years of Data Engineering / Big Data experience
• Strong experience handling TB/PB-scale datasets
• Experience working with large Spark clusters
• Expertise in Spark optimization techniques:
• Partitioning
• Caching
• Broadcast joins
• Shuffle optimization
• Memory tuning
• Cluster tuning
• Experience solving distributed computing challenges

Preferred Skills

• Kafka
• Spark Streaming
• Databricks
• Snowflake
• CI/CD Pipelines
• Terraform
• Python
• SQL
• Cloud Migration Projects

Responsibilities

• Build and maintain scalable Big Data platforms on AWS
• Develop and optimize PySpark/Scala applications
• Design enterprise-grade ETL/ELT frameworks
• Create and maintain Airflow DAGs and orchestration workflows
• Optimize Spark jobs and EMR cluster performance
• Implement monitoring, alerting, and reliability solutions
• Work closely with business and analytics teams to deliver data solutions
• Troubleshoot production issues and improve platform stability