AWS Pyspark Developer

Job Summary

We are seeking an experienced AWS PySpark Developer with strong expertise in building scalable data engineering solutions on AWS Cloud. The ideal candidate should have hands-on experience in PySpark, Spark, AWS data services, ETL development, and large-scale data processing. The role involves designing, developing, and optimizing cloud-native data pipelines, data lakes, and analytics solutions.

Key Responsibilities

• Design and develop scalable ETL/ELT pipelines using PySpark.
• Build and maintain data ingestion frameworks on AWS.
• Develop batch and real-time data processing solutions.
• Optimize Spark jobs for performance and cost efficiency.
• Work with structured, semi-structured, and unstructured datasets.
• Develop data lake and data warehouse solutions.
• Implement data quality checks, monitoring, and alerting.
• Collaborate with business stakeholders, architects, and analytics teams.
• Support production deployments and troubleshoot critical issues.
• Follow DevOps and CI/CD best practices.

Mandatory Technical Skills

AWS Services

• Amazon S3
• AWS Glue
• AWS EMR
• AWS Lambda
• Amazon Redshift
• AWS IAM
• CloudWatch
• AWS Step Functions

Big Data & Data Engineering

• PySpark
• Apache Spark
• Spark SQL
• DataFrames & RDDs
• ETL/ELT Development
• Data Modeling
• Data Warehousing Concepts

Programming

• Python
• SQL
• Shell Scripting

Database Technologies

• Redshift
• PostgreSQL
• MySQL
• DynamoDB (Preferred)

DevOps & CI/CD

• Git
• Jenkins / AWS CodePipeline
• Docker
• Terraform (Preferred)

Good-to-Have Skills

• Databricks
• Kafka
• Apache Airflow
• Delta Lake
• Snowflake
• Kubernetes
• Streaming Technologies
• Machine Learning Exposure