AWS EMR,Python with Pyspark

Job Description - Senior AWS Engineer

• Design and implement distributed data processing on AWS EMR using PySpark for batch/stream workloads.
• Build robust, scalable ETL/ELT pipelines integrating diverse data sources (files, RDBMS, APIs, streaming).
• Optimize Spark jobs (partitioning, caching, broadcast joins), and tune EMR clusters (instance types, autoscaling, bootstrap actions).
• Implement data lake patterns on S3 with appropriate file formats (Parquet/ORC), schema evolution, and table management.
• Orchestrate workflows using AWS Step Functions, Airflow, or EMR managed workflow, and automate CI/CD.
• Ensure cost governance (spot instances, right-sizing, scaling policies) and observability (CloudWatch, Spark metrics).
• Enforce security & compliance (IAM, KMS, encryption at rest/in transit, VPC, SGs, Glue Catalog access controls).
• Collaborate with analytics teams to expose curated datasets to Redshift, Athena, Glue, or downstream BI tools.
• Troubleshoot performance bottlenecks and production issues; implement resiliency patterns and SLAs.
• Document architecture, standards, runbooks; mentor junior engineers.