AWS Senior Data Engineer

AWS Senior Data Engineer (Kafka & PySpark)

Experience: 10-15 Years

Location: Chennai, Bengaluru, Hyderabad

We are seeking an experienced AWS Data Engineer with strong expertise in Kafka, PySpark, Spark SQL, Python, and AWS to build and optimize large-scale batch and real-time data platforms.

Key Skills

• PySpark, Spark SQL, Python, SQL
• Apache Kafka (Kafka Connect, Schema Registry, Streaming)
• AWS: S3, Glue, EMR, Lambda, IAM, Redshift, Step Functions, CloudWatch
• ETL/ELT & Real-Time Data Pipelines
• Spark Structured Streaming
• Data Modeling, Data Lakes & Lakehouse Architecture
• CI/CD, Git, Jenkins/GitHub Actions
• Data Quality, Governance & Security

Responsibilities

• Design and develop scalable batch and streaming data pipelines on AWS.
• Build high-performance PySpark and Spark SQL applications.
• Implement Kafka-based event-driven and real-time streaming solutions.
• Integrate and optimize AWS data services for reliability and cost efficiency.
• Implement data quality, governance, monitoring, and security best practices.
• Build CI/CD pipelines and support production deployments.
• Troubleshoot performance bottlenecks and production issues.

Good to Have

• Amazon MSK, Kinesis
• Databricks, Delta Lake, Unity Catalog
• Airflow / MWAA
• Snowflake, DBT
• Terraform

Primary Skills: AWS Data Engineering, Kafka, PySpark, Spark SQL, Python, Streaming Data Engineering.