Data Engineer (Python, SQL, PySpark & AWS) For Bangalore

Skills- Data Engineer (Python, Spark, SQL,AWS)

We're looking for a skilled Data Engineer to design, build, and maintain scalable data pipelines that power analytics, reporting, and machine learning initiatives. You'll work extensively with PySpark for large-scale data processing and the AWS ecosystem to build reliable, cloud-native data infrastructure.

Key Responsibilities

• Design, develop, and maintain ETL/ELT pipelines using PySpark for batch and streaming data processing
• Build and manage data lake and data warehouse solutions on AWS (S3, Redshift, Glue, EMR, Athena, Lake Formation)
• Develop and orchestrate workflows using AWS Step Functions, Apache Airflow, or AWS Glue Workflows
• Optimize Spark jobs for performance, cost, and scalability (partitioning, caching, cluster tuning)
• Ingest data from multiple sources (APIs, databases, flat files, streaming platforms like Kafka/Kinesis)
• Implement data quality checks, validation frameworks, and monitoring/alerting for pipeline health
• Collaborate with data analysts, data scientists, and business stakeholders to understand data requirements
• Design and maintain data models (star/snowflake schemas) for analytics use cases
• Write clean, well-documented, testable code following engineering best practices (CI/CD, version control)
• Ensure data security, governance, and compliance (IAM policies, encryption, access controls)
• Troubleshoot and resolve production data pipeline issues