Data Engineer

Responsibilities:

• Design and build scalable data pipelines and ETL/ELT workflows.
• Develop data solutions using Python, SQL and PySpark/Spark.
• Design and implement AWS-based data architectures.
• Work with large-scale data across Data Lakes, Data Warehouses and Lakehouse platforms.
• Build and optimise batch and real-time data pipelines.
• Implement data quality, monitoring and pipeline reliability practices.
• Work with AI/ML teams to build data platforms supporting AI and analytics use cases.
• Contribute to technical architecture and mentor junior engineers.
• Collaborate with global clients and cross-functional teams.

Requirements:

• 8-12 years of experience in Data Engineering / Big Data Engineering.
• Strong AWS experience - Mandatory.
• Strong Cloud experience - Mandatory.
• Strong hands-on Python and SQL.
• Experience with PySpark / Spark.
• Strong ETL/ELT and data pipeline development experience.
• Experience with AWS services such as S3 Glue, EMR, Redshift, Athena, Lambda, Kinesis.
• Experience with Snowflake / Databricks / Redshift or similar data platforms.
• Strong understanding of data modelling and distributed systems.
• Experience with CI/CD and DevOps practices.
• Strong communication skills and ability to work with global stakeholders.
• Comfortable working in the 4:00 PM - 1:00 AM IST shift.
• Good to have: Palantir Foundry, Kafka, Airflow, Docker/Kubernetes, Terraform, MLOps/AI data pipelines.