Data Engineer

We are looking for a Data Engineer with strong experience in Python, PySpark, Databricks, and AWS to build and optimize scalable data pipelines and cloud-based data platforms. The ideal candidate should have strong hands-on experience with data processing, ETL/ELT, and AWS data services.

Responsibilities:

• Design, develop, and maintain scalable ETL/ELT pipelines using Python and PySpark.
• Build and optimize data processing workflows using Databricks.
• Work extensively with AWS data services such as S3 Glue, Athena, Redshift, Lambda, EMR, and CloudWatch.
• Develop data pipelines capable of handling large volumes of structured and semi-structured data.
• Perform data transformation, cleansing, validation, and quality checks.
• Optimize Spark jobs for performance, scalability, and cost.
• Use Boto3 for AWS automation and integration.
• Work with Docker for containerized data applications and deployments.
• Implement monitoring, logging, and troubleshooting for production data pipelines.
• Collaborate with data scientists, ML engineers, and other engineering teams to support analytics and ML workloads.
• Follow Git-based development practices and CI/CD processes.

Requirements:

• 4-8 years of data engineering experience.
• Strong Python programming.
• Strong hands-on PySpark.
• Hands-on experience with Databricks.
• Strong AWS experience, particularly S3 Glue, Athena, EMR, Redshift, Lambda, CloudWatch, IAM, and VPC.
• Strong understanding of ETL/ELT and data pipeline architecture.
• Experience with data lakes / data warehouses / lakehouse concepts.
• Hands-on experience with Boto3
• Strong experience with Docker.
• Experience in Git/GitHub and CI/CD exposure, Linux/Shell scripting.
• Good understanding of data processing and performance optimization.