Aws Data Engineer

Role & responsibilities

xperience: 58 Years
Work Location: Hyderabad / Bangalore / Chennai
Job Type: Full-Time
Role: Data Engineer

Job Summary

We are looking for an experienced Data Engineer with strong expertise in SQL, Python, AWS Glue, Databricks, and ETL development. The ideal candidate will be responsible for designing, developing, optimizing, and maintaining scalable data pipelines and data integration solutions to support business intelligence, analytics, and data-driven initiatives.

The candidate should have hands-on experience in data transformation, data processing, cloud-based data engineering, and building reliable ETL/ELT workflows.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines using Python, SQL, AWS Glue, and Databricks.
• Build and optimize data integration workflows to ingest, transform, and process structured and semi-structured data.
• Develop complex SQL queries, stored procedures, joins, aggregations, and performance-tuned data transformations.
• Use Python for data processing, automation, validation, and pipeline development.
• Develop and maintain AWS Glue jobs, crawlers, and Data Catalog components for data processing and integration.
• Work with Databricks to develop data transformation workflows using PySpark, Spark SQL, and Delta Lake.
• Integrate data from multiple sources, including databases, cloud storage, and enterprise applications.
• Implement data quality checks, error handling, logging, monitoring, and reconciliation processes.
• Optimize ETL performance, query execution, and distributed data processing workloads.
• Collaborate with data architects, analysts, developers, and business stakeholders to understand data requirements.
• Follow best practices for data security, access management, version control, and deployment.
• Troubleshoot production data pipeline issues and ensure data availability, accuracy, and reliability.

Required Qualifications

• Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.
• 58 years of hands-on experience in Data Engineering, ETL development, and data processing.
• Strong practical experience in SQL and Python programming.
• Hands-on experience with AWS Glue and Databricks.
• Good understanding of data warehousing concepts, data modeling, and ETL/ELT architecture.
• Experience working with large datasets and developing scalable data pipelines.
• Strong analytical, troubleshooting, and problem-solving skills.

Preferred candidate profile

Must-Have Skills

• SQL: Complex queries, joins, CTEs, window functions, aggregations, query optimization, and performance tuning.
• Python: Data processing, scripting, automation, and ETL pipeline development.
• AWS Glue: Glue Jobs, Crawlers, Data Catalog, ETL workflows, and job monitoring.
• Databricks: Notebooks, PySpark, Spark SQL, data transformations, and distributed processing.
• ETL/ELT: Data ingestion, extraction, transformation, loading, data validation, and pipeline orchestration.
• Data Engineering: Data warehousing, data integration, data quality, and pipeline optimization.

Good-to-Have Skills

• Experience with Amazon S3 and other AWS data services.
• Knowledge of Delta Lake and Lakehouse architecture.
• Exposure to AWS Lambda, Amazon Redshift, CloudWatch, and IAM.
• Experience with Apache Spark and distributed data processing.
• Familiarity with Git, CI/CD pipelines, and deployment automation.
• Knowledge of Airflow or other workflow orchestration tools.
• Understanding of dimensional modeling, star schemas, and data lake architectures.
• Experience working in Agile development environments.

Perks and benefits