Aws Data Engineer

Role & responsibilities

• Develop and maintain scalable ETL/ELT pipelines using AWS Glue, Databricks, and PySpark to process large volumes of data.
• Schedule and monitor ETL pipelines using AWS Glue Workflows, Databricks Workflows, or Apache Airflow.
• Write Python scripts for data ingestion, transformation, automation, and error handling.
• Use PySpark DataFrames, Spark SQL, joins, aggregations, and window functions to transform large datasets.
• Write complex SQL queries using joins, CTEs, subqueries, window functions, and aggregations.
• Work with AWS services such as Amazon S3, AWS Glue, Lambda, IAM, Redshift, and CloudWatch.
• Develop Databricks notebooks and manage jobs, clusters or compute, and Delta Lake tables.
• Implement incremental data loads, change data capture (CDC), and data processing workflows.
• Optimize Spark jobs, partitioning, joins, caching, and shuffle operations to improve performance and reduce cloud costs.
• Optimize SQL queries, table design, and file formats such as Parquet and Delta Lake for efficient data processing.
• Implement data quality checks to identify null values, duplicate records, and data inconsistencies.
• Troubleshoot ETL failures, analyze execution logs, and resolve pipeline performance issues.
• Implement error handling, retry mechanisms, logging, and alerting for reliable pipeline execution.
• Apply AWS IAM roles, access controls, and encryption to maintain data security.
• Use Git for version control and collaborate with team members through code reviews and deployment workflows.
• Monitor pipeline execution, resource utilization, job failures, and processing costs using AWS CloudWatch and Databricks monitoring tools.

Preferred candidate profile

• Strong knowledge of Python programming, including functions, data structures, exception handling, and file processing.
• Proficiency in SQL, including joins, CTEs, subqueries, window functions, and query optimization.
• Hands-on experience with PySpark for large-scale data processing, transformations, joins, and aggregations.
• Experience with Databricks notebooks, jobs, Delta Lake, and data processing workflows.
• Strong understanding of AWS services, including Amazon S3, AWS Glue, Lambda, IAM, Redshift, and CloudWatch.