Data Engineer- ETL

Role & responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines to extract, transform, and load data from multiple sources.
• Develop Python scripts for data ingestion, transformation, automation, and data processing.
• Use PySpark to process large datasets, perform distributed transformations, joins, aggregations, and data cleansing.
• Develop and maintain Databricks notebooks, jobs, and data processing workflows.
• Write complex SQL queries using joins, CTEs, subqueries, window functions, and aggregations.
• Build and optimize data pipelines to ensure data accuracy, consistency, and reliability.
• Implement incremental data loading, deduplication, and change data capture (CDC) where required.
• Optimize Spark jobs, partitioning, caching, joins, and shuffle operations to improve performance.
• Optimize SQL queries and data processing logic to reduce execution time and resource consumption.
• Work with data formats such as CSV, JSON, and Parquet, and manage Delta Lake tables.
• Implement data quality checks, validation rules, and reconciliation processes.
• Schedule and monitor ETL workflows using Databricks Workflows or orchestration tools such as Apache Airflow.

Preferred candidate profile

• Strong knowledge of ETL/ELT concepts, data integration, data extraction, transformation, and loading processes.
• Proficiency in Python programming, including functions, data structures, exception handling, and file processing.
• Hands-on experience with PySpark DataFrames, transformations, actions, joins, aggregations, and window functions.
• Experience with Databricks notebooks, jobs, clusters or compute, and Delta Lake.
• Strong SQL skills, including joins, CTEs, subqueries, window functions, aggregations, and query optimization.
• Understanding of Apache Spark architecture, lazy evaluation, partitioning, caching, shuffling, and performance tuning.
• Knowledge of data formats such as CSV, JSON, Avro, and Parquet, along with Delta Lake tables.