Azure Data Engineer

Role & responsibilities

Develop and maintain ADF pipelines for data ingestion and orchestration.

Use Azure Databricks and SQL,Pyspark for data transformations, cleaning, joins, and aggregations.

Manage data in azure data lake storage using medallion architecture (bronze,sliver,gold ) layers

Implementing incremental loads using CDC, SCD types, metadata -driven pipelines.

Perform SQL queries and data validations to ensure accuracy and quality of data.

Monitor pipelines and troubleshoot production failures, including root cause analysis.

Perform data quality checks, including finding duplicates, nulls handing, schema validation .

Troubleshoot Pyspark/SQL job failure and performance issues and identify root causes

Skills

SQL Complex queries, joins, CTEs, window functions, optimizations, Stored procedures.

Azure Data Factory ETL/ELT Pipelines,triggers,meta data driven pipelines,activities, parameterization.

Databricks/Pyspark Data cleaning, transformations, joins, aggregations, partitioning, delta lake, performance optimizations and databricks workflows & jobs.

Azure data lake storage layers, security, blob vs gen2

Data warehouse data modeling, incremental loads, CDC, CDF, SCD types

DevOps & CI/CD Azure DevOps, Git, Branching, Deployment Pipelines.