Azure DataBrick Professional

Job ID :

1781

Roles & Responsibilities

• Design and develop ETL pipelines using Azure Databricks and Delta Lake, focusing on batch processing (autoloader) and Spark structured streaming.

• Create and manage end-to-end environments, including catalogs, schemas, tables, materialized views, functions, and volumes using Unity Catalog.

• Implement slowly changing dimensions (SCD1 and SCD2) on dimension tables and build change data capture (CDC) pipelines.

• Utilize Lakehouse federation to create foreign catalogs for accessing data from external sources.

• Optimize data processing through effective partitioning and liquid clustering in Databricks.

• Collaborate with cross-functional teams to ensure data governance and security practices are adhered to.

• Participate in CI/CD pipeline development and DevOps practices to enhance deployment efficiency.

Essential Knowledge, Skills, and Experience

• Expert-level proficiency in SQL

• Regular experience with CI/CD practices

Preferred Experience

• Azure Databrick , ETL Pipeline, Delta Lake, SQL, CI/CD