Azure Data Engineer

Job Title: Azure Databricks Engineer Unity Catalog

Experience: 6-10 Years
Location: Bangalore
Employment Type: Full-Time

Job Summary

We are looking for an experienced Azure Databricks Engineer with strong hands-on expertise in Databricks, Unity Catalog, PySpark, SQL, and Azure Data Services. The candidate will be responsible for designing, developing, securing, and optimizing scalable data solutions using the Databricks Lakehouse platform.

The ideal candidate should have strong experience working with Unity Catalog for data governance, access control, lineage, and data discovery, along with expertise in building production-grade ETL/ELT pipelines.

Key Responsibilities

• Design, develop, and maintain scalable data engineering solutions using Azure Databricks.
• Develop robust ETL/ELT pipelines using PySpark, Python, and SQL.
• Implement and manage Unity Catalog for centralized data governance and access management.
• Configure and manage Catalogs, Schemas, Tables, Views, External Locations, and Storage Credentials in Unity Catalog.
• Implement role-based access control and data access policies using Unity Catalog.
• Work with Delta Lake for reliable, scalable, and high-performance data processing.
• Build and optimize Databricks Workflows / Jobs for production data pipelines.
• Integrate Databricks with Azure services such as ADLS Gen2, Azure Data Factory, Azure Synapse, Event Hubs, and Key Vault.
• Implement data quality, security, governance, and lineage requirements.
• Optimize Spark jobs, SQL queries, clusters, and Delta tables for performance and cost.
• Develop incremental data processing and slowly changing dimension (SCD) pipelines.
• Troubleshoot production data pipelines and resolve performance and data-quality issues.
• Implement CI/CD practices for Databricks notebooks, jobs, workflows, and code.
• Collaborate with data architects, analysts, application teams, and business stakeholders.
• Follow enterprise standards for security, governance, monitoring, and documentation.

Required Technical Skills

• Strong hands-on experience with Azure Databricks
• Strong expertise in Unity Catalog
• Advanced PySpark
• Strong Python and SQL
• Hands-on experience with Delta Lake
• Experience with Azure Data Lake Storage Gen2 (ADLS)
• Experience with Databricks Workflows / Jobs
• Good understanding of Apache Spark
• Experience with ETL/ELT and data pipeline development
• Understanding of data governance, security, access control, and data lineage
• Experience with Git and CI/CD

Good to Have

• Azure Data Factory
• Azure Synapse Analytics
• Azure Key Vault
• Azure Event Hubs
• Terraform
• Azure DevOps / GitHub Actions
• Databricks Asset Bundles
• Delta Live Tables / Lakeflow Declarative Pipelines
• Structured Streaming
• Kafka
• Experience with Medallion Architecture
• Experience with cloud data warehouse / lakehouse architecture

Unity Catalog Expertise – Mandatory

Candidates should have practical experience in:

• Catalog and schema creation
• Managed and external tables
• External Locations
• Storage Credentials
• Access control and privileges
• Workspace/catalog binding
• Data discovery and lineage
• Secure data sharing and governance
• Migration from legacy Hive Metastore to Unity Catalog
• Unity Catalog best practices for enterprise environments