Senior Databricks Platform Engineer

Role & responsibilities

About the Role

Highly skilled Senior Databricks Platform Engineer to help build and scale a next-generation data platform supporting analytics, AI, governance, and self-service data consumption. The role focuses on engineering reusable platform capabilities, enabling enterprise-scale governance, improving observability, and driving operational excellence across the Databricks ecosystem.

Key Responsibilities

• Design and develop scalable data pipelines using Databricks, PySpark, SQL, and Delta Lake.
• Build CDC, batch, and streaming ingestion frameworks for enterprise-scale workloads.
• Implement and manage Medallion Architecture (Bronze, Silver, Gold).
• Develop governance solutions using Unity Catalog, including row-level security, data masking, and fine-grained access controls.
• Build and maintain Data Quality, Monitoring, and Observability frameworks.
• Implement CI/CD pipelines using Databricks Asset Bundles (DABS), Git, and Infrastructure as Code.
• Optimize Databricks clusters, SQL Warehouses, and workloads for performance and cost efficiency.
• Enable self-service analytics through semantic layers, metric views, governed datasets, and certified data products.
• Partner with platform, security, analytics, and business teams to drive adoption of Databricks capabilities.
• Participate in architecture discussions, platform modernization initiatives, and stakeholder workshops.

Required Skills

Databricks & Lakehouse

• Databricks Lakehouse Platform
• Delta Lake
• Unity Catalog
• Databricks Workflows
• Auto Loader
• Structured Streaming
• Databricks SQL

Data Engineering

• PySpark
• Advanced SQL
• Data Modeling
• ETL/ELT Frameworks
• CDC Architectures
• Real-Time Data Processing

Governance & Quality

• Row-Level Security
• Data Masking
• Data Lineage
• Data Quality Frameworks

Cloud & DevOps

• AWS/ GCP / Azure
• Git , CI/CD
• Terraform (nice to have)

Preferred Skills (These are not a must, nice to have)

• Scala
• dbt
• Kafka
• Airflow
• GenAI / AI Agents
• Enterprise Observability Solutions