Platform Engineer (Data bricks)

Role Overview This is an offshore role responsible for administering and operating the Databricks platform that supports the analytics, data engineering, and ML/AI workloads of an enterprise Agentic AI solution, including the streaming ingestion pipelines that feed it. The role ensures Databricks environments are reliable, secure, governed, and cost-efficient across non-production and production, and operates as part of a broader Azure platform engineering function within a follow-the-sun delivery model alongside onshore engineering and architecture teams.

Key Responsibilities

• Administer and operate Databricks workspaces, clusters, instance pools, jobs, and workflows across non-production and production environments.
• Design and enforce cluster policies for security, stability, and cost control, and manage autoscaling and job/all-purpose compute.
• Operate Spark Structured Streaming ingestion pipelines that consume from Kafka/Confluent and Azure Service Bus or Event Hubs and land data into Delta tables.
• Administer Unity Catalog for data governance, access control, and lineage, and manage Delta Lake optimisation, versioning, and retention.
• Contribute to data integration patterns between Databricks, upstream messaging, and downstream platform services.
• Build and maintain Terraform Infrastructure-as-Code for Databricks and related Azure data services.
• Implement CI/CD for notebooks, jobs, and configuration using GitHub Actions and Git-based workflows.
• Monitor platform health using Azure Monitor and Dynatrace, and troubleshoot Spark job failures, performance, memory, and scaling issues.
• Perform capacity planning, usage analysis, and cost optimisation across Databricks environments.
• Apply runtime upgrades and configuration changes following change-management best practices, and provide follow-the-sun / on-call support.

Qualifications & Experience

• 4+ years experience administering Databricks or Spark-based data platforms in enterprise environments.
• Deep understanding of Apache Spark architecture, execution, and performance tuning.
• Hands-on experience with Unity Catalog, Delta Lake, and lakehouse architectures.
• Experience with Spark Structured Streaming and ingestion from Kafka or Azure messaging services.
• Working knowledge of Microsoft Azure, including Azure Data Lake Storage, networking, and core PaaS services.
• Experience with Terraform Infrastructure-as-Code and Git-based CI/CD.
• Proficiency in Python, SQL, and Bash scripting.
• Strong troubleshooting and analytical skills, and strong communication for distributed collaboration.
• Bachelor’s degree in Computer Science, Engineering, Data Engineering, or a related discipline.

Preferred Skills

• Databricks Administrator or Data Engineer certification.
• Hands-on experience with Kafka/Confluent and Azure Service Bus or Event Hubs.
• Experience integrating data pipelines with messaging and API-based services.
• Familiarity with Azure Monitor, Application Insights, and Dynatrace observability.
• Familiarity with ITIL-based enterprise service management processes.