Lead Databricks Engineer

The Databricks Engineer designs, builds, and maintains scalable, secure, and audit-ready data pipelines and analytical workloads on Databricks. The role supports actuarial and financial platforms by enabling reliable data ingestion, transformation, calculation, and reporting, with a strong focus on data quality, performance, and governance.

The candidate will have responsibilities across the following functions:

Databricks Development and Engineering:

• Design, develop, and maintain data pipelines and analytical workloads using Databricks.
• Implement transformations using PySpark / Spark SQL and Delta Lake.
• Support end-to-end data flows across ingestion, curation, and analytics layers.
• Contribute to the evolution of common Databricks development frameworks and standards.

Data Quality, Performance and Reliability:

• Implement robust data validation, logging, and error handling mechanisms.
• Optimise workloads for performance, scalability, and cost efficiency.
• Ensure pipelines are reliable, recoverable, and observable.

Security, Governance and Audit:

• Implement and operate data governance controls using Unity Catalogue (access control, lineage, auditability).
• Ensure solutions comply with internal control, segregation of duties, and audit requirements.
• Support traceability of data and results for regulatory and business reviews.

Collaboration and Delivery:

• Work closely with actuarial SMEs, testers, and architects to translate requirements into technical solutions.
• Support SIT, UAT, and release activities by resolving data, performance, or pipeline-related issues.
• Contribute to documentation, technical reviews, and knowledge sharing.

Requirements:

• Approximately 5 years of experience in data engineering or analytics engineering roles.
• Hands-on experience building solutions on Databricks in enterprise environments.
• Bachelor's degree in Computer Science, Engineering, Mathematics, Data Science, or a related field (or equivalent experience).

Technical Skills:

• Solid coding experience in Python (PySpark) and SQL.
• Experience with Delta Lake, data modelling, and data transformation patterns.
• Experience building and operating data pipelines in cloud environments (Azure or AWS).
• Familiarity with version control and CI/CD practices.

Databricks and Platform Knowledge:

• Experience working with Unity Catalogue or similar data governance frameworks is an advantage.
• Understanding of performance tuning, cluster usage, and workload optimisation.
• Exposure to actuarial, financial, or risk-related data domains is a plus, but not mandatory.

Professional Skills:

• Strong analytical and problem-solving skills.
• Structured and quality-focused approach to engineering.
• Effective communicator able to work with technical and nontechnical stakeholders.
• Open, learning-oriented mindset and willingness to continuously improve.