Databricks Engineer

Role Overview

We are looking for a Data Engineer with strong experience in Azure data engineering, Databricks and data lake technologies, along with an understanding of IT Security/Cybersecurity use cases. The ideal candidate is a generalist who can work across security data engineering, analytics, Data Science and emerging GenAI use cases.

Key Responsibilities

• Design, develop and maintain scalable data pipelines for IT Security and Cybersecurity use cases.
• Build and manage data solutions using Azure Data Lake Storage (ADLS/ADLS Gen2), Azure Databricks and Azure data services.
• Ingest, transform and process security data from multiple sources such as SIEM, endpoint, identity, network, cloud and application security platforms.
• Develop Databricks/PySpark solutions for processing and analysing large volumes of security telemetry.
• Work with security logs, alerts, vulnerability data, threat intelligence, authentication data and other security datasets.
• Work with Azure Security services and understand security-related data generated across the Azure/Microsoft security ecosystem.
• Develop data models and curated datasets to support Security Operations, Threat Detection, Risk Analytics and reporting.
• Collaborate with Data Scientists to prepare datasets for machine learning, anomaly detection, behavioural analytics and predictive use cases.
• Support GenAI/LLM/RAG use cases involving security data, knowledge retrieval and intelligent security analytics.
• Develop automation scripts using Python Build and maintain Power BI dashboards and reports for security metrics, trends, risks and operational KPIs.
• Implement data quality, governance, access controls and monitoring across data pipelines.
• Troubleshoot data ingestion, transformation and pipeline issues and optimize performance and cost.
• Collaborate with Cybersecurity, SOC, Data Engineering, Data Science and Cloud Engineering teams.

Required Skills & Experience

• 8+ years of experience in Data Engineering, Data Analytics, Security Data Engineering or a related field.
• Strong hands-on experience with Microsoft Azure data services. Strong experience with Azure Data Lake Storage (ADLS/ADLS Gen2).
• Hands-on experience with Azure Databricks and PySpark.
• Strong SQL and Python skills. Understanding of IT Security/Cybersecurity data or security-related use cases.
• Good understanding of Azure Security / Microsoft security ecosystem. Experience in data ingestion, transformation, data modelling and pipeline development.
• Experience with Power BI dashboards and reporting.

Data Science / AI Exposure

• Good understanding of Data Science concepts, including statistics, machine learning, anomaly detection and feature engineering.
• Ability to work with Data Scientists and understand ML/data requirements.
• Exposure to GenAI, LLM, RAG or AI-powered analytics is preferred. Ability to explore and analyze large security datasets and identify meaningful patterns and trends.

Preferred Certifications:

• Databricks Certified Data Engineer Associate / Professional or equivalent Databricks certification.
• Microsoft security certifications such as SC-200 Security Operations Analyst Associate, SC-500 – Cloud and AI Security Engineer Associate
• Other relevant certifications such as Security+, CEH, CISSP or equivalent are an advantage.