Lead Engineer, Data & Analytics Cloud engineering

Job Description

Lead Engineer, EDT Data & Analytics Cloud Engineering

The Lead Cloud Platform Engineer supports the operational excellence, security, and reliability of the enterprise data & analytics platform on Azure and Databricks. The role combines platform operations, DevOps automation, and Infrastructure as Code across the core stack - Azure Databricks, Synapse, ADLS Gen2, Azure SQL, Azure Data Factory, Key Vault, and Unity Catalog - with a reliability-engineering mindset: automate repeatable work, apply security best practices by default, and help keep the platform highly available for global data engineering and analytics teams.

What you will do:

DevOps Pipeline Management & CI/CD

• Design, build, test, and operate Azure DevOps CI/CD pipelines that package code, run automated quality checks, deploy infrastructure and data workloads, and promote releases across development, testing, and production environments.
• Own day-to-day pipeline execution and release readiness by monitoring runs, diagnosing failed builds and deployments, correcting pipeline definitions, and improving rollback, approval, and promotion controls.

Infrastructure as Code & Environment Provisioning

• Write, review, and maintain Terraform modules for repeatable provisioning of Azure and Databricks platform components; test changes through plan and deployment cycles and keep modules versioned and reusable.
• Provision and configure workspaces, compute, storage, networking, private connectivity, identities, and access through automated IaC processes; detect configuration drift and implement corrective changes.

Platform Security & Access Management

• Configure and troubleshoot RBAC, service principals, managed identities, secrets, storage permissions, cluster policies, and Unity Catalog grants across Azure Data Factory, Databricks, Synapse, Azure SQL, ADLS Gen2, and Key Vault.
• Apply and verify least-privilege access, identity integration, and secrets-management controls through hands-on configuration reviews, access testing, and remediation of security findings.

Platform Operations & Reliability Engineering

• Build and tune monitoring, alerts, dashboards, and operational runbooks; participate directly in incident response, perform root-cause analysis, and implement preventive automation and reliability fixes.
• Monitor integration runtimes, clusters, jobs, pipelines, and platform services; inspect logs and metrics, reproduce failures, resolve runtime and performance issues, and validate service recovery against SLAs.

Data Engineering Operations

• Work directly with Data Platform and Data Product teams to deploy and operationalize ingestion pipelines, orchestration, Databricks jobs, and supporting platform services; provide hands-on support through production rollout.

Collaboration

• Translate stakeholders into implementable technical changes, contribute code and configuration, lead technical working sessions, and document the resulting solution and operational handoff.

What you need:

• Bachelors or masters degree in computer science, Engineering, Information Systems, Business, or a related technical discipline.
• 5-7 years of experience designing, engineering, and supporting cloud-based data and analytics platforms, data warehouses, and modern data ecosystems.
• Strong hands-on expertise with Azure data services, including Azure Databricks, Synapse Analytics, ADLS Gen2, Azure SQL, Azure Data Factory, Key Vault, and Unity Catalog.
• Proven experience implementing platform security controls, governance frameworks, and Azure data platform best practices to ensure secure, scalable, and compliant solutions.
• Proficiency in Infrastructure as Code (IaC) using Terraform and Azure DevOps CI/CD pipelines, with experience automating deployments across multiple environments.
• Demonstrated ability to monitor, optimize, and troubleshoot cloud data platforms, including performance tuning, alerting, incident management, and root cause analysis.
• Excellent analytical, problem-solving, and communication skills, with the ability to collaborate effectively across engineering, operations, security, and business stakeholders.
• Proactive mindset with a track record of driving automation, process optimization, operational improvements, and scalable platform enhancements that increase efficiency and reliability.
• Cloud Engineering certifications on Azure/Databricks platform is a plus.