Senior Data Engineer

Job description

Role Summary We are looking for an experienced Azure Databricks Platform Admin with 57 years of experience in cloud data platform administration operations and support The role is responsible for administering securing monitoring and optimizing Azure Databricks environments across development test and production landscapes The ideal candidate should have strong handson experience with workspace administration cluster management Unity Catalog security controls automation and Azure platform services

Key Responsibilities

Administer and maintain Azure Databricks workspaces clusters jobs notebooks and platform configurations

Configure and manage Unity Catalog metastores catalogs schemas external locations and storage credentials

Implement and enforce security controls including access management rolebased permissions service principals secret scopes and integration with Microsoft Entra ID

Manage cluster policies autoscaling runtime versions job clusters and cost optimization measures to improve platform efficiency

Integrate Azure Databricks with Azure services such as ADLS Gen2 Azure Key Vault Azure Monitor Private Link VNets and managed identities

Monitor platform health audit logs usage patterns job performance and troubleshoot failures connectivity issues and performance bottlenecks

Support CICD and infrastructure automation using tools such as Terraform Azure DevOps Databricks CLI or REST APIs

Maintain environment standards across Dev QA and Prod including deployment controls documentation and operational runbooks

Collaborate with data engineering architecture security and support teams to ensure platform reliability governance and compliance

Drive continuous improvement in platform operations observability security posture and cost management

Required Skills and Qualifications

57 years of overall experience in cloud platform administration data platform operations or infrastructure support

Handson experience administering Azure Databricks in enterprise environments

Strong knowledge of Unity Catalog Delta Lake workspace governance access controls and Databricks administration concepts

Good understanding of Azure services including ADLS Gen2 Key Vault Entra ID networking private endpoints and monitoring services

Experience with cluster optimization troubleshooting Spark job issues workload tuning and platform monitoring

Exposure to automation and DevOps practices using Terraform Azure DevOps GitHub Actions Python Bash or PowerShell

Knowledge of identity and access management audit logging compliance and data security best practices

Working knowledge of SQL and familiarity with Python or scripting for platform automation

Strong problemsolving stakeholder communication and documentation skills

Preferred Qualifications

Certification in Azure Administration Azure Data Engineering or Databricks platform technologies is preferred

Experience supporting platform setup in secure enterprise environments with governance and network controls

Familiarity with FinOps practices tagging strategies and cloud cost reporting

Exposure to data engineering workloads orchestration frameworks and performance optimization best practices

Experience creating SOPs knowledge articles and platform support documentation

Skills

Mandatory Skills : Azure Functions, Databricks
Good to Have Skills : Azure Key Vault