Senior Dev Ops Engineer

Role Description

Role Summary

We are looking for a hands-on Senior DevOps / Databricks Platform Engineer to support an enterprise migration from Azure Databricks to AWS Databricks. The engineer will own standard AWS Databricks platform provisioning, Terraform automation, cloud networking and security integration, CI/CD enablement, platform configuration migration, production cutover, and hypercare. The role requires independent delivery capability and leadership qualities appropriate for an L3/L4 profile.

Role Description Technical

• Design, provision, configure, and maintain AWS Databricks environments across DEV, QA/UAT, and PROD using reusable Terraform modules and infrastructure-as-code standards.
• Configure AWS services required for Databricks, including VPCs, subnets, routing, security groups, IAM roles and policies, S3, KMS, CloudWatch, Secrets Manager, and VPC endpoints.
• Assess Azure Databricks platform configurations and map workspaces, compute, cluster policies, identities, permissions, storage, secrets, and networking to suitable AWS equivalents.
• Support migration of Databricks workspaces, jobs/workflows, repositories, libraries, Unity Catalog objects, storage credentials, external locations, volumes, grants, and service principals.
• Retrofit platform references from ADLS Gen2/abfss paths to Amazon S3 and migrate the approved secret-management and authentication patterns from Azure to AWS.
• Build and maintain basic CI/CD pipelines for Terraform and Databricks assets using Azure DevOps, GitHub Actions, Jenkins, GitLab CI, or equivalent tools.
• Implement SSO, SCIM, Unity Catalog access controls, logging, monitoring, auditability, and standard platform security controls.
• Support Databricks Runtime and library compatibility, Azure-to-AWS compute mapping, deployment promotion, cutover, rollback planning, production validation, and hypercare.
• Troubleshoot networking, IAM, authentication, authorization, cluster, workflow, Terraform, and deployment issues across Azure, AWS, and Databricks.
• Create deployment guides, architecture/configuration documents, operational runbooks, and knowledge-transfer material.
  
  

Role Description Project Delivery

• Collaborate with solution architects, data engineers, QA, BI, security, networking, and customer platform teams to complete environment readiness and migration activities.
• Participate in discovery, sprint planning, technical reviews, release planning, cutover activities, and Agile ceremonies; proactively track risks, dependencies, and blockers.
• Provide estimates, review Terraform and CI/CD changes, guide junior engineers, and communicate technical decisions clearly to customer and internal stakeholders.
• Coordinate required access and approvals and identify requirements outside standard platform setup that may need design review or change control.
  
  

Mandatory Skills

• Experience in DevOps, cloud infrastructure, platform engineering, with strong hands-on AWS and working knowledge of Azure.
• Hands-on expertise in AWS IAM, VPC, subnets, routing, security groups, S3, EC2, KMS, CloudWatch, Secrets Manager, and private connectivity concepts.
• Strong Terraform experience: reusable modules, remote state, state locking, environment separation, imports, module versioning, plan/apply workflows, and code reviews.
• Strong Databricks administration experience covering workspaces, compute, cluster policies, jobs/workflows, Unity Catalog, external locations, permissions, service principals, and secrets.
• Experience with Azure Databricks, ADLS Gen2, Microsoft Entra ID/Azure AD, Azure Key Vault, service principals, managed identities, and Azure networking concepts.
• Experience implementing CI/CD for infrastructure and Databricks assets; proficiency with Git and at least one scripting language such as Python, Bash, or PowerShell.
• Solid understanding of cloud networking, IAM, encryption, secrets management, monitoring, production deployments, cutover, rollback, and incident troubleshooting.
• Strong analytical, documentation, and verbal/written communication skills; ability to work independently in an onshore-offshore delivery model.
  
  

Good to Have

• Prior hands-on Azure Databricks to AWS Databricks migration experience will be a strong advantage.
• Experience with Databricks REST APIs/CLI, Databricks Asset Bundles, Terraform Databricks provider, workspace migration automation, and cross-cloud data movement.
• Knowledge of Delta Lake and medallion architecture; working familiarity with Python, PySpark, SQL, Power BI connectivity, and Databricks cost/compute controls.
• Experience developing or supporting Apache Airflow DAGs and Databricks operators.
• Certifications such as HashiCorp Terraform Associate, Databricks certification, AWS Solutions Architect/DevOps Engineer, or relevant Azure certification.