DevOps Engineer II - Databricks and Databases

We are looking for a DevOps Engineer II with strong hands-on experience in Databricks, databases, Kubernetes, and AWS.

The core responsibilities for the job include the following:

Databricks:

• Manage and maintain Databricks infrastructure and production workloads.
• Manage clusters, jobs, compute resources, configurations, permissions, and integrations.
• Monitor Databricks workloads and troubleshoot performance and reliability issues.
• Drive Databricks cost optimization through cluster sizing, autoscaling, job optimization, scheduling, and resource utilization.
• Establish best practices around Databricks governance, security, access management, and operational efficiency.

Database and Data Infrastructure:

• Strong hands-on experience managing databases in production environments.
• Perform database maintenance, monitoring, troubleshooting, performance tuning, backup/recovery, and capacity management.
• Work with databases such as PostgreSQL, MySQL, Aurora, Redis, or similar technologies.
• Troubleshoot database performance, connectivity, storage, replication, and availability issues.
• Identify opportunities for database cost and performance optimization.

AWS and Cloud Infrastructure:

• Hands-on experience with AWS infrastructure and services, including EKS, EC2, RDS/Aurora, S3, IAM, CloudWatch, VPC, ECR, and related services.
• Manage and maintain production AWS infrastructure with a strong focus on reliability, security, scalability, and cost optimization.
• Troubleshoot complex infrastructure and networking issues.

Kubernetes and DevOps:

• Strong hands-on experience with Kubernetes, preferably AWS EKS.
• Experience with deployments, troubleshooting, scaling, networking, workloads, ingress, storage, and cluster maintenance.
• Experience with Helm, ArgoCD, Terraform, GitHub Actions/Bitbucket Pipelines, or similar DevOps tools.
• Build and maintain CI/CD pipelines and automate repetitive operational tasks.
• Implement Infrastructure as Code and follow best practices for infrastructure provisioning and management.

Monitoring and Reliability:

• Experience with monitoring and observability tools such as Prometheus, Grafana, CloudWatch, OpenTelemetry, or similar.
• Build dashboards, alerts, and operational metrics for production systems.
• Participate in incident management, root-cause analysis, and reliability improvements.
• Ensure production systems meet availability, performance, and operational requirements.

Requirements:

• 4-7+ years of DevOps/cloud/informational experience.
• Strong hands-on experience with Databricks.
• Strong experience with AWS.
• Strong hands-on experience with Kubernetes / EKS.
• Strong database administration and troubleshooting experience.
• Experience in Terraform / Infrastructure as Code, CI/CD and Git-based workflows, Helm and GitOps tools such as ArgoCD, monitoring and observability.
• Strong Linux and networking fundamentals.
• Strong troubleshooting and problem-solving skills.
• Experience working with production environments at scale.