Senior Technology Engineer

Senior Technology Engineer Hadoop / CDP & DevOps

Role Overview

We are looking for a Senior Technology Engineer with strong hands-on experience in Hadoop and Cloudera Data Platform (CDP), complemented by DevOps and automation expertise. The role will focus primarily on CDP cluster administration, performance optimization, security, troubleshooting, automation, and containerized data services in a production environment.

The ideal candidate will work closely with Data Engineering and Platform teams to ensure highly available, secure, performant, and automated data platforms.

Key Responsibilities

Cloudera Data Platform / Hadoop

• Install, configure, upgrade, and maintain Cloudera Data Platform (CDP) clusters in production environments.
• Work with CDP 7.1.7+ and/or the latest available CDP versions.
• Configure and maintain Cloudera Data Services 1.5.4 / 1.5.5 or later.
• Implement and maintain CDP security requirements, including authentication, authorization, encryption, and security controls.
• Monitor and optimize cluster health, capacity, resource utilization, and overall performance.
• Troubleshoot complex issues across Spark, Hive, YARN, and Hadoop ecosystem components.
• Analyze and resolve Spark job performance issues in collaboration with Data Engineers.
• Perform cluster capacity planning, tuning, patching, upgrades, and maintenance activities.
• Support CDP containerization and containerized data services.

DevOps & Automation

• Develop and maintain automation using Ansible, Shell scripting, Terraform, Groovy, and Python.
• Automate CDP provisioning, configuration, patching, maintenance, and operational tasks.
• Build and maintain CI/CD pipelines and automated workflows for data and analytics projects.
• Work with tools such as GitHub Actions, Jenkins, and Argo CD.
• Automate security, compliance, and regulatory checks.
• Implement automated patching and configuration management processes.
• Apply Infrastructure as Code and DevOps best practices to data-platform environments.

Containerization & Platform Engineering

• Work with container technologies including Docker and Podman.
• Deploy and manage containerized workloads on Kubernetes and OpenShift (OCP).
• Support integration of CDP data services with containerized environments.
• Troubleshoot container, orchestration, networking, and platform-level issues.

Observability & Operations

• Implement and maintain platform monitoring, logging, alerting, and observability solutions.
• Establish operational dashboards and health checks for Hadoop/CDP environments.
• Proactively identify performance, capacity, reliability, and availability issues.
• Participate in production incident troubleshooting and root-cause analysis.

Required Technical Skills

Must-Have

• Strong hands-on production experience with Hadoop and Cloudera Data Platform (CDP).
• Experience installing, configuring, upgrading, and maintaining CDP 7.1.7+ or the latest CDP version.
• Experience with Cloudera Data Services 1.5.4 / 1.5.5 or later.
• Strong troubleshooting experience with:
• Apache Spark
• Hive
• YARN
• Hadoop
• Experience with CDP security configuration and administration.
• Experience optimizing Hadoop/CDP cluster and Spark job performance.
• Strong Linux/Unix administration skills.
• Good understanding of Linux networking fundamentals.
• Hands-on experience with Ansible and Shell scripting.
• Experience with Terraform and/or Infrastructure as Code.
• Experience developing or maintaining CI/CD pipelines using tools such as Jenkins, GitHub Actions, or Argo CD.
• Experience with Git/source control.
• Hands-on experience with Docker, Podman, Kubernetes, and/or OpenShift.
• Strong production troubleshooting and problem-solving skills.

Preferred

• Python development/scripting experience.
• Groovy scripting experience.
• Experience automating security and regulatory compliance checks.
• Experience with automated patching and configuration management.
• Experience implementing observability for data platforms.
• Experience working in Agile engineering environments.
• Experience collaborating with Data Engineering teams on Spark and analytics workload optimization.

DevOps / Engineering Practices

• Follow Agile engineering practices, including code reviews, CI/CD, automated testing, and continuous improvement.
• Develop reusable automation and operational tooling.
• Maintain infrastructure and application configurations through version control.
• Promote automation-first approaches to platform operations.
• Contribute to platform standards, documentation, and operational best practices.

Ideal Candidate Profile

The ideal candidate is a Hadoop/CDP-focused technology engineer who combines strong production platform administration skills with modern DevOps capabilities. They should be comfortable troubleshooting complex data-platform issues, automating repetitive operational activities, working with container platforms, and partnering with Data Engineering teams to improve reliability and performance.