Platform Engineer

Payroll Company: Intverse IT Solutions

Employement: Full time

Job title: Platform Engineer

Job Title: Open Source Spark on Kubernetes Platform Engineer / Architect

Role Overview

We are looking for an experienced **Spark / Kubernetes Platform Engineer** to design, implement, and enable an enterprise-grade **open-source Apache Spark compute platform running on Kubernetes**.

The platform will provide scalable Spark compute capabilities for enterprise data engineering workloads involving **Snowflake and Databricks**, with automated deployment and lifecycle management using **GitHub and GitHub Actions**.

The ideal candidate will have strong hands-on experience with Apache Spark, Kubernetes, containerization, cloud-native infrastructure, CI/CD, security, observability, and enterprise data platforms.

Key Responsibilities

Open-Source Apache Spark Platform

- Design and implement an enterprise-grade **open-source Apache Spark platform on Kubernetes**.

- Deploy and operate Spark drivers and executors as Kubernetes workloads.

- Configure Spark for dynamic allocation, autoscaling, resource quotas, namespaces, and workload isolation.

- Build standardized Spark runtime/container images for enterprise workloads.

- Establish reusable configurations and templates for Spark applications.

- Implement Spark job submission and lifecycle management on Kubernetes.

- Optimize Spark workloads for performance, scalability, reliability, and infrastructure cost.

- Support batch processing and large-scale distributed data transformation workloads.

Kubernetes Platform Engineering

- Design Kubernetes architecture required to support distributed Spark workloads.

- Configure namespaces, service accounts, RBAC, secrets, ConfigMaps, resource quotas, network policies, and storage.

- Implement Kubernetes node pools and compute configurations optimized for Spark.

- Enable Kubernetes autoscaling based on Spark workload demand.

- Implement workload isolation for different applications, environments, and business teams.

- Troubleshoot Spark driver, executor, networking, storage, and Kubernetes scheduling issues.

- Establish platform standards for development, QA, staging, and production environments.

Snowflake Integration

- Enable Apache Spark workloads to securely read and write data from **Snowflake**.

- Configure and manage Spark-Snowflake connectivity and required JDBC/connectors.

- Implement secure authentication and secrets management.

- Optimize data movement between Spark compute and Snowflake.

- Establish reusable connectivity patterns for enterprise Spark applications.

- Support large-volume data ingestion and transformation workloads involving Snowflake.

Databricks Integration

- Enable interoperability between the open-source Spark platform and **Databricks** ecosystems.

- Support Spark-based processing of data used by Databricks workloads.

- Implement integration patterns for Delta/Parquet-based data where applicable.

- Support migration or portability of appropriate Spark workloads between Databricks and open-source Spark.

- Identify dependencies on Databricks-specific functionality and define open-source/cloud-native alternatives where required.

- Establish common engineering standards across Spark and Databricks workloads.

GitHub & GitHub Actions

- Establish **GitHub-based source control and GitOps practices** for the Spark platform.

- Develop reusable **GitHub Actions** workflows for CI/CD.

- Automate:

- Spark application build and testing

- Docker/container image creation

- Security and vulnerability scanning

- Deployment to Kubernetes

- Configuration promotion across environments

- Spark job deployment

- Rollback and release management

- Implement branching, pull request, code review, and release strategies.

- Manage reusable GitHub Actions workflows for multiple application teams.

- Integrate GitHub Actions with Kubernetes and cloud infrastructure securely.

Containerization & DevOps

- Build optimized Docker images for Spark drivers and executors.

- Maintain standardized enterprise Spark base images.

- Automate dependency and package management.

- Integrate artifact/container repositories with GitHub Actions.

- Implement Infrastructure as Code and configuration-as-code practices.

- Establish automated deployment pipelines across DEV, QA, UAT, and PROD.

Security

- Implement Kubernetes RBAC and least-privilege access.

- Secure Spark connectivity to Snowflake and other enterprise data platforms.

- Integrate enterprise secrets-management solutions.

- Secure credentials, certificates, tokens, and service accounts.

- Implement container vulnerability scanning and dependency scanning.

- Support enterprise security, audit, and compliance requirements.

Monitoring & Observability

- Implement monitoring for Spark applications and Kubernetes infrastructure.

- Capture Spark driver/executor metrics, application logs, job status, resource utilization, and failures.

- Integrate with enterprise observability platforms such as:

- Prometheus

- Grafana

- OpenSearch/ELK

- Develop dashboards and alerts for Spark platform health.

- Establish troubleshooting and operational runbooks.

- Implement monitoring for CPU, memory, executor utilization, shuffle performance, job duration, failed jobs, and Kubernetes capacity.

Required Technical Skills

- 7+ years of experience in Data Engineering, Big Data, Cloud Platform Engineering, or DevOps.

- Strong hands-on experience with **Apache Spark**.

- Strong experience running **Spark workloads on Kubernetes**.

- Strong Kubernetes architecture and administration experience.

- Strong knowledge of Spark drivers, executors, partitions, shuffle, memory management, dynamic allocation, and performance tuning.

- Experience building and maintaining Docker/container images.

- Strong **GitHub and GitHub Actions** experience.

- Experience developing enterprise CI/CD pipelines.

- Experience with **Snowflake** and Spark/Snowflake integration.

- Experience with **Databricks and Apache Spark ecosystems**.

- Strong scripting/programming skills using Python, PySpark, Bash, or similar technologies.

- Experience with Kubernetes RBAC, secrets, service accounts, networking, storage, and autoscaling.

- Experience with monitoring and observability technologies.

Preferred Skills

- Spark Operator / Kubernetes Operator experience.

- Helm and Kubernetes package management.

- Terraform or equivalent Infrastructure as Code experience.

- Argo CD or GitOps experience.

- Delta Lake / Apache Iceberg experience.

- Airflow or other workflow orchestration experience.

- Prometheus and Grafana.

- OpenSearch / ELK.

- Cloud experience with AWS, Azure, or GCP.

- Experience supporting enterprise-scale data platforms.

- Experience migrating Spark workloads from proprietary or managed platforms to open-source Spark.

Key Deliverables

The successful candidate will help establish:

**Open Source Spark Kubernetes Snowflake / Databricks**

with:

**GitHub GitHub Actions Container Build Kubernetes Deployment Spark Execution Monitoring & Operations**

Key deliverables include an enterprise Spark-on-Kubernetes architecture, standardized Spark container images, automated GitHub Actions CI/CD pipelines, Snowflake connectivity, Databricks interoperability, security/RBAC standards, autoscaling and resource-management standards, observability dashboards, operational runbooks, and reusable onboarding templates for data engineering teams.

Ideal Candidate

We are looking for someone who is not only a Spark developer but has strong **platform engineering experience** and can build the underlying Spark-on-Kubernetes capability.

The candidate should be comfortable owning the solution end-to-end across **Apache Spark, Kubernetes, Snowflake, Databricks, GitHub, GitHub Actions, containers, security, CI/CD, and observability**.