Scala Spark developer

Role & responsibilities

Senior Data Platform Engineer (Python/Spark Platform Engineering)

We are looking for a highly skilled and self-driven Senior Data Platform Engineer with strong expertise in Python, Spark, Big Data Platforms, and Cloud Data Engineering. The ideal candidate should be capable of independently understanding existing data frameworks, enhancing them with new capabilities, and leading modernization initiatives involving cloud-native technologies.

Key Responsibilities

• Design, develop, and maintain scalable data ingestion and transformation frameworks using Python, Spark, and Shell Scripting.
• Independently analyze existing frameworks and implement new features and enhancements.
• Drive platform modernization through cloud-native technologies, automation, and engineering best practices.
• Develop optimized integrations with object storage platforms, databases, and enterprise data systems.
• Build and support high-performance data pipelines on large-scale big data platforms.
• Leverage AI-assisted development tools, AI agents, and LLMs to accelerate engineering productivity and solution delivery.

Required Skills

• Strong hands-on expertise in Python and Apache Spark development and framework design.
• Experience building reusable and scalable data ingestion and transformation frameworks.
• Deep understanding of Hive, Impala, HDFS, and the Hadoop ecosystem, preferably on Cloudera-based platforms.
• Strong knowledge of HDFS internals, data storage architecture, compaction strategies, partition management, resource utilization, and cluster performance optimization.
• Proven ability to troubleshoot complex platform, storage, compute, and cluster-level issues, perform root cause analysis, and drive performance improvements.
• Experience working with diverse database technologies and object storage platforms, with a focus on optimized connectivity and data access patterns.
• Strong understanding of distributed data processing, Spark optimization, and platform engineering best practices.

Preferred Skills

• Experience developing applications and frameworks using Scala.
• Strong hands-on experience with Kubernetes-based application development, containerized Spark workloads, pod management, scaling, troubleshooting, and operational support.
• Experience building and deploying Spark workloads using JFrog-managed container images and CI/CD pipelines.
• Hands-on experience with Databricks and modern cloud-native data platforms.
• Exposure to cloud migration and enterprise data platform modernization initiatives.
• Experience leveraging AI Agents, Generative AI, and LLM-powered development tools to accelerate software delivery.

Ideal Candidate

A strong Python/Spark architect, designer, and developer who can quickly understand existing enterprise frameworks, work independently with minimal supervision, and drive platform evolution, cloud modernization, and performance optimization across both application and infrastructure layers. The candidate should possess a balanced mix of development expertise, platform engineering knowledge, Hadoop/Cloudera operations experience, and Kubernetes-based deployment capabilities.