Specialist - Data Platform Engineering

About the Role

We are looking for a Data Platform Engineer (IC2) to build and operate scalable, reliable, and high-performance data platforms. In this role, you will develop streaming and batch data pipelines, manage Databricks environments, and support our event-driven data infrastructure using Kafka and Kafka Connect.

This is an excellent opportunity for engineers with a strong foundation in distributed data processing who want to work on modern data engineering technologies at scale.

Key Responsibilities

• Develop, maintain, and optimize batch and streaming data pipelines using Apache Spark and Apache Flink.
• Build scalable ETL/ELT workflows for processing large datasets.
• Manage and maintain Databricks workspaces, clusters, jobs, notebooks, and workflows.
• Monitor, troubleshoot, and optimize Databricks jobs for performance and cost efficiency.
• Work with Apache Kafka for event-driven data processing and streaming applications.
• Configure, deploy, and troubleshoot Kafka Connect connectors for integrating various data sources and sinks.
• Ensure data reliability through monitoring, alerting, and operational best practices.
• Collaborate with software engineers, platform engineers, and data consumers to design robust data solutions.
• Participate in production support, incident response, and root cause analysis.
• Contribute to automation, documentation, and continuous improvement of the data platform.

What Youll Learn

• Design and operate enterprise-scale data platforms.
• Build high-throughput streaming applications using Spark and Flink.
• Manage production Databricks environments efficiently.
• Develop expertise in Kafka ecosystem components, including Kafka Connect.
• Work with modern cloud-native data engineering technologies and best practices.

Success Metrics

• Deliver reliable and scalable batch and streaming pipelines.
• Maintain high platform availability and operational excellence.
• Optimize job performance and infrastructure costs.
• Improve data quality, observability, and platform automation.
• Collaborate effectively across engineering teams to deliver business value. Qualifications

Required Qualifications

• Bachelors degree in Computer Science, Engineering, or a related field, or equivalent practical experience.
• 3-4 years of experience in software engineering or data engineering.
• Strong programming skills in Java, Scala, or Python.
• Hands-on experience with Apache Spark for large-scale data processing.
• Knowledge of Apache Flink for stream processing.
• Experience working with Databricks, including cluster management and job orchestration.
• Good understanding of Apache Kafka, including topics, partitions, producers, and consumers.
• Familiarity with Kafka Connect and connector deployment.
• Understanding of distributed systems, data pipelines, and streaming architectures.
• Experience with Git and CI/CD practices.
• Strong problem-solving and debugging skills.

Preferred Qualifications

• Experience with Delta Lake and Lakehouse architecture.
• Knowledge of cloud platforms such as AWS, Azure, or GCP.
• Familiarity with Airflow, Argo Workflows, or similar orchestration tools.
• Experience with monitoring tools such as Prometheus and Grafana.
• Understanding of Kubernetes and containerized deployments.
• Knowledge of SQL optimization and data modeling.