Product Development Engineer II

We are seeking a Product Development Engineer II to contribute to the development, support, and optimisation of Disney's Ad Technology data pipelines and big data platforms. This role will focus on hands-on engineering in Databricks, Spark, Flink, and Airflow (MWAA), helping to build reliable data workflows and supporting production operations. As a mid-level engineer, you will work under the guidance of Senior and Lead Engineers to develop SQL/Python-based pipelines, troubleshoot issues, and participate in monitoring and automation efforts. This role is ideal for someone with strong technical fundamentals who is eager to grow into advanced data engineering responsibilities.

The candidate will have responsibilities across the following functions:

Pipeline Development and Support:

• Develop and maintain ETL/ELT pipelines in Databricks, Spark, or Airflow.
• Write and optimise SQL and Python jobs for data ingestion and transformation.
• Assist in troubleshooting pipeline issues and job failures.

Monitoring and Operations:

• Support monitoring of pipeline health, performance, and resource usage using Datadog, CloudWatch, or similar.
• Respond to alerts and escalate incidents as needed.

Data Quality and Optimisation:

• Contribute to data validation, quality checks, and performance tuning of data pipelines.
• Assist in optimising pipelines for cost efficiency and performance.

Collaboration:

• Work with senior engineers, data engineering, and analytics teams to support project delivery and production operations.
• Participate in cross-team initiatives such as platform upgrades and security reviews.

Documentation and Learning:

• Contribute to documentation, runbooks, and operational checklists.
• Learn and adopt best practices for observability, automation, and incident response.

Requirements:

• 2+ years of experience in data engineering or related software engineering roles.
• Hands-on experience with SQL and scripting languages such as Python.
• Exposure to big data platforms (Databricks, Spark, Flink, or Airflow).
• Familiarity with AWS services (S3 Glue, Athena, EMR) for data workflows.
• Understanding of observability concepts (monitoring, logging, alerts) with tools like CloudWatch or Datadog.
• Strong problem-solving skills and willingness to learn from senior engineers.

Preferred Qualifications:

• Prior experience supporting data pipelines in production.
• Familiarity with infrastructure-as-code (Terraform) or CI/CD systems (GitLab, Jenkins, or GitHub Actions).
• Exposure to data governance, security, or compliance practices.
• Interest in pursuing certifications such as Databricks Data Engineer Associate or AWS Cloud Practitioner.
• Bachelor's degree in Computer Science, Engineering, or related technical field (or equivalent experience).