Senior Data Engineer

Role: Senior Data Engineer – GCP Data Platform

Objective of the Role

We are looking for a hands-on Senior Data Engineer to design and build scalable data solutions and reusable data-platform capabilities on Google Cloud Platform (GCP).

This role combines Data Engineering and Data Platform Engineering. You will provide technical leadership for complex data-processing capabilities while remaining actively involved in architecture, design and implementation.

You will help define how data products are transformed, orchestrated, governed and operated at scale, with a strong focus on reusability, automation, performance, reliability and engineering standards.

You Will

• Own the technical design and architecture of scalable data-processing and transformation solutions on GCP.
• Design and build production-grade batch, incremental and streaming data pipelines.
• Develop and optimise complex transformations using BigQuery, SQL and Dataform / dbt.
• Build reusable, metadata-driven and configuration-driven frameworks that reduce bespoke pipeline development.
• Design orchestration patterns using Cloud Composer / Apache Airflow, including dynamic workflows, dependencies, retries and recovery.
• Develop distributed and streaming processing solutions using Dataflow / Apache Beam and Dataproc / Spark.
• Define standards for data modelling, schema design, schema evolution and data contracts.
• Design and evolve modern analytical storage patterns, including Apache Iceberg.
• Establish engineering patterns for data quality, reconciliation, metadata, lineage and observability.
• Optimise large-scale workloads for performance, scalability, latency and GCP cost.
• Define reusable engineering standards for testing, versioning, CI/CD and production readiness.
• Lead technical design and code reviews and guide engineers on complex implementation decisions.
• Identify systemic technical debt and drive improvements to shared platform capabilities.
• Work closely with Data Engineering, Platform Engineering, DevOps, Architecture, Governance and Product teams.
• Take technical ownership from solution design through production deployment and operational stability.

You Must Have

• 6+ years of hands-on experience designing and building enterprise-scale data platforms and data products.
• Strong production experience with Google Cloud Platform (GCP).
• Deep hands-on experience with BigQuery, including data modelling, partitioning, clustering, query optimisation and cost optimisation.
• Advanced SQL skills for complex transformation and analytical workloads.
• Strong programming experience with Python, including modular development, testing and automation.
• Strong experience with Dataform and/or dbt.
• Hands-on experience with Cloud Composer / Apache Airflow and complex DAG/workflow design.
• Hands-on experience with Google Cloud Dataflow / Apache Beam.
• Experience with Dataproc and Spark / PySpark for distributed processing.
• Experience with Apache Iceberg or modern lakehouse table formats.
• Strong understanding of batch, incremental, idempotent and streaming processing patterns.
• Experience designing reusable data-processing frameworks rather than only individual pipelines.
• Experience with metadata-driven and configuration-driven processing.
• Strong data-modelling skills, including dimensional, normalized and denormalized models.
• Experience defining and implementing data contracts and schema-management patterns.
• Strong understanding of data quality, metadata, lineage and observability.
• Experience with Git, automated testing and CI/CD for data workloads.
• Experience designing solutions for reliability, recoverability and production operations.
• Ability to independently make architectural decisions and communicate technical trade-offs.

Technical Skills

Cloud & Storage

• Google Cloud Platform
• BigQuery
• Google Cloud Storage
• Apache Iceberg

Transformation & Orchestration

• Advanced SQL
• Dataform / dbt
• Cloud Composer / Apache Airflow

Distributed & Streaming Processing

• Dataflow
• Apache Beam
• Dataproc
• Spark / PySpark

Programming & Engineering

• Python
• Git
• CI/CD
• Automated testing
• Monitoring and observability

Data Platform Capabilities

• Batch and incremental processing
• Streaming and event processing
• Data modelling
• Data contracts and schema management
• Metadata-driven processing
• Configuration-driven processing
• Data quality and reconciliation
• Metadata and lineage
• Performance and cost optimisation
• Reusable platform frameworks

Good to Have

• Practical experience with Data Product / Data Mesh principles.
• Experience developing self-service or internal data-platform capabilities.
• Experience with Change Data Capture and event-driven architectures.
• Exposure to semantic and metric modelling.
• Experience with Terraform / Infrastructure as Code.
• Experience working with multi-domain or multi-market enterprise data platforms.

Strong Interpersonal Skills

• Strong technical ownership and ability to drive complex initiatives from concept to production.
• Ability to lead technical discussions and communicate architectural decisions clearly.
• Strong problem-solving and production troubleshooting skills.
• Ability to mentor engineers and improve engineering practices across a team.
• Comfortable collaborating with engineering, architecture, product and business stakeholders.
• Ability to work effectively within distributed and multicultural teams.