Data Engineer

Your future duties and responsibilities
We are seeking a Senior Data Engineer to design, build, and optimize our next-generation data architecture on Google Cloud Platform. You will be responsible for the "heavy lifting" of dataconstructing robust ETL pipelines, managing large-scale Spark clusters, and ensuring our BigQuery warehouse is performant and scalable.
Beyond core engineering, you will help bridge the gap between traditional data processing and emerging GenAI capabilities, exploring how Vertex AI can be integrated into our production pipelines.

Technical Stack & Responsibilities
Core Data Engineering
ETL/ELT Architecture: Design and implement complex data pipelines from disparate sources into our centralized data lake/warehouse.
Big Data Processing: Leverage Apache Spark (via Dataproc) and Python/Scala to handle petabyte-scale transformations.
Advanced SQL: Write and optimize complex queries for data modeling, ensuring high performance in BigQuery.
Stream & Batch: Develop real-time processing solutions using Pub/Sub and Dataflow (Apache Beam).
GCP Infrastructure
Orchestration: Manage workflow automation using Cloud Composer (Airflow).
Storage & Databases: Architect solutions across Cloud Storage, Cloud SQL, and Spanner, choosing the right tool for the specific latency and consistency requirements.
Environment Management: Ensure security, logging, and monitoring are integrated into every pipeline.