Software Engineer- Scala Developer

We are seeking a Senior Data Engineer to design, build, and optimize our next-generation data architecture on Google Cloud Platform. You will be responsible for the "heavy lifting" of dataconstructing robust ETL pipelines, managing large-scale Spark clusters, and ensuring our BigQuery warehouse is performant and scalable.
Beyond core engineering, you will help bridge the gap between traditional data processing and emerging GenAI capabilities, exploring how Vertex AI can be integrated into our production pipelines.

Technical Stack & Responsibilities
Core Data Engineering
ETL/ELT Architecture: Design and implement complex data pipelines from disparate sources into our centralized data lake/warehouse.
Big Data Processing: Leverage Apache Spark (via Dataproc) and Python/Scala to handle petabyte-scale transformations.
Advanced SQL: Write and optimize complex queries for data modeling, ensuring high performance in BigQuery.
Stream & Batch: Develop real-time processing solutions using Pub/Sub and Dataflow (Apache Beam).
GCP Infrastructure
Orchestration: Manage workflow automation using Cloud Composer (Airflow).
Storage & Databases: Architect solutions across Cloud Storage, Cloud SQL, and Spanner, choosing the right tool for the specific latency and consistency requirements.
Environment Management: Ensure security, logging, and monitoring are integrated into every pipeline.

Must to have :
Programming: Mastery of Python or Scala.
Spark Expertise: Proven experience tuning and debugging Spark jobs at scale.
GCP Professional: Deep hands-on experience with the full Google data suite (BigQuery, Dataflow, Dataproc, Pub/Sub).
SQL Mastery: Ability to perform complex window functions, CTEs, and query optimization.

Good to have :

AI/ML Integration: Familiarity with Vertex AI and the GCP AI Platform.
GenAI Literacy: A basic understanding of LLMs, prompt engineering, and how data pipelines feed into RAG (Retrieval-Augmented Generation) architectures.
Infrastructure as Code: Experience with Terraform for managing GCP resources.

Note on GenAI: While your primary focus is data stability and performance, this role offers the unique opportunity to collaborate with our Data Science teams to operationalize Generative AI models.

• Airflow
• Apache Spark
• Python
• Cloud architecture
• Scala

That same commitment to fairness extends to how we use technology. To support our recruitment team, AI tools may be used to help assess applications though they never replace human judgement. All hiring decisions remain entirely in the hands of our recruitment professionals.