Gcp Data Engineer

Role & responsibilities

Section III - Job Requirements and Responsibilities

Section Details / Example Content

Job

Requirements* We are seeking an experienced GCP Data Engineer to design, develop, and maintain scalable data solutions on the Google Cloud Platform (GCP). The ideal candidate will have strong expertise in building modern data pipelines, data warehousing, big data processing, and cloud-native analytics solutions. The role requires close collaboration with business stakeholders, data architects, data scientists, and application teams to deliver reliable and high-performance data platforms.

Key

Responsibilities* • Design, develop, and optimize scalable data pipelines using GCP services. • Build and maintain batch and real-time data ingestion and processing frameworks. • Develop enterprise-grade data warehousing solutions using BigQuery. • Implement ETL/ELT processes for structured and unstructured data. • Integrate data from multiple sources including databases, APIs, files, and streaming platforms. • Ensure data quality, governance, security, and compliance across data platforms.

• Optimize query performance, storage, and processing costs within GCP. • Automate deployment and operational processes using CI/CD and Infrastructure as Code (IaC). • Monitor data pipelines and troubleshoot production issues. • Collaborate with Data Architects, Business Analysts, and Data Scientists to translate business requirements into technical solutions. • Implement data models, metadata management, and data lineage best practices. • Support migration of on-premise or multi-cloud data platforms to GCP. • Mentor junior engineers and participate in technical design reviews.

Required

Technical Skills • Google Cloud Storage (GCS) • BigQuery • Cloud Composer (Apache Airflow) • Dataflow • Dataproc • Pub/Sub • Cloud Functions • Cloud Run • Cloud SQL • BigLake (preferred) • Vertex AI integration (good to have) • Python • SQL (Advanced) • PySpark

Section Details / Example Content

Domain CMI

Soft Skills - Excellent communication - Team collaboration - Documentation and knowledge sharing

Education

Requirements Bachelor's/Masters in Computer Science or equivalent (Preferred)

Certifications GCP Cloud Data Engineer (Preferred)

Section V - Sample Questions for Training Model

Technical Question Expected Answer What to Evaluate 1. Describe a GCP data pipeline you designed that handled both batch and real-time data processing requirements Candidate should explain: • Business requirement involving multiple data sources and different latency needs. • Batch ingestion using GCS, Dataproc, Composer, or BigQuery. • Real-time ingestion using Pub/Sub and Dataflow. • Data storage strategy (Data Lake in GCS and Data Warehouse in BigQuery). • Monitoring and alerting implementation. • Challenges such as late-arriving data, schema drift, duplicate events, and scalability. End-to-end understanding of data pipeline built and service integration lifecycle and readiness of output data to be loaded. 2. How would you optimize a BigQuery query that is taking several minutes to run on multi-terabyte datasets? Candidate should discuss: • Reviewing execution plan and query stages. • Using partitioned tables. • Applying clustering on frequently filtered columns.