GCP Data Engineer

Job Description

• Strong hands-on GCP Data Engineering experience across scalable cloud data platforms.


• Expertise in Google Cloud Storage, BigQuery, Dataflow, Dataproc, Pub/Sub, Cloud Composer, Cloud Functions, Cloud Monitoring, IAM and Secret Manager.


• Strong experience in batch and real-time streaming using Pub/Sub, Dataflow / Apache Beam, Spark Structured Streaming or Kafka.


• Hands-on coding proficiency in Python, PySpark and SQL, including debugging and performance optimization.


• Experience building high-volume, low-latency ingestion and transformation pipelines with retry, idempotency, deduplication and dead-letter handling.


• Strong experience in Databricks, Delta Lake, Structured Streaming, Auto Loader and Spark performance tuning.


• Experience handling TB/PB-scale datasets, partitioning, clustering, file compaction, schema evolution and data lifecycle management.


• Strong understanding of Data Lake, Lakehouse and modern cloud data-platform architectures.


• Practical data-modeling experience for analytics, reporting and downstream consumption.


• Experience implementing CI/CD using Cloud Build, GitHub Actions, Jenkins or equivalent, with Terraform for Infrastructure as Code.


• Strong troubleshooting, observability, production support and cloud cost-optimization skills.


• Excellent client communication, stakeholder management and problem-solving skills.

Roles Responsibilities

• Responsibilities and JD in brief along with additional criteria to be considered:


• Design and develop scalable GCP Data Engineering solutions and cloud-native data platforms.


• Build batch and real-time pipelines using BigQuery, Cloud Storage, Dataflow, Pub/Sub, Dataproc, Python, PySpark and Databricks.


• Develop and optimize Databricks, Delta Lake, Spark and Structured Streaming workloads for large-scale processing.


• Implement high-throughput streaming solutions using Pub/Sub and Dataflow, including windowing, watermarking, late-arriving data, duplicate handling and dead-letter queues.


• Handle high-volume TB/PB-scale data with focus on performance, scalability, reliability and cost optimization.


• Develop complex SQL transformations, data models and analytics-ready BigQuery datasets using partitioning and clustering.


• Orchestrate pipelines using Cloud Composer / Airflow and implement CI/CD and Terraform-based Infrastructure as Code.


• Ensure data quality, governance, security, monitoring, alerting and production support.


• Collaborate with architects, stakeholders and cross-functional teams in an Agile delivery model.


• Troubleshoot and optimize end-to-end GCP, Databricks and streaming data pipelines.

Technical Requirement

• 1. GCP BigQuery, Cloud Storage, Dataflow, Pub/Sub, Dataproc, batch and real-time streaming, Databricks , Pyspark
  2. Python, strong SQL
  Databricks with Delta Lake and Structured Streaming
  
  Responsibilities and JD in brief along with additional criteria to be considered:


• Design and develop scalable GCP Data Engineering solutions and cloud-native data platforms.


• Build batch and real-time pipelines using BigQuery, Cloud Storage, Dataflow, Pub/Sub, Dataproc, Python, PySpark and Databricks.


• Develop and optimize Databricks, Delta Lake, Spark and Structured Streaming workloads for large-scale processing.


• Implement high-throughput streaming solutions using Pub/Sub and Dataflow, including windowing, watermarking, late-arriving data, duplicate handling and dead-letter queues.


• Handle high-volume TB/PB-scale data with focus on performance, scalability, reliability and cost optimization.


• Develop complex SQL transformations, data models and analytics-ready BigQuery datasets using partitioning and clustering.


• Orchestrate pipelines using Cloud Composer / Airflow and implement CI/CD and Terraform-based Infrastructure as Code.


• Ensure data quality, governance, security, monitoring, alerting and production support.


• Collaborate with architects, stakeholders and cross-functional teams in an Agile delivery model.


• Troubleshoot and optimize end-to-end GCP, Databricks and streaming data pipelines.

Additional Responsibility

• Infosys is a global leader in next-generation digital services and consulting. Within the Data Analytics unit, you will work on cutting-edge data engineering and analytics initiatives for global clients across industries. The role offers opportunities to work with cloud-native data platforms, modern analytics ecosystems, AI-driven solutions, and large-scale enterprise data modernization programs. Employees benefit from continuous learning programs, certifications, and career growth opportunities within Infosys' data and analytics practice

Educational Requirement

• Master Of Technology,Bachelor Of Science,Bachelor of Engineering,Bachelor Of Technology

Preferred Skills

• Technology->Cloud Platform->GCP Database

Service Line

• Data Analytics Unit