Gcp Data Engineer

Position Description:As a Data Engineer, you will be responsible for designing, building, and maintaining the robust data architectures and scalable pipelines that power the organizations data-driven decision-making using Google Cloud Platform. You will transform raw, unstructured data into high-quality datasets by implementing efficient ETL/ELT processes, managing cloud-based data warehouses, and ensuring the reliability and security of the entire data infrastructure. Working at the intersection of software engineering and data science, you will collaborate with cross-functional teams to optimize data accessibility and performance, ultimately enabling Data Scientists and Analysts to derive actionable insights from complex information systems. Responsibilities: Design and implement scalable ETL/ELT pipelines using Python and SQL to ingest, transform, and load large-scale datasets into Google Cloud Storage and BigQuery. Orchestrate complex data workflows and DAGs through Astronomer/Airflow to ensure high availability and reliable scheduling of data processes. Utilize Terraform to manage and provision cloud infrastructure as code, ensuring consistent and reproducible environments across the Google Cloud Platform. Build and maintain automated CI/CD pipelines using Tekton/ GCP Cloud Native services to streamline the testing, containerization, and deployment of data services. Deliver AI-ready data products that are specifically structured and optimized with applying metadata on objects for machine learning consumption and advanced data analysis. Optimize BigQuery performance through strategic schema design, partitioning, and clustering to minimize costs and maximize query efficiency. Conduct deep-dive data analysis to validate data integrity and ensure that all outputs meet the rigorous quality standards required for production-level AI systems. Partner with cross-functional teams to translate business requirements into technical data specifications and highly curated data assets.Skills Required:Google Cloud Platform (BigQuery, Vertex AI, Cloud Run, CloudBuild, Secret Manager, Pub/Sub, DataFlow, DataForm, Dataproc, GIT Actions, Artifact Registry, Docker), Strong Python + SQL + Fast API, CI/CD Pipelines, Tekton/Terraform, Astronomer/Airflow Frontend (React, Streamlit etc,,)