Data Engineer – Azure & GCP

Job Title - Data Engineer

Years of Experience - 5 to 8 Years Location - Bangalore

Notice Period - Immediate to 30 Days About the Role We are looking for a skilled Data Engineer with hands-on experience in building and managing scalable data pipelines across cloud platforms. The ideal candidate should have strong expertise in ETL frameworks and exposure to multiple cloud environments.

Key Responsibilities • Design, develop, and maintain scalable end-to-end ETL/ELT data pipelines (batch and near real-time) across AWS, Azure, and GCP environments (AWS preferred). • Implement robust data ingestion patterns including CDC, API-based, file-based, and streaming integrations ensuring data quality, reliability, and performance. • Develop and maintain cloud-native data solutions using MySQL/SQL Server, Azure Data Factory, MFTS, GCP BigQuery, and GCS (preferred: AWS Glue, Redshift, EC2, RDS). • Build and manage workflow orchestration using Apache Airflow (or equivalent tools) for scheduling, monitoring, and dependency management. • Work on data ingestion, transformation, and integration across multiple enterprise systems and platforms. • Develop CI/CD pipelines for data infrastructure and ETL code using Git-based workflows and Infrastructure-as-Code (IaC) best practices. • Collaborate with cross-functional teams to understand data requirements and deliver scalable and efficient data solutions. • Work closely with security and compliance teams to implement encryption, IAM least privilege access, network segmentation (VPN/VPC), and governance controls. • Build monitoring, alerting, and auditing mechanisms for data pipelines including latency tracking, failure monitoring, retries, DLQ, and data quality metrics. • Optimize data storage, partitioning, and query performance for analytics workloads in BigQuery and Redshift environments. • Troubleshoot production issues, perform root-cause analysis, and implement corrective and preventive actions. • Produce and maintain technical documentation including architecture diagrams, runbooks, and data contracts while supporting onboarding and mentoring activities. • Provide ongoing sustainment and operational support for production data platforms and pipelines.

Required Qualifications • 5–8 years of total experience in data engineering with at least 3–5 years specifically building cloud-native data pipelines • Strong experience with Azure Cloud services, especially Azure Data Factory (ADF), data ingestion, and transformation workflows • Hands-on expertise with GCP services such as BigQuery, Google Cloud Storage (GCS), and cloud-native analytics solutions • Proficiency in Apache Airflow for workflow orchestration, scheduling, monitoring, and dependency management • Experience implementing batch and near real-time data pipelines using CDC, API, file based, and streaming ingestion patterns • Strong understanding of data engineering concepts, data modeling, pipeline optimization, and performance tuning • Experience with CI/CD pipelines, Git-based workflows, and Infrastructure-as-Code (IaC) practices • Proficiency in SQL and scripting languages such as Python for data processing and automation • Experience with monitoring, troubleshooting, and production support for enterprise data platforms

Preferred Qualifications • Exposure to AWS services such as Glue, Redshift, EC2, and RDS • Experience working in multi-cloud enterprise environments (Azure, GCP, AWS) • Knowledge of data governance, IAM, encryption, and security best practices • Experience handling large-scale distributed and analytics data systems • Strong problem-solving skills and experience working in Agile environments