data engineer

Job Summary: We are seeking a Senior Data Engineer with 5+ years of experience in designing, developing, and maintaining scalable cloud-based data solutions. The ideal candidate should possess strong expertise in SQL, Python, PySpark, and Google Cloud Platform (GCP) services including BigQuery, Cloud Composer (Apache Airflow), Dataproc, and Pub/Sub. The role involves building enterprise-grade data pipelines, implementing scalable ETL/ELT solutions, and collaborating with Product, Engineering, and Business teams to deliver reliable, high-quality data products.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines using Python and PySpark.
• Build and optimize cloud-native data solutions on Google Cloud Platform (GCP).
• Develop optimized SQL queries and transformation logic in BigQuery.
• Design and maintain orchestration workflows using Cloud Composer (Apache Airflow).
• Develop, optimize, and troubleshoot Spark applications running on Dataproc.
• Design event-driven data ingestion pipelines using Google Pub/Sub.
• Ensure data quality through validation, reconciliation, monitoring, and exception handling.
• Develop reusable frameworks and utilities to improve engineering productivity.
• Collaborate with Product Owners, Business SMEs, and Engineering teams to understand requirements and deliver scalable solutions.
• Participate in code reviews and contribute to engineering best practices.
• Create and maintain technical documentation for data pipelines and architecture.

Required Technical Skills

• Python (Advanced)
• SQL (Advanced)
• PySpark
• Google Cloud Platform (GCP)
• BigQuery
• Cloud Composer (Apache Airflow)
• Dataproc
• Pub/Sub
• Cloud Storage
• Data Engineering
• ETL / ELT Development
• Batch & Streaming Data Processing
• Data Validation & Reconciliation
• Pipeline Performance Optimization
• Logging & Monitoring
• Data Warehouse Concepts
• Data Warehouse Architecture
• Star & Snowflake Schema
• Fact & Dimension Modeling
• Slowly Changing Dimensions (SCD)
• Partitioning & Clustering
• Data Lake vs Data Warehouse concepts
• DevOps & Version Control
• Git / GitHub
• CI/CD Concepts
• Docker (Preferred)

Preferred Qualifications

• 5+ years of experience in Data Engineering.
• Strong hands-on experience with SQL, Python, and PySpark.
• Experience building production-grade data pipelines.
• Hands-on experience working with Google Cloud Platform (GCP).