Data Engineer

Experience

410 Years

Role

Data Engineer – GCP / AWS

Job Summary

We are looking for an experienced Data Engineer with strong hands-on expertise in Python and SQL and experience working with GCP and/or AWS cloud platforms.

The ideal candidate should have strong experience in designing, developing, and optimizing data pipelines, ETL/ELT processes, data transformation, and cloud-based data solutions.

Mandatory Skills

• Strong hands-on experience in Python
• Strong proficiency in SQL
• Experience in Data Engineering / ETL / ELT
• Hands-on experience in GCP or AWS
• Strong understanding of data pipelines and data processing
• Experience working with large datasets
• Good understanding of data modelling and database concepts

GCP Skills – Preferred

Experience with one or more of the following:

• Google BigQuery
• Google Cloud Storage (GCS)
• Dataflow
• Dataproc
• Pub/Sub
• Cloud Functions
• Composer / Airflow
• Cloud Run

AWS Skills – Preferred

Experience with one or more of the following:

• AWS S3
• AWS Glue
• EMR
• Lambda
• Redshift
• Athena
• Step Functions
• MWAA / Airflow

Data Engineering Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines.
• Develop data transformation and processing solutions using Python and SQL.
• Build and optimize batch and/or real-time data processing pipelines.
• Develop data ingestion pipelines from multiple source systems.
• Perform data cleansing, transformation, validation, and aggregation.
• Work with cloud data platforms such as GCP and/or AWS.
• Optimize SQL queries and data processing jobs for performance.
• Work with data warehouses, data lakes, and cloud storage platforms.
• Monitor pipeline performance and troubleshoot production issues.
• Implement data quality and validation checks.
• Collaborate with business, application, and analytics teams to understand data requirements.
• Follow best practices for security, scalability, reliability, and maintainability.

Good to Have

• PySpark / Apache Spark
• Airflow / Cloud Composer
• Kafka / Pub/Sub
• BigQuery / Snowflake / Redshift
• Databricks
• Terraform
• CI/CD
• Docker / Kubernetes
• Git
• Data modelling
• Cloud migration experience