Data Engineer

Key Skills: GCP, SQL, Pyspark, Bigquery, Data Engineer, Python, ETL

Roles and Responsibilities:

• Develop and optimize data processing jobs using PySpark on GCP.
• Build and maintain batch and streaming pipelines using Dataflow and Dataproc.
• Load and manage datasets in BigQuery, ensuring data quality and correctness.
• Orchestrate pipeline workflows using Cloud Composer (Airflow).
• Manage data storage and movement using Cloud Storage and SQL-based transformations.

Skills Required:

• 6-9 years of relevant data engineering experience
• Strong hands-on experience with PySpark and Python.
• Strong experience with Google BigQuery.
• Hands-on experience with Google Cloud Dataflow and Dataproc.
• Experience with Cloud Composer (Airflow) for workflow orchestration.
• Strong knowledge of Google Cloud Storage.
• Strong SQL skills with experience in data transformation and optimization.
• Experience in developing and maintaining scalable data engineering pipelines.

Good to Have

• Experience in ETL/ELT pipeline development.
• Experience working on American Express (AMEX) projects or financial services domain.

Education:

• Bachelor's Degree in Computer Science or related field
• Master's Degree in Computer Science or related field