Python, Aws Services, Pyspark Developer

Job Title: Python, AWS & PySpark Developer

Position: Python, AWS & PySpark Developer
Experience: 5+ Years
Technology: Python, AWS, PySpark, Big Data Engineering
Work Location: Offshore
Employment Type: Full-Time

Position Description

CGI is seeking an experienced Python, AWS, and PySpark Developer to design, develop, optimize, and maintain scalable data engineering solutions on cloud platforms. The ideal candidate will have strong hands-on experience in Python programming, distributed data processing using PySpark, and building data pipelines using AWS services.

The candidate will be responsible for developing robust ETL/ELT pipelines, processing large volumes of structured and unstructured data, implementing data transformations, and ensuring data quality, reliability, and performance. The role requires close collaboration with Data Engineers, Data Architects, Business Analysts, and cross-functional teams to deliver enterprise-grade data solutions.

Your Future Duties and Responsibilities

• Design, develop, test, and maintain scalable data processing applications using Python and PySpark.
• Build and maintain ETL/ELT pipelines for data ingestion, transformation, validation, and integration.
• Develop distributed data processing solutions using Apache Spark and PySpark to process large datasets efficiently.
• Design, implement, and manage cloud-based data pipelines using AWS services.
• Work with AWS services such as Amazon S3, AWS Glue, AWS Lambda, Amazon EMR, Amazon Redshift, AWS Step Functions, and Amazon CloudWatch, as applicable to the project.
• Develop data extraction, transformation, and loading processes from multiple data sources into data lakes and data warehouses.
• Implement data transformation logic, business rules, aggregations, joins, and data cleansing using Python and PySpark.
• Optimize Spark jobs through partitioning, caching, join strategies, and resource tuning to improve performance and reduce processing time.
• Develop and maintain reusable Python modules, libraries, utilities, and automation scripts.
• Implement data validation, reconciliation, error handling, logging, and exception management within data pipelines.
• Monitor pipeline execution, troubleshoot failures, and resolve performance bottlenecks in production environments.
• Work with relational databases and data platforms using SQL for data extraction, analysis, validation, and reconciliation.
• Implement secure data processing practices using appropriate AWS IAM roles, policies, encryption, and access controls.
• Support CI/CD pipelines, automated deployments, version control, and release management using tools such as Git and AWS-native or other DevOps solutions.
• Collaborate with Data Architects, Data Engineers, DevOps teams, and business stakeholders to translate requirements into technical solutions.
• Participate in code reviews, unit testing, integration testing, production deployments, and technical documentation.
• Perform root cause analysis and implement corrective actions for data pipeline failures and production incidents.
• Ensure adherence to coding standards, data governance requirements, security policies, and enterprise development practices.

Required Qualifications to Be Successful in This Role

• Minimum 5+ years of hands-on experience in software development, data engineering, or cloud-based data processing.
• Strong programming skills in Python, including object-oriented programming, functions, exception handling, modules, and reusable code development.
• Strong hands-on experience with PySpark and Apache Spark for distributed data processing.
• Practical experience developing and supporting ETL/ELT pipelines and large-scale data processing workflows.
• Hands-on experience with AWS services used for data engineering, particularly Amazon S3, AWS Glue, Amazon EMR, AWS Lambda, or Amazon Redshift.
• Strong SQL skills, including complex queries, joins, aggregations, subqueries, and performance optimization.
• Experience working with structured, semi-structured, and unstructured data formats such as CSV, JSON, and Parquet.
• Good understanding of data lake and data warehouse architectures.
• Experience with data quality checks, data reconciliation, error handling, and pipeline monitoring.
• Strong troubleshooting, debugging, analytical, and problem-solving skills.
• Experience working with Git or other version control systems.
• Understanding of software development lifecycle (SDLC), Agile methodologies, testing practices, and production support.
• Strong communication and collaboration skills, with the ability to work effectively in distributed teams.

Desired Qualifications / Preferred Skills

• Experience with AWS Glue Data Catalog, Glue Crawlers, and Glue workflows.
• Knowledge of AWS Step Functions, Amazon EventBridge, Amazon CloudWatch, and AWS CloudFormation.
• Experience with Amazon Athena, Amazon Redshift, and data lakehouse architectures.
• Familiarity with Apache Airflow, AWS Managed Workflows for Apache Airflow (MWAA), or other workflow orchestration tools.
• Experience with data formats and table technologies such as Avro, ORC, Delta Lake, or Apache Iceberg.
• Knowledge of data partitioning, bucketing, schema evolution, and incremental data processing.
• Experience implementing CI/CD pipelines using Jenkins, GitLab CI/CD, AWS CodePipeline, or similar tools.
• Familiarity with Docker, infrastructure as code, and cloud deployment automation.
• Understanding of data security, encryption, IAM policies, and compliance requirements.
• Experience with monitoring, alerting, incident management, and production support in enterprise environments.
• Exposure to Control-M or other enterprise scheduling tools is an added advantage.

Primary Skills

• Python Programming
• PySpark and Apache Spark
• AWS Data Engineering Services
• ETL/ELT Pipeline Development
• SQL and Data Processing
• Data Lake and Data Warehouse Solutions
• Performance Tuning and Optimization

Secondary Skills

• AWS Glue, Amazon S3, Amazon EMR
• AWS Lambda, Amazon Redshift, Amazon Athena
• Apache Airflow / AWS MWAA
• Git, CI/CD, and DevOps
• Data Quality and Reconciliation
• Production Monitoring and Troubleshooting