Here is an Exciting Opportunity - Pyspark with ADB role - Capgemini

Job Description: We are looking for a skilled PySpark Developer with experience in Azure Databricks (ADB) and Azure Data Factory (ADF) to join our team. The ideal candidate will play a crucial role in designing, developing, and implementing data solutions using PySpark for large-scale data processing and analytics.

Responsibilities:

• Design, develop, and deploy PySpark applications and workflows on Azure Databricks for data transformation, cleansing, and aggregation.
• Implement data pipelines using Azure Data Factory (ADF) to orchestrate ETL/ELT processes across heterogeneous data sources.
• Collaborate with Data Engineers and Data Scientists to integrate and process structured and unstructured data sets into actionable insights.
• Optimize PySpark jobs and data pipelines for performance, scalability, and reliability.
• Conduct regular financial risk assessments to identify potential vulnerabilities in data processing workflows.
• Ensure data quality and integrity throughout all stages of data processing.
• Develop and implement strategies to mitigate financial risks associated with data transformation and aggregation.
• Troubleshoot and debug issues related to data pipelines and processing.
• Ensure compliance with regulatory requirements and industry standards in all data processing activities.
• Implement best practices for data security, compliance, and privacy within Azure environment.
• Document technical specifications, data flows, and solution architecture.

Requirements:

• Bachelors degree in computer science, Engineering, or a related field; Master’s degree preferred.
• Proven experience as a PySpark Developer or similar role with a strong understanding of Apache Spark internals.
• Hands-on experience with Azure Databricks (ADB) and Azure Data Factory (ADF).
• Proficiency in Python programming language and solid understanding of SQL.
• Experience designing and optimizing data pipelines for ETL/ELT processes.
• Familiarity with cloud platforms, preferably Microsoft Azure.
• Excellent problem-solving skills and ability to think critically.
• Strong communication skills with the ability to collaborate effectively in a team environment.
• Experience in Financial, Risk, Compliance, or Banking domains is a plus.
• Experience identifying and mitigating financial risks in data processes.
• Ability to analyse data for potential risk factors and develop strategies to minimize financial risk.
• Ensure all data processes comply with relevant regulatory requirements and industry standards.

Preferred Qualifications:

• Certification in Azure Data Engineering or related field.
• Knowledge of other big data technologies such as Hadoop, Hive, or Kafka.

• Familiarity with machine learning frameworks and techniques.