Senior Software Engineer- Azure Data Developer
Coralai · Chennai · onsite · Posted 68d ago
Undisclosed4–7 yrsfulltimeSourced from
About a minute. You'll make an account on the way through.
About this role
Position Description:
Founded in 1976, CGI is among the largest independent IT and business consulting services firms in the world. With 94,000 consultants and professionals across the globe, CGI delivers an end-to-end portfolio of capabilities, from strategic IT and business consulting to systems integration, managed IT and business process services and intellectual property solutions. CGI works with clients through a local relationship model complemented by a global delivery network that helps clients digitally transform their organizations and accelerate results. CGI Fiscal 2024 reported revenue is CA$14.68 billion and CGI shares are listed on the TSX (GIB.A) and the NYSE (GIB). Learn more at cgi.com.
Job Title: ETL Developer
Position: Software Engineer
Experience: 4- 7 Years
Category: Software Development/ Engineering
Shift: 1-10PM (Hybrid)
Main location: Bangalore / Chennai
Position ID: J0726-1184
Employment Type: Full Time
Education Qualification: Bachelor's degree in Computer Science or related field or higher with minimum 3 years of relevant experience.
Position Description: We are looking for an experienced .Net Developer to join our team. The ideal candidate should be passionate about coding and developing scalable and high-performance applications. The successful candidate will be responsible for designing, developing, optimizing, and supporting enterprise-grade ETL and ELT pipelines primarily using Azure Databricks, PySpark, SQL, and related Microsoft Azure services. This is a hands-on engineering role focused on building scalable, reliable, and high-performing data pipelines across development, testing, and production environments. The engineer will work closely with data architects, business analysts, application teams, quality assurance teams, and production support teams to deliver secure and well-governed data solutions in a regulated financial-services environment.
#LI-MP14
Job Title: ETL Developer
Position: Software Engineer
Experience: 4- 7 Years
Category: Software Development/ Engineering
Shift: 1-10PM (Hybrid)
Main location: Bangalore / Chennai
Position ID: J0726-1184
Employment Type: Full Time
Education Qualification: Bachelor's degree in Computer Science or related field or higher with minimum 3 years of relevant experience.
Position Description: We are looking for an experienced .Net Developer to join our team. The ideal candidate should be passionate about coding and developing scalable and high-performance applications. The successful candidate will be responsible for designing, developing, optimizing, and supporting enterprise-grade ETL and ELT pipelines primarily using Azure Databricks, PySpark, SQL, and related Microsoft Azure services. This is a hands-on engineering role focused on building scalable, reliable, and high-performing data pipelines across development, testing, and production environments. The engineer will work closely with data architects, business analysts, application teams, quality assurance teams, and production support teams to deliver secure and well-governed data solutions in a regulated financial-services environment.
#LI-MP14
Your future duties and responsibilities:
Design, develop, test, deploy, and support ETL and ELT pipelines using Azure Databricks.
Build scalable data transformation workflows using PySpark, Spark SQL, and Databricks notebooks.
Develop reusable Databricks frameworks, utilities, and common pipeline components.
Ingest and process data from relational databases, flat files, APIs, cloud storage, and enterprise applications.
Design and implement batch and near-real-time data pipelines.
Build and manage Databricks jobs, workflows, schedules, dependencies, retries, and alerts.
Develop data cleansing, validation, enrichment, reconciliation, and transformation logic.
Implement data quality controls, audit checks, exception handling, and restart capabilities.
Optimize Spark workloads, SQL queries, cluster configuration, partitioning, caching, and file formats.
Work with Delta Lake to support reliable, scalable, and transactional data processing.
Design and maintain bronze, silver, and gold data layers within a lakehouse architecture.
Support source-to-target mapping, data profiling, data lineage, and technical design activities.
Develop and maintain SQL queries, stored procedures, views, and data reconciliation scripts.
Integrate Databricks pipelines with Azure Data Lake Storage and other Azure services.
Build and maintain automated deployment processes using Azure DevOps CI/CD pipelines.
Promote code, notebooks, jobs, and configuration across development, testing, and production environments.
Monitor pipeline performance and troubleshoot failures, data issues, and production incidents.
Perform root-cause analysis and implement permanent fixes for recurring issues.
Create and maintain technical documentation, data mappings, deployment procedures, and support runbooks.
Ensure solutions comply with CIBC security, data governance, risk management, and regulatory requirements.
Build scalable data transformation workflows using PySpark, Spark SQL, and Databricks notebooks.
Develop reusable Databricks frameworks, utilities, and common pipeline components.
Ingest and process data from relational databases, flat files, APIs, cloud storage, and enterprise applications.
Design and implement batch and near-real-time data pipelines.
Build and manage Databricks jobs, workflows, schedules, dependencies, retries, and alerts.
Develop data cleansing, validation, enrichment, reconciliation, and transformation logic.
Implement data quality controls, audit checks, exception handling, and restart capabilities.
Optimize Spark workloads, SQL queries, cluster configuration, partitioning, caching, and file formats.
Work with Delta Lake to support reliable, scalable, and transactional data processing.
Design and maintain bronze, silver, and gold data layers within a lakehouse architecture.
Support source-to-target mapping, data profiling, data lineage, and technical design activities.
Develop and maintain SQL queries, stored procedures, views, and data reconciliation scripts.
Integrate Databricks pipelines with Azure Data Lake Storage and other Azure services.
Build and maintain automated deployment processes using Azure DevOps CI/CD pipelines.
Promote code, notebooks, jobs, and configuration across development, testing, and production environments.
Monitor pipeline performance and troubleshoot failures, data issues, and production incidents.
Perform root-cause analysis and implement permanent fixes for recurring issues.
Create and maintain technical documentation, data mappings, deployment procedures, and support runbooks.
Ensure solutions comply with CIBC security, data governance, risk management, and regulatory requirements.
Required qualifications to be successful in this role:
Must Have Skill:
-Databricks
-Delta Lake Architecture
- Data Vault 2.0
-Kimball data modeling
-Pyspark Spark SQL
-DQX (Data Quality Framework)
-Azure Devops
-Spark declarative pipeline
-Github Copilot usage/handling
-Production Job troubleshooting debugging
Good To have:
-Power BI knowledge
-Scheduling tools such as autosys
-Azure data factory
CGI is an equal opportunity employer. In addition, CGI is committed to providing accommodation for people with disabilities in accordance with provincial legislation. Please let us know if you require reasonable accommodation due to a disability during any aspect of the recruitment process and we will work with you to address your needs.
-Databricks
-Delta Lake Architecture
- Data Vault 2.0
-Kimball data modeling
-Pyspark Spark SQL
-DQX (Data Quality Framework)
-Azure Devops
-Spark declarative pipeline
-Github Copilot usage/handling
-Production Job troubleshooting debugging
Good To have:
-Power BI knowledge
-Scheduling tools such as autosys
-Azure data factory
CGI is an equal opportunity employer. In addition, CGI is committed to providing accommodation for people with disabilities in accordance with provincial legislation. Please let us know if you require reasonable accommodation due to a disability during any aspect of the recruitment process and we will work with you to address your needs.
Skills:
- Communication
- Database
- Banking
- Databricks
- SQLite
Sourced from naukri
Skills
dbmsC SharpcProgrammingHTMLC plus plusData Structuressoftware testingPythonJavascriptCSSMySQLJQuerySQLSpring BootAlgorithmsJavaSoftware DevelopmentSoftware EngineeringSQL ServerOOPS