Senior Software engineer- Azure Databricks

Founded in 1976, CGI is among the largest independent IT and business consulting services firms in the world. With 94,000 consultants and professionals across the globe, CGI delivers an end-to-end portfolio of capabilities, from strategic IT and business consulting to systems integration, managed IT and business process services and intellectual property solutions. CGI works with clients through a local relationship model complemented by a global delivery network that helps clients digitally transform their organizations and accelerate results. CGI Fiscal 2024 reported revenue is CA$14.68 billion and CGI shares are listed on the TSX (GIB.A) and the NYSE (GIB). Learn more at cgi.com.

Job Title: Azure Databricks
Position: Software Engineer
Experience: 6- 9 Years
Category: Software Development/ Engineering
Shift: 1-10PM (Hybrid)
Main location: Bangalore / Chennai
Position ID: J0726-1184
Employment Type: Full Time

Education Qualification: Bachelor's degree in Computer Science or related field or higher with minimum 3 years of relevant experience.

Position Description: We are looking for an experienced .Net Developer to join our team. The ideal candidate should be passionate about coding and developing scalable and high-performance applications. The successful candidate will be responsible for designing, developing, optimizing, and supporting enterprise-grade ETL and ELT pipelines primarily using Azure Databricks, PySpark, SQL, and related Microsoft Azure services. This is a hands-on engineering role focused on building scalable, reliable, and high-performing data pipelines across development, testing, and production environments. The engineer will work closely with data architects, business analysts, application teams, quality assurance teams, and production support teams to deliver secure and well-governed data solutions in a regulated financial-services environment.

#LI-BK7

Your future duties and responsibilities

• Design, develop, test, deploy, and support scalable ETL/ELT pipelines using Azure Databricks, PySpark, Spark SQL, Delta Lake, and Azure Data Factory.
• Implement end-to-end Lakehouse and Medallion Architecture patterns across Bronze, Silver, and Gold data layers, with strong lineage, replayability, and operational controls.
• Develop advanced ingestion patterns including full loads, incremental loads, CDC, Auto Loader, schema evolution, watermarking, late-arriving data handling, and restartable/idempotent processing.
• Build and maintain Spark Declarative Pipelines / declarative Databricks pipeline patterns with dependency management, data quality expectations, observability, and production-grade orchestration.
• Design and implement Data Vault 2.0 structures including Hubs, Links, Satellites, PIT tables, hash keys, hash diffs, Raw Vault and Business Vault patterns where appropriate.
• Design Kimball dimensional models including facts, dimensions, conformed dimensions, slowly changing dimensions, role-playing dimensions, and bridge tables for downstream analytical consumption.
• Develop reusable PySpark and Spark SQL frameworks for cleansing, standardization, enrichment, deduplication, reconciliation, and business-rule processing.
• Implement enterprise data quality using DQX, including completeness, uniqueness, validity, consistency, referential integrity, auditability, and quarantine/reject patterns.
• Create and maintain audit logging, source-to-target reconciliation, control totals, record-count balancing, SLA monitoring, and production support dashboards/alerts.
• Optimize Spark and Delta Lake workloads using partitioning, clustering, caching, broadcast joins, AQE, skew handling, file compaction, OPTIMIZE/VACUUM, and efficient query design.
• Build and support Azure DevOps CI/CD pipelines for Databricks code, notebooks, declarative pipelines, ADF assets, configuration, and environment promotion.
• Use GitHub Copilot responsibly for development acceleration, code review support, test generation, debugging, and productivity while applying appropriate validation and secure coding practices.
• Troubleshoot and debug failed or slow production jobs, perform root-cause analysis, restore service within SLA, and implement permanent corrective actions.
• Work closely with architects, business analysts, QA/QE, Power BI developers, and application teams to ensure data products meet business, quality, performance, security, and reporting requirements.
  

Required qualifications to be successful in this role
Must Have Skill:

-Databricks

-Delta Lake Architecture

- Data Vault 2.0

-Kimball data modeling

-Pyspark & Spark SQL

-DQX (Data Quality Framework)

-Azure Devops

-Spark declarative pipeline

-Github Copilot usage/handling

-Production Job troubleshooting & debugging

Good To have:

-Power BI knowledge

-Scheduling tools such as autosys

-Azure data factory

CGI is an equal opportunity employer. In addition, CGI is committed to providing accommodation for people with disabilities in accordance with provincial legislation. Please let us know if you require reasonable accommodation due to a disability during any aspect of the recruitment process and we will work with you to address your needs.