Senior Specialist - Architecture

Role Description

Project

Objective of the project is to implement a solution to retain all data documents needed to satisfy a Retention requirements of all Acts and Regulations governing the Credit Cards business eg ECOA FCRA EFTA TILA etc b GS Privacy Policy and c Any litigation and legal holds The solution should enable retrieval of datadocuments to service Regulatory inquiries Audit requests Litigation requests as well as all account documentation requirements from other external parties related to the business

Role Responsibilities

Design and maintain scalable data pipelines for data ingestion transformation and processing

Build and optimize ETLELT workflows using PySpark Databricks SQL and Airflow

Develop data models and solutions for retention reporting and analytics needs

Ensure data quality integrity and reliability through validation and monitoring frameworks

Integrate data from multiple sources and optimize processing performance

Collaborate with crossfunctional teams to deliver data solutions and resolve data issues

Maintain technical documentation data lineage and mapping artifacts

Support production deployments troubleshooting and continuous improvement through engineering best practices

Location Bengaluru Office Full time work from office

Technical Skills Requirements

1PySpark Spark SQL Handson experience writing productiongrade PySpark code

2Databricks Proven experience working within Databricks environments including cluster configuration and Delta Lake

3Apache Airflow Strong experience building and managing pipelines in Apache Airflow

4SQL Mastery Advanced SQL skills window functions CTEs query optimization indexing strategies

5Data Modeling Solid understanding of modern data warehousing concepts

6Version Control Proficiency with Git and CICD deployment workflows

7AWS Experience in Amazon Web Services AWS cloud infrastructure

8ETLELT Proven track record of designing building and optimizing scalable ETLELT pipelines and data platforms