Data Engineering Modernization Architect

Job Title: Data Engineering Modernization Architect

Location: Chennai

Work Mode: Work from office

Interview F2F

Job Description:

12+ years of overall experience in data engineering, data platforms, analytics, or enterprise data architecture, with 5+ years in data platform modernization/cloud data engineering and strong hands-on expertise in Databricks, Microsoft Fabric and Azure Data Factory.

Role Overview

We are looking for an experienced Data Engineering Modernization Architect to lead the transformation of legacy data platforms, ETL pipelines, and data warehouses into modern, scalable, cloud-native data platforms.

The architect will define the target-state data architecture, modernization strategy, migration roadmap, engineering standards, and reusable frameworks. The role requires a strong combination of architecture, hands-on engineering, cloud technologies, data modelling, migration, performance optimization, and automation.

The ideal candidate should have successfully delivered at least two enterprise-scale data engineering modernization projects, involving migration from legacy ETL/data warehouse technologies to platforms such as Databricks and Microsoft Fabric, One Lake with Azure Data Factory or equivalent orchestration.

Key Responsibilities

Lead modernization of legacy data platforms, ETL/ELT processes and data warehouses into modern cloud-based architectures. Assess existing data platforms, pipelines, databases, ETL jobs, data models and workloads. Define As-Is, To-Be and transition architectures, Develop modernization roadmaps, Identify opportunities to simplify and consolidate legacy data pipelines, Define reusable modernization patterns and frameworks.

Key Skills Requirements

Strong hands-on and architectural expertise in Databricks Lakehouse architecture, PySpark, Delta Lake, Delta Live Tables / Lakeflow capabilities, Unity Catalog, Workflows / Jobs, Databricks SQL, Change Data Capture, Schema evolution, Databricks security and governance, Azure Data Factory ADF pipelines, Copy activities, Mapping Data Flows, Parameters and variables, Metadata-driven pipelines, Dynamic pipelines, Integration Runtime, Self-hosted Integration Runtime, Trigger mechanisms, Error handling, Retry frameworks, Dependency management, Pipeline monitoring, CI/CD

The candidate should be strong in Enterprise Data Architecture, Modern Data Lakehouse, Data Warehouse, Data Lake, Lakehouse architecture, Medallion architecture, Dimensional modelling, Star/Snowflake schemas, Data Vault, Data mesh concepts, Batch and streaming architecture, CDC architecture, Event-driven data pipelines, Master/reference data, Data governance, Data lineage, Data quality, Metadata management

Strong experience in developing reusable frameworks for Metadata-driven ingestion, Dynamic pipeline generation, Automated schema discovery, Automated table creation, Incremental load framework, CDC framework, Data quality framework, Pipeline monitoring, Exception handling, Automated reconciliation, Data validation, Deployment automation, Environment provisioning

Strong experience in Legacy ETL modernization, Data warehouse migration, Data lake migration, ETL ELT transformation, On-premises Azure migration, Legacy database Lakehouse migration, Batch near-real-time modernization, Stored procedure modernization, Legacy ETL code conversion.