SENIOR DATA ARCHITECT - Databricks

Job Description ? Senior Databricks Architect 

Position 

Senior Databricks Architect  

Location 

Offshore ? Bangalore 

Domain 

Retail / Grocery 

Experience 

10+ years overall IT experience, with 5+ years in Databricks / Cloud Data Engineering / Data Architecture 

Role Overview 

We are looking for an experienced?Senior Databricks Architect?to lead the architecture and technical execution of a large-scale?Talend-to-Databricks migration?for a leading Retail/Grocery client. 

The architect will be responsible for designing the target?Databricks Lakehouse architecture, defining migration patterns, guiding the data engineering team, and ensuring successful modernization of existing Talend-based ETL/ELT workloads onto Databricks. 

The role requires strong hands-on expertise in?Databricks, Apache Spark, SQL, cloud data platforms, data architecture, ETL modernization, and migration strategy, along with a good understanding of?Retail/Grocery business processes and data domains. 

Key Responsibilities 

Architecture & Solution Design 

• Lead the end-to-end architecture for migration of Talend ETL/ELT workloads to Databricks. 

• Define the target-state?Databricks Lakehouse architecture, including data ingestion, transformation, storage, orchestration, governance, and consumption layers. 

• Design scalable, secure, highly available, and cost-optimized data solutions. 

• Define architecture standards, design patterns, coding standards, and best practices for Databricks development. 

• Establish migration frameworks and reusable patterns for converting Talend jobs into Databricks/Spark-based pipelines. 

• Evaluate existing Talend jobs and determine appropriate migration strategies, including re-platforming, re-engineering, consolidation, or retirement. 

• Provide technical leadership for complex data pipelines and integration scenarios. 

Talend to Databricks Migration 

• Analyze existing Talend workflows, jobs, mappings, dependencies, schedules, and data transformations. 

• Develop migration strategies for batch and incremental data processing workloads. 

• Translate Talend transformations and business rules into?PySpark/SQL/Databricks?implementations. 

• Identify opportunities to simplify and optimize legacy ETL processes during migration. 

• Define data reconciliation and validation strategies to ensure parity between Talend and Databricks outputs. 

• Establish migration sequencing based on business criticality, dependencies, complexity, and risk. 

• Support migration of high-volume and business-critical data pipelines. 

Databricks & Data Engineering 

• Design and implement solutions using?Databricks, Apache Spark, Delta Lake, PySpark, and SQL. 

• Design?Medallion Architecture (Bronze, Silver, Gold)?and appropriate data processing patterns. 

• Implement incremental processing, CDC, SCD Type 1/Type 2, data quality, error handling, and audit frameworks. 

• Optimize Spark workloads, Delta tables, SQL queries, partitioning, clustering, and job execution. 

• Leverage?Delta Live Tables / Lakeflow capabilities, Workflows, Unity Catalog, and other Databricks platform services where appropriate. 

• Design data pipelines integrating structured and semi-structured data from databases, files, APIs, and enterprise applications. 

• Establish monitoring, logging, alerting, operational support, and performance-management patterns. 

Cloud & Integration 

• Strong experience with at least one major cloud platform, preferably?Azure or AWS. 

• Design integration between Databricks and cloud storage, databases, messaging platforms, APIs, and enterprise applications. 

• Experience with technologies such as?ADLS/S3, Azure Data Factory, Kafka, REST APIs, relational databases, and cloud-native services?is highly desirable. 

• Define secure connectivity, networking, secrets management, and access-control patterns. 

Retail / Grocery Domain 

• Work closely with business and technology stakeholders to understand Retail/Grocery data requirements. 

• Experience with retail data domains such as: 

• Product / Item 

• Store / Location 

• Customer / Loyalty 

• Sales / Transactions 

• Pricing 

• Promotions 

• Inventory 

• Supply Chain 

• Vendors / Suppliers 

• Orders 

• Forecasting 

• Merchandising 

• Understand retail-specific challenges such as high-volume transaction processing, store-level data, product hierarchies, promotions, pricing, inventory movements, and customer analytics.