Data Architect - Databricks AI GenAI

Purpose -

To architect and govern ABHFL's enterprise Data, AI and Generative AI ecosystem, leveraging Databricks, Agent bricks , databricks Apps , AIOps and modern AI capabilities to establish a scalable, secure and governed enterprise data platform.

The role will be a hands-on architecture position, responsible for designing and prototyping modern data and AI solutions, including Lakehouse architecture, Unity Catalog, RAG, LLMs, AI Agents and Agentic AI, while ensuring performance, security, data governance and regulatory compliance.

Role & responsibilities

1. Manage and enhanced a scalable enterprise Lakehouse architecture supporting ABHFL's increasing Data, Analytics, ML and GenAI requirements.

2. Integrate data across lending, customer, CRM, DMS, finance, collections, risk, regulatory and enterprise platforms.

3. Establish consistent enterprise data models and data products while reducing duplication across analytics and reporting solutions.

4. Architect production-grade AI/GenAI solutions covering RAG, LLMs, Document Intelligence and Agentic AI.

5. Establish comprehensive governance through Unity Catalog, lineage, access controls, data quality and lifecycle management.

6. Develop an architecture that provides appropriate performance and scalability while optimizing Databricks/cloud consumption.

7. Establish robust MLOpss/LLMOps frameworks covering development, deployment, evaluation, tracing, monitoring and model/AI lifecycle governance.

8. Convert emerging AI and Databricks capabilities from PoCs into secure and supportable enterprise solutions.

9. Work across Business, Data Engineering, BI, Applications, Cloud, Information Security, Risk, Privacy, Audit and Compliance stakeholders.

10. Ensure architecture supports ABHFL's security, audit, privacy and regulatory requirements.

KRAS

Enterprise Data Architecture

Define scalable enterprise Data/Lakehouse architecture covering structured, semi-structured and unstructured data.

Databricks Platform

Architecture across Databricks Lakehouse, Delta, Unity Catalog, Data Engineering and Data/AI workloads.

Data Engineering

Design reusable ingestion, ETL/ELT, transformation and data-serving frameworks using Python, PySpark and SQL.

AI/BI dashboards and Data Bricks Apps

Define governed integration and serving architecture between Databricks and AI/BI Dashboards

AI/ML

Architect ML lifecycle from data preparation and experimentation through deployment and monitoring.

AIOPS/MLOPS

Establish model lifecycle, CI/CD, versioning, testing, deployment and monitoring standards.

Generative AI

Architect LLM, RAG, embeddings, enterprise search and document intelligence solutions.

Agentic AI

Design production-grade AI Agent architectures and evaluate Databricks Agent Bricks and related emerging capabilities.

Data Governance

Establish Unity Catalog, access control, lineage, data-quality and metadata architecture

Security and Regulatory

Embed privacy, security, auditability and regulatory controls into Data and AI architectures.

Innovation

Continuously evaluate emerging Data, AI and Databricks capabilities and take relevant PoCs into production.

Preferred candidate profile

Location - Thane

Experience: 15-16 years of relevant experience in Data Engineering and Advanced Analytics.

Core Skills: Strong hands-on expertise in Databricks, Generative AI (GenAI), and Agentic AI frameworks, along with related technologies and use cases as outlined in the JD.

Educational Qualification: MCA or Engineering Graduate preferred.

Additional Preference: Candidates with proven experience in designing and implementing scalable AI-driven solutions, data platforms, and intelligent automation frameworks will be preferred