Databricks Architect

JOB DESCRIPTION

Databricks Data Architect (Lakehouse & Agentic AI)

Role Title: Databricks Data Architect Experience: 12+ Years (4+ in Databricks)

Job Location: Hybrid, Offshore

Employment Type: Full-Time / Permanent

Key Mandatory Skills: Databricks Lakehouse, Unity Catalog, Genie Agent

Secondary Skills: AI Search, Databricks Apps, AWS Services

1. Position Summary

We are seeking an experienced and visionary Databricks Data Architect to join our high-growth Data & AI Practice. In this pivotal role, you will serve as the principal data authority on a cutting-edge client engagement designed to unify enterprise data and deliver sophisticated Agentic AI solutions on the Databricks Lakehouse platform. You will bridge high-level solution architecture with deep data engineering executionowning end-to-end data pipeline designs, data modeling, entity reconciliation, and Unity Catalog governance, while enabling next-generation Databricks AI capabilities (such as Genie Agents, AI Search / Vector Search, Knowledge Assistants and Databricks Apps). You will actively lead and mentor the engineering build team and serve as the trusted front-facing advisor for client stakeholders.

2. Project Architecture & Solution Scope

The upcoming engagement implements an advanced Lakehouse + Multi-Agent AI architecture. As the Data Architect, your technical canvas encompasses:

Multi-Source Data Ingestion & Lakehouse Foundation: Integrating heterogeneous enterprise transactional and master datasets via Databricks Connect and structured views, alongside streaming and batch ingestion of unstructured documents from AWS S3.

Entity Reconciliation & Metadata Enrichment: Architecting robust data cleansing, deduplication, and Canonical ID reconciliation models across source systems to produce high-fidelity gold-standard lakehouse tables.

Unified Governance via Unity Catalog: Establishing enterprise-grade data and AI governance including Metric Views, Agent Configurations, Fine-Grained RBAC, Row/Column-Level Security (RLS/CLS), MCP Registry, and Trusted Asset certification.

Databricks AI & Agentic Integration: Structuring curated semantic models and domain ontologies to power Databricks Genie Agents; orchestrating AI Search Vector Indexes feeding Knowledge Assistants; and collaborating with GenAI/agent developers on multi-agent supervisor/summarizer agent patterns and Databricks Apps.

Serverless Compute & Performance Optimization: Leveraging Serverless SQL Warehouses and automated compute clusters to guarantee cost efficiency, sub-second query performance, and seamless concurrency.

3. Key Responsibilities & Ownership Areas

A. Solution & Data Architecture Ownership

Lakehouse Blueprint: Design, review, and own the comprehensive data architecture, data modeling (conceptual, logical, physical), and medallion pipeline architecture on Databricks.

Data Harmonization: Drive entity resolution and canonical data mapping across disparate enterprise data sources (e.g., GCDB, GIMD, operational MI). Design metadata enrichment.

Governance & Security: Architect data governance frameworks utilizing Unity Catalog, enforcing row/column security, lineage tracking, data quality metrics, and secure LLM/agent resource access.

Unstructured Data & Search Pipeline: Define document ingestion strategies (AWS S3 + Auto Loader), text parsing, metadata enrichment, and synchronization with Databricks AI Search / Vector Indexes.

B. Databricks AI & Modern Platform Offerings

Genie Agent Enablement (Mandatory): Partner with business and data analysts to model Genie Agents to empower self-service conversational BI.

Knowledge Assistant & AI Search: Ensure data readiness, chunking strategies, and index refreshes for Lakehouse Knowledge Assistants and Agent Bricks consumers.

Databricks Apps & Extensions (Good-to-Have): Advise on integration patterns for custom UI/data applications hosted natively within Databricks Apps, interfacing with backend Serverless SQL and agent workflows.

C. Technical Leadership & Build Team Guidance

Engineering Mentorship: Lead, coach, and provide technical guidance to senior and mid-level Data Engineers, instilling software engineering best practices, Delta Lake optimization (Z-Order, Liquid Clustering, compaction), and reusable coding standards.

Code & Design Governance: Conduct rigorous peer design reviews, code reviews, and establish CI/CD pipeline automation (Databricks Asset Bundles / DABs, Git integration).

Troubleshooting & Quality Assurance: Act as the escalation point for complex technical bottlenecks in PySpark jobs, streaming triggers, connection adapters, and warehouse concurrency.

D. Client-Facing Leadership & Stakeholder Management

Trusted Client Advisor: Serve as the primary technical point-of-contact for client architects, data product owners, and business leaders on all data-centric discussions.

Requirements Translation: Translate ambiguous business queries, AI requirements, and functional use cases into robust data contracts, technical specifications, and scalable sprint backlogs.

Steering & Demonstrations: Participate in architecture review boards (ARBs), sprint showcases, and POC demonstrations to executive client stakeholders.

4. Skills & Qualifications Matrix

A. Mandatory Technical Competencies

Databricks Lakehouse Mastery: Extensive hands-on experience designing and deploying enterprise solutions on Databricks (Delta Lake, Lakehouse architecture, PySpark, Spark SQL).

Databricks Genie Agent (Mandatory): Hands-on knowledge of Databricks Genie configuration, ontology modeling, trusted asset curation, and metric definition for conversational AI querying.

Unity Catalog Governance: Deep experience configuring Unity Catalog, defining RBAC/RLS policies, data lineage, external locations, and centralized metadata.

Ingestion & Storage: Proven expertise with Databricks Auto Loader, batch/micro-batch ETL, AWS S3 integration, and Databricks Connect / external database views.

Data Modeling & Canonical Entity Design: Demonstrated success designing dimensional models, canonical ID reconciliation, deduplication, and master data management (MDM) logic.

Databricks AI Search / Vector Search: Strong working understanding of Databricks Vector Search / AI Search Index creation, metadata enrichment, and retrieval patterns for GenAI.

Compute Optimization: Deep tuning expertise in Serverless SQL Warehouses, cluster sizing, photon engine, cost optimization, and query profiling.

B. Desired & Good-to-Have Competencies

Databricks Apps: Familiarity with building, containerizing, or deploying interactive front-end applications natively within Databricks Apps.

Agentic Frameworks & LLMs: Understanding of Databricks Agent Bricks, MLflow 2.x/3.x evaluate for agents, LangChain/LlamaIndex, Supervisor-Worker agent topologies, or Model Serving endpoints.

DevOps / MLOps: Hands-on with Databricks Asset Bundles (DABs), automated deployment pipelines.

C. Leadership, Consulting & Soft Skills

Consultative Presence: Outstanding verbal, presentation, and written communication skills with proven ability to articulate complex technical trade-offs to client stakeholders.

Team Mentorship: Demonstrated experience leading multi-disciplinary engineering pods (5+ engineers) in an agile global delivery model.

Ownership & Delivery Focus: Pragmatic problem solver with high ownership, able to navigate ambiguity, unblock dependencies, and guarantee production-readiness.

5. Education & Professional Certifications

Education: B.E. / B.Tech / M.E. / M.Tech in Computer Science, Information Technology, Data Science, or related engineering discipline.

Preferred Certifications: Databricks Certified Solutions Architect, Databricks Certified Data Engineer Professional, Databricks Generative AI Engineer Associate, or AWS Solutions Architect Associate/Professional.

Srikala Elpula

Talent Acquisition

[email removed]
M: [phone removed]