Azure Databricks Engineer ( Agentic AI )

Job Summary


Responsibilities

• Build batch and streaming pipelines with PySpark, Delta Lake, Delta Live Tables and Auto Loader on the medallion (bronze/silver/gold) pattern.
• Ingest structured and unstructured enterprise data (ERP, SharePoint, PDFs, email, contracts) and prepare it for LLM use: parsing, chunking, metadata enrichment and embeddings.
• Govern data, models, functions and vector indexes in Unity Catalog with lineage, access control and PII handling.

• Build and tune retrieval pipelines on Mosaic AI Vector Search, including hybrid search, re-ranking and index sync from Delta tables.
• Develop tool-calling agents with the Mosaic AI Agent Framework, LangChain/LangGraph or similar, exposing Unity Catalog functions and SQL as agent tools.
• Integrate foundation models via Databricks Model Serving, Azure OpenAI and external model endpoints.
• Support multi-agent workflows for finance, procurement and operations use cases (e.g. invoice validation, quotation comparison, contract retrieval).

• Track, version and register agents and models with MLflow run evaluations with Mosaic AI Agent Evaluation (groundedness, relevance, safety) and LLM-as-judge.
• Deploy agents to Model Serving endpoints monitor latency, cost and quality through inference tables and tracing.
• Automate delivery with Databricks Asset Bundles, Git integration and Azure DevOps or GitHub Actions.
• Optimise cluster, job and serving cost apply guardrails, prompt-injection protection and data-leak controls.

• Work with solution architects and business analysts to turn BRDs into data and agent designs.
• Expose agents to business users through apps, APIs, Copilot Studio or Power Platform integrations.
• Document designs, runbooks and evaluation results for client handover.

Required skills and qualifications

• Bachelor s degree in Computer Science, Engineering or a related field.
• About 3 years in data or ML engineering, with at least 1.5 years on Azure Databricks in production.
• Strong Python and PySpark solid SQL and Spark performance tuning.
• Hands-on with Delta Lake, Unity Catalog, Databricks Workflows and MLflow.
• At least one delivered GenAI project: RAG, embeddings, vector search or an LLM agent in production or pilot.
• Working knowledge of an agent framework (Mosaic AI Agent Framework, LangChain/LangGraph, Semantic Kernel or AutoGen) and of prompt engineering and tool/function calling.
• Familiarity with Azure services: ADLS Gen2, Azure Data Factory, Key Vault, Azure OpenAI, Entra ID.
• Git, CI/CD and basic infrastructure-as-code (Terraform or Bicep).
• Clear written and verbal communication comfortable in client-facing discussions.

Good to have

• Certifications: Databricks Certified Data Engineer Associate, Databricks Generative AI Engineer Associate, or Microsoft DP-203 / AI-102.
• Azure AI Foundry, Azure AI Search or Document Intelligence experience.
• Exposure to Microsoft Power Platform or Copilot Studio integrations.
• Domain exposure to finance, procurement, construction or HSE processes.
• Knowledge of LLM evaluation, guardrails and Responsible AI practices.
• Experience with Model Context Protocol (MCP) or agent-to-agent orchestration patterns.

Disclaimer: This job posting and Location has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.