Data Engineer - AI & Agentic Solutions

Job Summary

The Data Engineer - AI & Agentic Solutions is a hands-on engineer who builds the data pipelines and agentic AI components that make Lillys Clinical and Non-Clinical data AI-ready and audit-ready. Working to the patterns and standards set by the data architect and senior engineers, this role writes production pipeline code, builds retrieval and tool-calling agent components, and ships them with the tests, evaluations, and observability a regulated environment requires. This is a builder role - approximately 90% hands-on engineering including daily coding, and 10% design contribution and documentation.

The role contributes directly to the Right Model, Right Task mandate - implementing and tuning the routing configurations that send each agentic and LLM workload to the model best suited to it on cost, latency, and accuracy grounds, using Lillys internal data platform for lineage and metadata rather than standing up disconnected stores. Agentic and AI-assisted ways of working are the expected default across every engineering, analysis, and documentation activity.

Key Responsibilities

Data Engineering & Pipeline Development (Hands-On)

• Build, test, and maintain production data pipelines across Clinical and Non-Clinical domains - ingestion, transformation, and publication into governed lakehouse layers (Bronze/Silver/Gold).
• Implement pipeline patterns defined by senior engineers and the architect - incremental loads, CDC, schema handling, and idempotent reprocessing - and extend them to new sources.
• Monitor and troubleshoot pipeline runs, resolve data quality and performance issues, and improve reliability against agreed SLAs.
• Contribute to curated, analytics-ready data products with documented schemas and contracts that downstream consumers and agentic systems depend on.

Agentic AI & LLM Application Development (Hands-On)

• Build agentic AI components against established patterns - tool-calling agents, retrieval-augmented generation (RAG) chains, and steps within multi-agent workflows - for Clinical and Non-Clinical use cases.
• Implement prompt, tool, and state-handling logic with attention to token/cost and latency budgets, retries, and graceful failure handling.
• Write and run evaluations, guardrail checks, and regression tests for agentic components, and act on the results to improve accuracy and groundedness.
• Instrument agents with logging, tracing, and observability so behaviour is traceable and auditable in a regulated environment.

Right Model, Right Task - Contributing to Data Engineering

• Implement and tune routing configurations that direct each agentic task to the appropriate model on complexity, cost, latency, and accuracy grounds, rather than defaulting to one model for every job.
• Wire agentic components into Lillys internal data platform (Data Hub catalog, lineage, and metadata services) rather than building parallel metadata stores.
• Contribute to lineage-aware knowledge graph pipelines that capture data provenance, sensitivity, and domain context for agent reasoning.
• Apply routing and access policy correctly in code - ensuring regulatory-sensitive Clinical data is only handled by approved/validated models.
• Run benchmarks and cost/accuracy comparisons across candidate models and share the findings with the squad.

Semantic, Retrieval & Knowledge Graph Engineering

• Build and tune retrieval components - embeddings, chunking strategies, vector indexes, and hybrid search - that ground LLMs in governed Clinical and Non-Clinical data.
• Contribute to semantic models, taxonomies, and knowledge graph structures under the guidance of senior engineers and the architect.
• Measure retrieval quality against defined benchmarks such as groundedness, recall, and citation accuracy, and iterate on the results.

Automation, Reusability & AI-Native Ways of Working

• Use AI-assisted and agentic tooling by default across the data lifecycle - pipeline generation, schema alignment, data-quality scoring, test generation, and documentation.
• Adopt and extend the teams reusable accelerators - agent templates, orchestration patterns, evaluation harnesses, and pipeline frameworks - and contribute improvements back.
• Automate repetitive checks such as lineage capture, schema-drift detection, and data quality validation instead of handling them manually.

Quality, Governance & Responsible AI

• Apply access control, encryption, and prompt/data-leakage safeguards correctly when building on regulated data.
• Build data quality, lineage, and compliance checks into pipelines and agentic components as part of the delivery, not afterwards.
• Follow engineering discipline consistently: version control, code review, automated testing, and CI/CD practices appropriate to a GxP environment.

Collaboration & Knowledge Sharing

• Work with senior engineers, the data architect, business SMEs, and platform teams to turn Clinical and Non-Clinical requirements into working solutions.
• Ask questions early, raise blockers and risks promptly, and seek review on design decisions before building.
• Share learnings, document what is built, and support teammates as the squad scales its AI-native delivery.

Technical Skills & Qualifications

Required - Data Engineering (Must-Have, Hands-On)

• Strong hands-on Python and SQL, with working experience in Spark/PySpark for distributed data processing.
• Hands-on experience building and maintaining production data pipelines on a cloud lakehouse platform (Databricks, or equivalent), including Delta/Parquet and orchestration tooling.
• Practical data modeling experience - relational, dimensional, and lakehouse/medallion patterns.
• Working proficiency with Git, code review, automated testing, and CI/CD practices.

Required - AI & Agentic Development (Must-Have, Hands-On)

• 2+ years hands-on experience building AI/agentic components that reached production or pilot: RAG

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.