AI Engineer

Description of the Role

As an AI Engineer at Vytalize Health, you will design, build, and maintain agentic systems and LLM-powered applications that automate complex healthcare workflows and accelerate our ability to deliver data-driven clinical solutions. Working at the intersection of applied AI and healthcare, you will build agents that orchestrate data retrieval, model inference, clinical logic, and tool use to solve problems that traditionally require manual effort or specialized expertise.

You will work cross-functionally with data engineering, platform, product, and clinical teams to identify high-impact opportunities for AI automation—from data source onboarding to clinical decision support to evidence synthesis. Your focus will be on building production-grade agentic systems with rigorous validation, clear confidence scoring, and human-in-the-loop oversight to ensure reliability in a regulated healthcare environment. You will establish patterns, best practices, and tooling that allow the organization to scale AI-driven automation across multiple domains. You will measure agent performance and impact—tracking accuracy, hallucination rates, and real-world clinical outcomes. You will be part of a growing AI team that values both cutting-edge AI capabilities and deep healthcare domain understanding.

Primary Responsibilities

• Design, build, and maintain agentic systems and LLM-powered applications that automate healthcare workflows, data pipelines, and clinical decision support — from conception through production deployment

• Build and orchestrate agents using LLM APIs (OpenAI, Anthropic, etc.) and agentic frameworks (LangChain, LangGraph, CrewAI, or custom orchestration) to solve complex, multi-step healthcare problems

• Develop prompt libraries, agent instructions, and reusable "skills" that improve agent accuracy, consistency, and reliability across different use cases and data domains

• Build validation and confidence-scoring layers that flag low-confidence agent decisions for human review before production deployment; establish guardrails and review workflows for agent-authored code and outputs

• Own end-to-end delivery of AI-automated systems — from problem scoping and requirements gathering through agent development, testing, and validated production deployment

• Implement rigorous evaluation and QA frameworks for agentic systems — including golden datasets, test cases, output validation, hallucination detection, and regression testing

• Establish and maintain evaluation metrics for agent performance, reliability, and clinical appropriateness; measure agent accuracy, hallucination rates, clinical validity, and real-world impact

• Implement observability, evaluation, and regression testing frameworks specific to agentic systems — decision tracing, lineage logging, and performance tracking

• Collaborate with data engineering and platform teams to integrate agent-built outputs (dbt models, transformation logic, recommendations) into existing data architectures and clinical workflows

• Ensure all agentic systems comply with healthcare regulations (HIPAA, FDA guidance on AI/ML) and responsible AI practices — including explainability, auditability, and clinician trust

• Continuously evaluate new LLM models, agent frameworks, prompt engineering techniques, and tooling; recommend adoption or migration based on healthcare-specific requirements (accuracy, cost, latency, regulatory alignment)

• Partner with data engineering to establish robust data validation and input validation layers for agents — agents are only as good as the data they operate on

• Lead experimentation and measurement of AI-automated systems impact on speed, quality, compliance, and cost across healthcare workflows

• Document agent architectures, prompt strategies, evaluation frameworks, and best practices for both technical and non-technical stakeholders

• Mentor AI Connector Engineers and other team members on agentic development patterns, LLM-powered application design, and responsible AI practices

• Work on-call as needed to support production agentic systems, troubleshoot agent issues, and respond to performance degradation or hallucination detection

Required Qualifications

• 3+ years of professional experience in data engineering, backend engineering, machine learning, or a related field

• 1+ years of hands-on experience building with LLM APIs and agentic orchestration frameworks — not just using AI coding assistants, but architecting agentic systems

• Strong Python and SQL proficiency

• Experience with cloud data platforms (AWS, Databricks)

• Solid understanding of data modeling, ETL/ELT patterns, and medallion architecture (Bronze/Silver/Gold)

• Experience building and consuming APIs

• Demonstrated experience with prompt engineering, agent evaluation, and validating LLM outputs

• Experience designing evaluation frameworks, test cases, and quality assurance for AI/ML systems

• Demonstrated ability to measure and track AI system performance through metrics and KPIs (accuracy, precision, recall, hallucination rates)

• Strong debugging and analytical skills, especially in ambiguous or novel technical territory

• Excellent written and verbal communication skills — this role requires documenting agent reasoning, decisions, and limitations clearly for both technical and non-technical audiences

• Comfortable working in a fast-moving environment with incomplete information and rapidly evolving AI/ML capabilities

Strong Pluses

• Experience with dbt or similar data transformation frameworks

• Familiarity with orchestration tools (Airflow, Databricks Workflows) and workflow automation

• Experience with agent evaluation and observability tooling (LangSmith, Langfuse, or custom frameworks)

• Background in healthcare, fintech, or another regulated/high-stakes domain where AI reliability is critical

• Experience building internal developer tooling, platform capabilities, or developer-facing products

• Hands-on experience with RAG (retrieval-augmented generation) or other grounding techniques for LLMs

• Familiarity with healthcare data formats and standards (FHIR, HL7, claims data, clinical NLP)

• Experience with model evaluation, fairness assessment, or bias detection in ML/AI systems

• Understanding of healthcare regulations (HIPAA, FDA guidance on AI/ML) and responsible AI practices

• Experience establishing QA frameworks, test plans, and quality metrics for ML/AI systems

• Startup or high-growth environment experience with rapid iteration and learning

• Published research, open-source contributions, or demonstrated thought leadership in AI/agentic systems

This job description is not designed to cover or contain a comprehensive listing of activities, duties, or responsibilities that are required of the employee. Other duties, responsibilities, and activities may change or be assigned at any time with or without notice.