AI Agent Evaluator

AI AGENT DEVELOPMENT & EVALUATION

Primary Care, Acute Symptom Triage & Travel Health | Contract, up to 10 hrs/week

ABOUT THE ROLE

We are seeking Nurses & Nurse Practitioners with clinical experience in one or more of the following areas:

• acute symptom management

• CHF and hypertension management

• diabetes management

• travel health

to serve as a subject matter expert supporting the evaluation of AI-driven clinical agents. In this role, you will evaluate agent conversations and recommendations spanning acute symptom triage and escalation decisions, insulin/diabetes titration, and pre-travel health guidance, ensuring clinical accuracy, guideline adherence, and patient safety. This is a part-time, remote contract position (1099) requiring up to 10 hours per week, with flexible scheduling.

KEY RESPONSIBILITIES

• Review recorded and simulated AI agent interactions involving acute symptom triage and assess the appropriateness of escalation and de-escalation decisions.

• Review agent conversations related to insulin and diabetes medication titration for clinical accuracy and guideline adherence.

• Evaluate agent-delivered travel health guidance, including pre-travel counseling, vaccination recommendations, and destination-specific risk assessment.

• Identify clinical safety concerns, edge cases, and gaps in the agent's triage logic or clinical reasoning.

• Score and annotate agent performance using structured evaluation rubrics.

• Provide clear, actionable written feedback to clinical and product teams to inform agent refinement.

• Participate in periodic case review and calibration sessions with other clinical SMEs to ensure consistent evaluation standards.

• Help develop and refine test scenarios used in ongoing agent quality assurance.

QUALIFICATIONS

• Registered Nurses: active, unrestricted RN License Nurse Practitioners: active, unrestricted Nurse Practitioner license in the United States, with current board certification (ANCC or AANP).

• Minimum of 5+ years of direct clinical experience in primary care, urgent care, telephone/virtual triage, or a comparable acute-care setting.

• Clinical experience managing patients with diabetes, including insulin titration.

• Strong, current working knowledge of relevant triage protocols and clinical practice guidelines.

• Ability to critically evaluate clinical reasoning and identify inaccuracies or safety risks in written or recorded clinical interactions.

• Comfortable working independently in a remote, asynchronous contract role.

• Strong written communication skills for delivering structured clinical feedback.

RECOMMENDED

It is not a requirement that candidates have all of the recommended criteria

• Experience in telephone triage or nurse advice line settings.

• Experience in travel medicine, occupational health, or pre-travel consultation.

• Familiarity with CDC and other travel medicine guidelines.

• Prior experience in clinical quality assurance, utilization review, or case review.

• Familiarity with AI-assisted clinical tools or digital health platforms.

ENGAGEMENT DETAILS

• Contract (1099) position, up to 10 hours per week.

• Fully remote; flexible scheduling.

• Hourly compensation, commensurate with experience.

• Engagement expected to run in support of ongoing agent development and FDA evaluation milestones.

Please be aware of recruitment scams impersonating Hippocratic AI. All recruiting communication will come from @hippocraticai.com http://hippocraticai.com email addresses. We will never request payment or sensitive personal information during the hiring process.