QA AI Evaluator (AI testing)

About the role

The QA AI Evaluator will be responsible for assuring the quality, reliability, governance, and business effectiveness of Agentic AI solutions. The role focuses on evaluating AI agents, prompts, LLM outputs, and end-to-end workflows to ensure accurate, consistent, and trusted outcomes.

• Define AI evaluation frameworks, standards, KPIs, and success criteria.
• Validate agent accuracy, reasoning, reliability, workflow quality, and business outcomes.
• Assess prompts, LLM responses, hallucination risks, and AI-generated content quality.
• Benchmark models, tools, prompts, and agent configurations to improve performance.
• Ensure Responsible AI, security, privacy, governance, and compliance requirements are met.
• Provide clear insights, risks, recommendations, and scorecards to leadership and stakeholders.

What you'll need to succeed (Skills & Experience)

• Strong experience in AI Testing, Quality Engineering, AI/LLM Evaluation, or Agentic AI Assurance.
• Deep understanding of LLMs, Prompt Engineering, RAG, AI Agents, and Multi-Agent Architectures.
• Expertise in defining evaluation frameworks, quality metrics, KPIs, scorecards, and benchmarking approaches.
• Experience validating AI accuracy, reasoning, reliability, hallucinations, and business outcomes.
• Knowledge of Responsible AI, Security, Privacy, Governance, Risk, and Compliance practices.
• Strong analytical and problem-solving skills with experience in root cause analysis and continuous improvement.
• Familiarity with AI evaluation tools, test automation, and data-driven quality assessment techniques.
• Ability to collaborate effectively with Product, Engineering, Architecture, QA, and Business stakeholders.
• Excellent communication and stakeholder management skills, including presenting insights to leadership.
• Experience driving AI adoption, change management, and business transformation initiatives.

What youll need to succeed (Skills & Experience)

Essential

• 3-4 Years of experience with strong experience in AI testing, quality engineering, AI/LLM evaluation, or Agentic AI assurance.
• Good understanding of LLMs, prompt engineering, RAG, AI agents, and multi-agent workflows.
• Ability to define evaluation frameworks, quality metrics, KPIs, scorecards, and success criteria.
• Hands-on experience validating AI accuracy, reasoning, reliability, hallucination risk, relevance, completeness, and business outcomes.
• Strong analytical, problem-solving, stakeholder management, and communication skills.

Desirable

• Minimum 3-4 years of experience with AI evaluation tools, automation frameworks, and data-driven quality assessment techniques.
• Exposure to benchmarking models, prompts, tools, and agent configurations to improve performance.
• Experience supporting AI adoption, user acceptance, change management, or wider business transformation initiatives.

What you will be doing (Role Accountabilities)

• Define and govern AI quality standards, evaluation frameworks, KPIs, and success criteria.
• Validate Agentic AI solutions for accuracy, reasoning, reliability, and business effectiveness.
• Assess prompts, LLM outputs, and end-to-end agent workflows to ensure quality outcomes.
• Benchmark models, prompts, tools, and agent configurations to optimize performance.
• Measure and improve AI quality metrics, including accuracy, relevance, completeness, and hallucination rates.
• Identify risks, quality gaps, and opportunities for continuous improvement.
• Ensure Responsible AI, security, privacy, governance, and compliance requirements are met.
• Support AI adoption, user acceptance, and organizational change initiatives.
• Collaborate with Product, Engineering, Architecture, QA, and business stakeholders.
• Present insights, risks, performance metrics, and strategic recommendations to leadership.