Agentic QA

Key Responsibilities

1. Agentic AI Testing, Evaluation & Automation

• Define and execute testing strategies for LLM-based, multi-agent, RAG, and Agentic AI systems.

• Validate autonomous agent behavior, reasoning, memory, tool usage, API/database integrations, and end-to-end workflows.

• Evaluate AI outputs for accuracy, relevance, groundedness, consistency, completeness, toxicity, bias, hallucination risk, and guardrail compliance.

• Define AI quality KPIs such as hallucination rate, groundedness score, agent success rate, task completion rate, response relevancy, latency, cost efficiency, and user satisfaction.

• Build automated evaluation pipelines, quality scoring mechanisms, dashboards, and CI/CD-integrated quality gates.

• Develop reusable test harnesses, simulators, and benchmarking frameworks to compare models, prompts, and agent configurations.

2. Team Leadership & Capability Building

• Build and lead a team of Agentic AI Quality Engineers.

• Define team structure, testing standards, best practices, and governance models.

• Mentor QA engineers in AI testing methodologies, evaluation techniques, and automation frameworks.

• Drive innovation and adoption of emerging AI testing tools and technologies.

• Collaborate with Product, Engineering, Data Science, and AI Research teams to improve overall AI quality.

3. Reporting & Stakeholder Management

• Provide quality assessments and recommendations to leadership and stakeholders.

• Present testing outcomes, risk assessments, KPI trends, and model evaluation reports.

• Drive quality governance for Agentic AI initiatives across the organization.

• Ensure traceability of testing activities, evaluation criteria, and quality benchmarks.

________________________________________

Required Skills & Experience

Technical Skills

• 7–12 years of experience in Software Testing, Quality Engineering, or Test Automation.

• Minimum 3+ years of hands-on experience in GenAI, LLM Testing, Agentic AI Testing, or AI Quality Engineering.

• Strong understanding of LLMs, AI agents, RAG, prompt validation, tool calling, agent memory, MCP, and multi-agent orchestration.

• Experience defining AI quality metrics, evaluation methodologies, benchmarking frameworks, and model comparison approaches.

• Hands-on automation experience with Python, Playwright, Pytest, API automation, test framework development, and CI/CD quality gates.

• Experience with AI evaluation frameworks such as DeepEval, Ragas, LangSmith, OpenAI Evals, or equivalent tools.

• Exposure to cloud platforms such as Azure, AWS, or GCP.

Soft Skills

• Strong communication and stakeholder management skills.

• Analytical mindset with strong problem-solving ability.

• Self-driven, outcome-oriented, and capable of leading multiple initiatives in a fast-evolving AI ecosystem.

________________________________________

Preferred Qualifications

• Experience testing enterprise Agentic AI platforms and autonomous AI systems.

• Hands-on exposure to frameworks or tools such as LangGraph, CrewAI, AutoGen, Semantic Kernel, Microsoft Copilot Studio, TruLens, or Promptfoo.

• Exposure to AI observability, monitoring, model governance, responsible AI, and AI safety practices.

• Experience building AI quality dashboards and KPI reporting systems.

• ISTQB, AI Testing, GenAI, Azure AI, AWS AI, or equivalent certifications.