Senior QA Engineer, AI Agents & Agentic Workflows
Webjaguar India · Pune · onsite · Posted 2d ago
Undisclosed3–5 yrsfulltimeSourced from
About a minute. You'll make an account on the way through.
About this role
Job Description
QAD is seeking a Sr QA Engineer to own the continuous validation and quality of Champion Assist, a multi-
agent AI product with ~20 evolving agents. You will test the agents themselves: their reasoning, tool use,
orchestration and handoffs, and the quality of their responses. This goes beyond UI and API checks. You will
design evaluation frameworks and regression suites that catch failures unique to non-deterministic, LLM-
driven systems.
agent AI product with ~20 evolving agents. You will test the agents themselves: their reasoning, tool use,
orchestration and handoffs, and the quality of their responses. This goes beyond UI and API checks. You will
design evaluation frameworks and regression suites that catch failures unique to non-deterministic, LLM-
driven systems.
Responsibilities:
- Design, run and maintain end-to-end business scenarios that exercise multi-step agentic workflows,
including agent routing, tool/function calls, and multi-turn conversations.
- Evaluate agent outputs for correctness, relevance, groundedness, consistency, tone and
completeness, using both human review and automated methods.
- Build and maintain eval datasets (golden sets, edge cases, adversarial prompts) and automated
regression suites that run as agents, prompts, models and tools change.
- Identify cross-agent, orchestration and integration issues: wrong agent selection, broken handoffs,
tool-call failures, context loss, looping or runaway behavior.
- Test for hallucination, prompt injection, jailbreaks, data leakage and unsafe or out-of-scope responses.
- Detect regressions caused by model upgrades, prompt changes or retrieval/knowledge changes, and
quantify them with metrics.
- Triage and root-cause issues with Engineering and Product, separating prompt, model, retrieval, tool
and orchestration causes.
- Re-test fixes, validate the end-user experience, and keep evals current as the product evolves.
- Document test strategy, eval results and quality trends clearly for global stakeholders. Qualifications
including agent routing, tool/function calls, and multi-turn conversations.
- Evaluate agent outputs for correctness, relevance, groundedness, consistency, tone and
completeness, using both human review and automated methods.
- Build and maintain eval datasets (golden sets, edge cases, adversarial prompts) and automated
regression suites that run as agents, prompts, models and tools change.
- Identify cross-agent, orchestration and integration issues: wrong agent selection, broken handoffs,
tool-call failures, context loss, looping or runaway behavior.
- Test for hallucination, prompt injection, jailbreaks, data leakage and unsafe or out-of-scope responses.
- Detect regressions caused by model upgrades, prompt changes or retrieval/knowledge changes, and
quantify them with metrics.
- Triage and root-cause issues with Engineering and Product, separating prompt, model, retrieval, tool
and orchestration causes.
- Re-test fixes, validate the end-user experience, and keep evals current as the product evolves.
- Document test strategy, eval results and quality trends clearly for global stakeholders. Qualifications
- Bachelors degree in software engineering, computer science or a related field.
- 3-5 years of QA experience, with at least 1-2 years testing LLM-based applications, AI agents or
agentic workflows.
- Hands-on experience evaluating non-deterministic outputs: defining pass/fail criteria, rubrics, and
acceptance thresholds for AI responses.
- Experience building or maintaining AI eval or regression suites (e.g., LLM-as-judge, golden datasets,
scoring rubrics).
- Understanding of agentic concepts: tool/function calling, multi-agent orchestration, planning,
memory/context, RAG.
- Experience with test automation tools (Selenium, Playwright, TestComplete or similar) and API testing.
- 3-5 years of QA experience, with at least 1-2 years testing LLM-based applications, AI agents or
agentic workflows.
- Hands-on experience evaluating non-deterministic outputs: defining pass/fail criteria, rubrics, and
acceptance thresholds for AI responses.
- Experience building or maintaining AI eval or regression suites (e.g., LLM-as-judge, golden datasets,
scoring rubrics).
- Understanding of agentic concepts: tool/function calling, multi-agent orchestration, planning,
memory/context, RAG.
- Experience with test automation tools (Selenium, Playwright, TestComplete or similar) and API testing.
This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.
Sourced from naukri
Skills
QAUx DesignAutomationPlaywrightRetrieval Augmented Generationqa engineerworkflowsorchestrationtest automationregressionTest strategyLarge Language Models (LLMs)uitriageAPI TestingSeleniumSoftware EngineeringArtificial Intelligence