Agentic AI Testing Engineer

Role & responsibilities

• Validate the entire agentic lifecycle, starting from user intent and prompting through agent reasoning and routing.
• Ensure accurate MCP tool execution and conduct thorough end-to-end workflow validation.
• Utilize strong observability skills to monitor and assess the effectiveness of workflows.
• Employ automation techniques to streamline testing and validation processes.
• Conduct AI quality assessments to ensure optimal functionality and performance of the system.
• Collaborate with cross-functional teams to refine and enhance lifecycle validation methodologies.

Mandatory skill sets:

• Strong experience in Agentic AI testing and validation.
• Must have hands-on knowledge of MCP testing, including tool invocation validation, payload/schema validation, orchestration, error handling, retries, and API/integration testing.
• Experience with AI observability and tracing tools such as LangFuse, Phoenix, or similar platforms for debugging, root cause analysis, and agent behavior validation.
• Strong understanding of AI quality metrics including hallucination detection, accuracy, correctness, groundedness, consistency, determinism, and latency.
• Experience designing and executing prompt testing strategies, including multi-turn, multi-hop ambiguous, negative, and guardrail/jailbreak scenarios.
• Understanding of intent recognition, routing validation, agent selection, context retention, memory validation, and reasoning/planning validation.
• Ability to create test strategies, evaluation rubrics, golden datasets, and quality metrics for AI systems.
• Experience with API testing and automation frameworks (e.g., Postman, Katalon, Playwright, or equivalent).