GenAI / Agent Testing Engineer

GenAI / LLM Agent Test Engineer

Experience : 5 to 9 Years

Location : Bangalore

Job Summary:

We are looking for a GenAI / LLM Agent Test Engineer to assure quality, reliability, and responsible behavior of LLMbased and RAGbased agents, including singleagent and multiagent workflows. The role focuses on prompt engineering, hallucination detection, agent reasoning validation, and automated evaluation of GenAI outputs using modern LLM testing frameworks.

Key Responsibilities:

• Test and validate LLMbased and RAGbased agents, including reasoning, memory, and decisionmaking behavior
• Design and execute prompt engineering and adversarial testing to uncover hallucinations, bias, and edge cases
• Perform hallucination testing, relevance checks, and factual accuracy validation
• Implement unit tests for LLMs using LLMasaJudge techniques
• Validate agent reasoning chains, memory states, and conversation persistence
• Embed LLMs within testing pipelines to: 
• Detect hallucinations
• Run adversarial prompt testing
• Perform automated evaluation of agent outputs
• Ensure AI Ethics and Responsible AI compliance in testing coverage
• Collaborate with platform, data, and GenAI teams for continuous quality improvement

Qualifications and Skills: 

• Bachelors or master’s degree in computer science, Engineering, Data Science, or equivalent experience.
• 6+ years of experience in QA automation or software engineering.
• 3+ years of experience working with AIdriven automation or intelligent testing frameworks.
• Minimum 2 years of handson experience in building or testing AIbased tools or platforms.

Must Have Tools

• LangChain + LangSmith
• DeepEval
•  Python (LLM testing harnesses) 
• OpenAI / Azure OpenAI APIs 
• Git + CI integration 
•  AWS AgentCore
•   AWS CloudWatch / Grafana / Prometheus 
•  Cursor AI / GitHub Copilot / Claude CLI