Senior LLM/ Gen AI Test Engineer

Role Overview

We are seeking a Senior LLM/GenAI Testing Engineer to ensure the quality, reliability, and safety of AI-powered applications such as chatbots, copilots, RAG systems, and AI agents. The role involves designing test strategies, building automated evaluation frameworks, validating AI outputs, and ensuring production readiness.

Key Responsibilities

• Test and evaluate LLM/GenAI applications for accuracy, relevance, hallucinations, and instruction-following.
• Develop automated evaluation frameworks using Python and GenAI testing tools.
• Validate RAG systems, chatbots, and agent-based workflows.
• Perform AI security, safety, and prompt injection testing.
• Test API performance, latency, and response quality.
• Create benchmark datasets, regression suites, and quality metrics.
• Collaborate with AI, Product, and Engineering teams to define release quality gates.

Required Skills

• 5+ years in QA/Test Automation; 2+ years in LLM/GenAI testing.
• Strong Python, PyTest, API Testing, and SQL skills.
• Hands-on experience with LLMs, GPT, GenAI, RAG, Chatbots, or Conversational AI.
• Experience with evaluation tools such as DeepEval, Ragas, Promptfoo, LangSmith, or similar.
• Knowledge of CI/CD, test automation, and quality assurance best practices.

Preferred Skills

• Experience with AI security testing, red teaming, and OWASP LLM.
• Knowledge of vector databases, embeddings, and Responsible AI practices.
• Exposure to prompt engineering and GenAI evaluation methodologies.