AI TEST LEAD

AI Test Lead

Experience: 7+ Years

Location: Remote

Position Summary

We are looking for an experienced AI Test Lead with 7+ years of experience in software testing and test automation, including hands-on exposure to AI/ML, Generative AI, LLM, and RAG-based applications.

The candidate will be responsible for defining test strategies, building automation frameworks, validating AI-driven applications, and leading testing initiatives across functional, API, performance, security, and AI quality dimensions.

Key Responsibilities

• Lead end-to-end testing activities for AI/ML and Generative AI applications.
• Define test strategy, test plans, test scenarios, and quality standards for AI solutions.
• Design and implement automated testing frameworks for web, API, and AI-based applications.
• Test LLM, RAG, prompt-based applications, chatbots, and AI agents.
• Validate AI responses for accuracy, relevance, consistency, toxicity, bias, hallucination, and reliability.
• Develop test datasets and evaluation methodologies for AI/GenAI applications.
• Perform API, functional, regression, integration, and end-to-end testing.
• Collaborate with developers, data scientists, product owners, and business stakeholders.
• Identify and troubleshoot defects and quality issues across application and AI components.
• Integrate automated tests into CI/CD pipelines.
• Mentor QA engineers and provide technical leadership to the testing team.
• Establish quality metrics, reporting, and continuous improvement practices.
• Ensure testing standards and best practices are followed across projects.

Required Skills

• 7+ years of experience in Software Testing / QA / Test Automation.
• Hands-on experience in AI/GenAI/LLM testing.
• Strong experience with Selenium, Playwright, Cypress, or similar automation tools.
• Strong programming skills in Python or Java.
• Experience in API testing using REST, Postman, Rest Assured, or similar tools.
• Understanding of LLM, RAG, prompt engineering, AI agents, and Generative AI applications.
• Experience with AI evaluation concepts such as hallucination, relevance, accuracy, toxicity, bias, and response quality.
• Exposure to AI testing/evaluation tools such as DeepEval, Promptfoo, OpenAI Evals, or equivalent.
• Good understanding of SQL and database testing.
• Experience with Git and CI/CD tools such as Jenkins, GitHub Actions, or Azure DevOps.
• Strong knowledge of SDLC, STLC, Agile/Scrum, defect management, and test management processes.

Good to Have

• Experience testing LangChain / LangGraph applications.
• Knowledge of vector databases and RAG architectures.
• Exposure to cloud platforms such as AWS, Azure, or GCP.
• Knowledge of AI security testing and responsible AI principles.
• Experience in performance and security testing.
• Previous experience leading or mentoring QA teams.

Education

Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.

Candidate Profile

The ideal candidate should have strong test leadership and automation experience along with practical exposure to AI/GenAI testing. The ability to work closely with engineering, data science, and product teams and translate AI quality requirements into effective testing strategies is essential.