AI Test Automation Specialist

AI Test Automation Specialist

Experience : 5 to 9 Years

Location : Bangalore

Job Summary:

We are seeking an AI Focused QA Automation Engineer to design, implement, and execute robust testing strategies for AI driven systems, including LLM based applications, RAG pipelines, and AI assisted workflows. The role requires strong foundations in QA automation, combined with AI test design, validation of model responses, and exploratory testing for nondeterministic systems.

Key Responsibilities:

• Design and implement automation frameworks for UI, API, and AI driven applications 
• Validate LLM responses, prompts, AI workflows, and RAG pipelines for accuracy, faithfulness, and context relevance 
• Develop AI test cases covering LLM, RAG, and promptdriven flows 
• Integrate LLMs with test automation frameworks to: 
• Autogenerate test cases
• Validate AI responses
• Assist in script creation using AIassisted coding
•  Perform functional, regression, and exploratory testing of AI systems 
• Evaluate RAG performance using frameworks like RAGAS (faithfulness, context precision, answer relevance) 
• Create and manage test datasets, telemetry, and logs for AI validation 
• Collaborate with engineering, data, and ML teams to improve test reliability and AI quality

Qualifications and Skills: 

• Bachelors or masters degree in computer science, Engineering, Data Science, or equivalent experience.
• 6+ years of experience in QA automation or software engineering.
• 3+ years of experience working with AIdriven automation or intelligent testing frameworks.
• Minimum 2 years of handson experience in building or testing AIbased tools or platforms.

Must Have Tools

• Selenium / Playwright / Cypress / Appium
• REST Assured / Postman
• Python / Java / JavaScript / TypeScript test frameworks
• PyTest / JUnit / TestNG
• Jira / TestRail
• AWS CloudWatch / Grafana / Prometheus
• Cursor AI / GitHub Copilot / OpenAI APIs

Good To Have Tools

• Testim / Mabl (SelfHealing Automation)
• Promptfoo (Prompt Validation)
• RAGAS (RAG Evaluation)
• Comet ML (Experiment Tracking)
• MLflow (Basic Metrics Tracking)
• K6 / Locust (Performance Testing)
• Katalon Studio
• TOSCA VisionAI