AI Test Automation Specialist

Role & responsibilities

Key Responsibilities: Design and implement automation frameworks for UI, API, and AI driven applications • Validate LLM responses, prompts, AI workflows, and RAG pipelines for accuracy, faithfulness, and context relevance

• Develop AI test cases covering LLM, RAG, and prompt-driven flows • Integrate LLMs with test automation frameworks to: • Auto-generate test cases • Validate AI responses • Assist in script creation using AI-assisted coding • Perform functional, regression, and exploratory testing of AI systems • Evaluate RAG performance using frameworks like RAGAS (faithfulness, context precision, answer relevance) • Create and manage test datasets, telemetry, and logs for AI validation • Collaborate with engineering, data, and ML teams to improve test reliability and AI quality Qualifications and Skills: • Bachelors or master’s degree in computer science, Engineering, Data Science, or equivalent experience. • 6+ years of experience in QA automation or software engineering. • 3+ years of experience working with AIdriven automation or intelligent testing frameworks. • Minimum 2 years of handson experience in building or testing AIbased tools or platforms. Tools & Technologies
• Must Have Tools •
• Selenium / Playwright / Cypress / Appium •
• REST Assured / Postman • Python / Java / JavaScript / TypeScript test frameworks •
• PyTest / JUnit / TestNG •
• Jira / TestRail •
• AWS CloudWatch / Grafana / Prometheus •
• Cursor AI / GitHub Copilot / Op

Preferred candidate profile