Qa Test Engineer

Job Summary

We are looking for a QA Automation Engineer to own fully automated testing of our AI products. This is an automation-only role with no manual testing. All functional, API, regression and LLM output tests must run automatically in the CI/CD pipeline and act as quality gates before every release.

Key Responsibilities

• Design and own the test automation strategy and framework for AI products.
• Build automated UI, API and end-to-end test suites using Playwright/Selenium and PyTest.
• Automate LLM and GenAI output evaluation, covering accuracy, relevance, hallucination, safety and prompt regression.
• Integrate all test suites into CI/CD pipelines (Azure DevOps/GitHub Actions) as release quality gates.
• Build automated regression, smoke and performance tests, including test data management.
• Track and report test coverage, pass rates and defect trends through dashboards.
• Work with AI/ML and DevOps engineers on shift-left testing and fast feedback loops.

Required Skills

• Strong test automation with Playwright and/or Selenium.
• Python with PyTest; API test automation (REST Assured, Postman/Newman or Python requests).
• CI/CD integration of tests in Azure DevOps or GitHub Actions.
• LLM evaluation testing using tools such as DeepEval, Ragas, promptfoo or similar.
• Git, test reporting (Allure or similar) and Agile/Scrum practices.
• Clear understanding of test design, coverage and defect lifecycle.

Preferred Skills

• Performance testing (JMeter, Locust or k6).
• Security test basics (OWASP ZAP) and accessibility testing.
• Testing in Docker/Kubernetes environments.
• ISTQB Advanced Test Automation Engineer certification.

Educational Qualification

B.E. / B.Tech. / M.E. / M.Tech. / MCA , IT, AI, Data Science or a related discipline. ISTQB or an equivalent test automation certification is a plus.