Senior QA Automation Engineer

Job Requirements

Senior/Lead QA Automation Engineer AI/ML & GenAI

Power Systems AI HUB

Power Systems builds highly available, massively scalable AI digital platforms for the enterprise. As part of our AI Hub organization, we are looking for a highly skilled Automation Test Engineer to support our AI/ML and GenAI product ecosystem.

Role Overview

We are seeking a Senior QA Engineer – Automation Testing with 5–6 years of experience in designing, developing, and executing automation frameworks for AI/ML-driven applications. The ideal candidate brings hands-on experience with GenAI Testing, LangSmith, PyTest, Agentic frameworks, and strong AI/ML evaluation expertise.

Key Responsibilities

AI / GenAI Testing

•         Design and execute V&V model strategies for classical ML and GenAI/ agentic systems.

•         Define test plans, tests cases, and execute them

•         Evaluate AI models for accuracy, performance, safety, robustness, hallucinations, bias, and correctness.

•         For GenAI, rely on LangSmith for LLM observability, tracing, monitoring, and evaluation, for offline and online evaluation

•         For classical ML UC, ensure V&V tests are done during the whole life cycle of the model (ML monitoring)

Test Automation

•         Build scalable automation frameworks using Python & PyTest integrated with CI/CD pipelines.

•         Develop automated test suites for APIs, microservices, and AI pipelines.

•         Perform sanity, integration, functional, regression, and UAT testing across QA, UAT, and Prod environments.

•         Implement automation strategies: Keyword-driven, Data-driven, BDD, Hybrid frameworks.

Quality Assurance & Reporting

•         Analyze test results and provide insights to engineering teams.

•         Log and track defects using standard bug-reporting tools.

•         Maintain high-quality documentation for all test processes and reports.

Required Skills & Experience

•         5–6 years of experience in software testing & automation.

•         Strong programming expertise in Python.

•         Good understanding of Data Science related metrics (for both classical ML and GenAI UC) Experience with AI/Chatbot products and testing AI applications.

•         Experience with Git, GitHub Copilot, and CI/CD pipelines.

•         Knowledge of Azure/AWS cloud environments.

•         Familiarity with MySQL, PostgreSQL, Oracle, SQL Server, NoSQL databases.

•         Strong verbal & written communication skills.

•         Experience with multi-stakeholder delivery models including onshore–offshore.

Preferred

•         Experience in AI/ML model evaluation.

•         Exposure to AI observability tools and automated prompt validation.

•         Hands-on expertise with LangSmith, Agentic frameworks, PyTest, BDD/V&V frameworks.

 Soft Skills

•         Analytical & Critical Thinking

•         Curiosity & Continuous Learning

•         Strategic Communication (Active Listening, Technical Translations to business impacts)

Work Experience

Educational Qualification

• Bachelor’s or Master’s degree in Computer Science, Electronics, or related field

Experience

• Minimum 5 to 6 years of experience in AI/ML & GenAI Testing