QA Engineer

Title QA Engineer

Location India (Remote)

Duration Contract 6 to 12 months

Job Description

Overview

The QA Engineer II plays a crucial role in ensuring the quality and reliability of our Conversational AI solutions. Unlike deterministic software, the AI agents under test are non-deterministic, so benchmarking emphasizes semantic/behavioral evaluation in addition to exact-match assertions.

The ideal candidate has a strong foundation in software testing methodologies, experience with test case design and execution, and familiarity with automation frameworks. This role is ideal for detail-oriented professionals who are eager to learn, contribute to test automation, and help drive continuous improvements in software quality.

Responsibilities

• Agentic Benchmark Design: Generate, import, create, extend, and manage benchmarks for non-deterministic LLM agents, validating behavior and responses.
• Functional & Regression Testing: Grow benchmarks and create new ones as new functionality or issues are uncovered.
• Benchmarking Automation Support: Contribute to streamlining the benchmarking process in alignment with product development work.
• Collaboration: Work closely with developers, product managers, and senior QA engineers to ensure test coverage aligns with product requirements.
• Defect Identification & Reporting: Log detailed defect reports and work with engineering teams to ensure timely resolution.
• Process & Traceability: Maintain requirement-to-test traceability and document detailed test plans and coverage to support audits/reviews for OEM programs.
• Release Reporting: Produce per-release quality reports as benchmarks evolve and grow over time.
• Performance & Load Testing Support: Assist in load and performance testing using tools such as JMeter.
• Agile Testing & Continuous Improvement: Participate in Agile development cycles, contributing to continuous process improvements.
  
  

Qualifications

Basic Qualifications (Non-Negotiable)

• 3+ years of experience in software/product quality assurance.
• Confident usage of Bash and Python.
• Experience with manual testing and test case execution.
• Basic familiarity with automated testing frameworks such as Playwright or Selenium.
• Experience with testing web protocols (WebSocket message payload)
• Basic knowledge of load and performance testing (e.g., JMeter).
• Strong analytical and problem-solving abilities.
• Systems thinking.
• Familiarity with prompt evaluation, function-calling, and tool-use testing.
  
  

Preferred Skills

• Exposure to test automation scripting using Python.
• Experience with CI/CD integration.
• Familiarity with Conversational AI platforms and NLP-based applications.
• Experience with test management and result tracking.
• Experience testing conversational-agent systems and evaluating non-deterministic outputs.
  
  

General Requirements

• Eager to learn and grow in the field of QA and test automation.
• Strong collaborative mindset and ability to work effectively in a global team environment.
• Ability to adapt to fast-paced development cycles and evolving requirements.
  
  

Title

Location:

Duration

SUMMARY:

RESPONSIBILITIES;