QA (AI products)

Who We Are Escalent is an award-winning data analytics and advisory firm that helps clients understand human and market behaviors to navigate disruption. As catalysts of progress for more than 40 years, our strategies guide the world’s leading brands. We accelerate growth by creating a seamless flow between primary, secondary, syndicated, and internal business data, providing consulting and advisory services from insights through implementation. Based on a profound understanding of what drives human beings and markets, we identify actions that build brands, enhance customer experiences, inspire product innovation and boost business productivity. We listen, learn, question, discover, innovate, and deliver—for each other and our clients—to make the world work better for people.

Why Escalent? Once you join our team you will have the opportunity to...

• Access experts across industries for maximum learning opportunities including Weekly Knowledge Sharing Sessions, LinkedIn Learning, and more.
• Gain exposure to a rich variety of research techniques from knowledgeable professionals.
• Enjoy a remote first/hybrid work environment with a flexible schedule.
• Obtain insights into the needs and challenges of your clients—to learn how the world’s leading brands use research.
• Experience peace of mind working for a company with a commitment to conducting research ethically.
• Build lasting relationships with fun colleagues in a culture that values each person

Role Overview We are seeking a QA Support Engineer to help ensure high-quality releases for AI products developed for research and consulting workflows. This role will drive structured and exploratory testing across product experiences, validate output quality and workflow reliability, and surface issues that matter in real-world usage. The ideal candidate brings strong QA fundamentals, a practical understanding of research workflows, and the ability to evaluate whether AI-driven features are accurate, useful, traceable, and fit for purpose in a fast-paced, distributed environment.

Responsibilities

• Execute structured manual and exploratory testing for AI products used in research and consulting workflows
• Test end-to-end user journeys, prompt flows, and agentic workflows to identify functional issues, quality gaps, and failure modes
• Evaluate AI-generated outputs for accuracy, relevance, consistency, usability, and alignment to intended business and research use cases
• Apply working knowledge of research methodologies and consulting workflows to determine what to test, what good output looks like, and where risks are most likely to appear
• Use Langfuse for tracing, feedback capture, and LLM-as-a-judge evaluations to support systematic quality assessment and continuous improvement
• Design and execute test scenarios covering edge cases, regressions, hallucinations, grounding issues, prompt sensitivity, and model behavior drift
• Build and maintain evaluation datasets and repeatable test scenarios to support regression testing, benchmark comparisons, and release readiness
• Support human-in-the-loop evaluation by partnering with product and domain experts to review outputs, calibrate quality standards, and improve evaluation criteria over time
• Validate prompt changes, model updates, and workflow revisions through version-aware testing to identify performance gains, regressions, and unintended behavior shifts
• Test retrieval-augmented and grounded experiences for retrieval quality, answer faithfulness, citation accuracy, and appropriate source usage where applicable
• Assess non-functional quality dimensions such as latency, reliability, consistency, and cost-awareness alongside functional correctness
• Validate completed tickets against acceptance criteria, user expectations, and quality standards for both deterministic and AI-driven functionality
• Log clear, reproducible defects and quality issues with strong evidence, including steps, examples, prompts, outputs, and severity assessment
• Maintain QA documentation, evaluation frameworks, test cases, release checklists, and quality feedback loops across products

Required Skills

• 5+ years of experience in software QA, ideally across SaaS, workflow, or AI-enabled products
• Experience testing AI-powered applications, including prompt-driven features, workflow automation, or LLM-based user experiences
• Working knowledge of research processes and deliverables, with the ability to assess whether outputs are credible, useful, and aligned to research and consulting needs
• Strong manual and exploratory testing skills, with sound judgment on risk-based test design and coverage
• Understanding of modern QA practices for AI products, including evaluation frameworks, failure mode analysis, hallucination detection, output quality assessment, and regression testing for model-driven behavior
• Experience using Langfuse or similar tools for tracing, feedback instrumentation, observability, and LLM-as-a-judge evaluations
• Ability to document issues clearly and collaborate effectively with product managers, engineers, and domain experts across distributed teams
• Resourceful, self-directed, and adaptable in ambiguous or fast-changing environments, with a strong bias toward practical quality improvement

Desirable Skills

• Familiarity with creating evaluation datasets, golden sets, or benchmark scenarios for repeatable testing of AI product quality
• Experience with prompt testing, version comparison, and validation of model or workflow changes across releases
• Understanding of retrieval-augmented generation quality checks, including retrieval relevance, grounding, faithfulness, and citation validation
• Awareness of non-functional AI quality concerns such as latency, reliability, consistency, and cost-performance trade-offs
• Exposure to Playwright, Cypress, Selenium, or similar automation frameworks is a plus
• Experience with GitHub Projects, Jira, or similar workflow and defect management tools Explore our Careers and Culture page to learn more about the people behind the brand: https://escalent.co/careers\-and\-culture/