AI Product Experience Harness

Role- AI Product Experience Harness

Location- Noida & Hyderabad

Experience- 15-21 Years

Role Summary:

Build and run the automated harness that validates the delivered product against the experience specification and the AI compliance/guardrail specification, so validation is a pipeline, not a headcount. This role executes automated checks against criteria that the Experience lead (functional/visual/accessibility) and the Compliance lead (safety/red-team/regulatory) define — it does not independently set the bar for what pass means; it makes pass mechanically checkable and evidence-backed.

Key Responsibilities:

Functional & experience validation:

• Build automated end-to-end test suites (Playwright or Cypress) covering critical user flows
• Build visual-regression testing into the pipeline (screenshot diffing against approved baselines)
• Build automated accessibility test suites (WCAG 2.1/2.2 AA conformance checks)
• Convert every experience-spec acceptance criterion into an executable check; flag criteria that cannot be automated and require manual review — this is a real skill, not a formality: a spec clause like feels responsive needs to become a measurable threshold (e.g., time-to-first-token, time-to-interactive) before it's testable

AI/agentic-specific validation:

• Build automated checks for agentic UI states defined by the Experience spec streaming/incremental rendering correctness, progress-indicator accuracy during multi-step agent tasks, correct rendering of failure/retry/escalation states
• Build and run automated red-team regression suites (using tools defined by/coordinated with the Compliance role) so that known jailbreak/injection/PII-leakage test cases are re-run on every release, not just at initial audit
• Automate LLM output evaluation checks — factual consistency, refusal-behavior conformance, format/schema conformance — as repeatable pipeline gates rather than one-off manual review

Operations:

• Run the harness against CTO org deliveries; produce pass/fail evidence per release, feeding the audit/evidence pack the Compliance role maintains
• Maintain test data, environments, and fixtures, including synthetic/adversarial prompt datasets for AI-specific test suites
• Publish a live quality dashboard for the product org, showing pass rates, flaky-test rates, and trend lines release over release
• Integrate all of the above into CI/CD so validation runs automatically pre-merge and pre-release, not on request

Must-Have Requirements:

• 12+ years full-stack engineering / SDET with real automation depth in Playwright or Cypress — not just test-writing, but framework/harness architecture
• Has built a test harness from scratch for at least one product — be ready to describe the architecture decisions, not just the tests inside it
• CI/CD pipeline integration experience (GitHub Actions, Jenkins, GitLab CI, or equivalent) — tests must run automatically, gating merges/releases
• Can read a spec and identify untestable clauses — work sample: given a sample spec, identify which acceptance criteria can't be automated as written and propose a measurable alternative
• Has built or integrated visual-regression testing (Percy, Chromatic, Applitools, BackstopJS, or Playwright's native visual comparison
• Practical experience automating evaluation of LLM/agentic outputs as part of a test pipeline — using named tools such as Ragas, DeepEval, Promptfoo, or a custom eval harness — be ready to describe one specific automated check and what it caught
• Working familiarity with red-team test automation for LLM applications (Garak, PyRIT, or equivalent), sufficient to integrate Compliance-authored test cases into the CI pipeline as repeatable regression checks — this role executes and automates red-team suites; it does not independently design the adversarial test strategy (that's the Compliance role's ownership)

Preferred:

• Accessibility testing depth beyond automated checks (manual screen-reader testing, WCAG AAA familiarity)
• Performance testing (Lighthouse, k6, WebPageTest) integrated into the same pipeline
• Direct prior collaboration with an AI-safety/compliance function, having built the automation layer for someone else's test design