QA Engineer

Responsibilities:

• Design the end-to-end quality strategy for Lighthouse and the AI systems built on it, covering APIs, web experiences, data and context pipelines, agent and workflow orchestration, enterprise integrations, permissions, and deployment paths.
• Build and maintain automated API, integration, contract, UI, and regression suites with meaningful CI/CD quality gates; decide what should be validated with deterministic assertions and what requires evaluation-based testing.
• Test agentic behavior beyond happy-path prompts: multi-step reasoning and workflows, tool selection and use, structured outputs, retrieval and context grounding, state and memory, human approvals, retries, fallbacks, and recovery from partial failure.
• Create evaluation datasets, golden cases, rubrics, thresholds, and error taxonomies for model and agent behavior so prompt, model, context, tool, and workflow changes can be measured before release.
• Actively test high-risk failure modes, including hallucinated or unsupported outputs, prompt injection, unsafe tool actions, permission bypass, tenant leakage, malformed inputs, schema drift, rate limits, timeouts, duplicate or delayed events, and degraded upstream systems.
• Validate enterprise integrations and data quality across databases, warehouses, documents, APIs, files, events, and MCP; verify authentication, authorization, freshness, provenance, traceability, and behavior under unreliable dependencies.
• Own release readiness, exploratory testing, defect triage, root-cause analysis, and production quality feedback. Turn escaped defects into durable regression coverage and reusable quality standards across projects.

Requirements:

• 4+ years in QA, test automation, or quality engineering for production software, with strong ownership of backend/API, integration, or enterprise systems from development through release.
• Strong hands-on automation skills in Python, JavaScript/TypeScript, or a comparable language, including API testing, SQL/data validation, test design, and CI/CD integration.
• Experience testing distributed or asynchronous systems involving queues, webhooks, background jobs, retries, idempotency, state transitions, eventual consistency, and failure recovery.
• Hands-on experience testing LLM, AI, or agentic systems, or strong applied evidence of working with evaluation, grounding, tool use, structured outputs, prompt/model regression, safety, and guardrails.
• Ability to translate ambiguous business workflows into risk-based test scenarios, acceptance criteria, edge cases, and coverage that reflects how real users and enterprise systems behave.
• Strong debugging and collaboration skills: comfortable working with engineers and product and delivery teams, reading logs and traces, reproducing failures, and driving root causes rather than only reporting symptoms.