Senior Evaluation Engineer

Responsibilities:

• Design and own evaluation systems for agent domains. Take architectural responsibility for eval coverage across one or more capability domains: multi-turn conversation evals, tool-selection accuracy, goal completion, policy compliance, and safety boundary probing. Coverage spans behavioral equivalence, SLOs, idempotency, chaos-as-test, context and retrieval quality, and tool/dependency failure modes; each is a distinct failure surface that functional correctness evals alone do not cover. Define what "evaluated" means for the domain, not just execute a checklist.
• Define acceptability. Translate product spec documents and product intent into explicit, measurable acceptance criteria and thresholds for nondeterministic behavior: what score, on what dimensions, at what confidence constitutes "working close enough to intent"? Set acceptance bars across quality, safety, latency, and cost (SLOs), own these thresholds, and evolve them with evidence.
• Build harnesses in parallel with development. Architect and implement the simulation environments, mock tool infrastructure, scenario suites, and scoring pipelines for new agent capabilities while they are in development, so evaluation readiness ships with the feature.
• Own the scoring and calibration pipeline. Design LLM-as-judge systems calibrated against human baselines, covering responsible AI dimensions (bias, fairness, toxicity) alongside behavioral quality. Own rubric design, measure inter-rater agreement, guard against judge bias and reward hacking, and ensure automated scores are meaningful signals rather than vanity metrics. Own the statistical methodology for your domain's metrics (sample sizes, confidence intervals, significance thresholds): teams will dispute regressions, and your job is to have a defensible measurement standard, not just a number.
• Attribute failures across the execution graph. Lead structured deep dives through every node of an agent's path (prompt, context assembly, retrieval, tool calls, model reasoning) to isolate where behavior diverges from intent. Deliver findings to engineering as specific, evidenced recommendations: fix this prompt, restructure this context, or accept this as a model capability limit and design around it.
• Operate production quality monitoring. Build and own sampled scoring of live traffic: rolling baselines by capability, drift and anomaly detection, and rollback signal generation. Distinguish real regressions from noise. Run new agent and prompt versions in shadow against live traffic before promotion: replay on historical data catches known failures; shadow evaluation catches the distribution shift that synthetic datasets miss.
• Prove your corpus has teeth. Run mutation testing against your domain's eval corpus: inject faults into known-good builds and verify the evals catch them. Own the incident-to-eval pipeline for your domain with a defined turnaround, from production incident through root cause to a merged eval case, so the same failure class never ships twice.
• Publish decision-grade trend lines. Own the dashboards that show score histories, improvement/regression trajectories, and coverage across your domains: historical data legible to the entire organization, from engineers to executives, answering "Are we getting better or worse? "
• Gate what ships. Own the integration between eval pipelines and deployment, with thresholds that block a rollout when behavior drops below defined acceptability. Run A/B comparisons and canary evaluations on prompt and model changes before full rollout. Eval owns the measurement and the ship/no-ship recommendation; overrides are explicit leadership decisions on risk tolerance, never silent bypasses.
• Build evaluation for governed and policy-aware agents. Design evaluation approaches that account for externalized business policies, deterministic control paths, and context provenance. Test not just whether the agent produced the right output, but whether it respected the right authority boundaries.
• Lead adversarial evaluation. Design and execute adversarial suites (prompt injection, boundary escape, policy bypass, and confirmation-logic circumvention) and convert every discovered vulnerability into permanent eval coverage.
• Mentor and raise the bar. Guide Eval Engineers I and II on error analysis, judge calibration, and harness design. Review their work for coverage gaps, calibration errors, and false confidence in green metrics.
• Influence engineering decisions. Engage with application engineering, AI platforms, and product teams as an authoritative voice on evaluation. Hold the line when changes ship without adequate eval coverage, and surface systemic quality risks through data, not opinion.

Requirements:

• 5+ years of software engineering experience, with at least 2 years building evaluation, measurement, or quality systems for AI/ML products.
• Engineering-parallel fundamentals: you can design, build, and operate evaluation infrastructure as production-grade distributed systems (services, pipelines, storage, and observability), not scripts and spreadsheets.
• Deep hands-on LLM evaluation experience: LLM-as-judge design and calibration, human annotation pipelines, golden dataset governance, and systematic error analysis from raw traces.
• Demonstrated ability to define acceptance criteria for nondeterministic behavior, turning fuzzy notions of quality into thresholds an organization can operate against.
• Experience diagnosing agentic systems across the full execution path: able to attribute a failure to prompt, context, tooling, or model capability, and defend the attribution with evidence.
• Experience wiring evaluation into deployment flows as a hard gate, blocking bad changes before they reach production rather than reporting on them after.
• Statistical fluency applied to evaluation: sample size design, confidence intervals, and significance testing, enough to defend a regression call when a team disputes it.
• Proficiency in Python and/or Java at the level needed to build and maintain a shared eval framework others depend on.
• Experience operating in environments where AI failures have real business or compliance consequences.

Preferred:

• Adversarial evaluation experience: red teaming, injection taxonomies, boundary analysis for AI systems in regulated contexts.
• Familiarity with policy-driven AI architectures (externalized business rules, per-action risk tiers, consent-aware execution flows) and how to verify conformance to them.
• Distributed systems background (event-driven architectures, microservices, eventual consistency) and an understanding of how these shape evaluation design.
• Fluency with trace-level observability (OpenTelemetry-style spans, structured logs) and using production telemetry to reconstruct agent decision paths.
• Experience with eval platforms and tooling: LangSmith, Langfuse, Braintrust, Ragas, Arize, W& B, Promptfoo, DeepEval, or internal equivalents.
• Experience evaluating voice, speech, or multimodal agent interactions.
• Experience with mutation testing or corpus-strength verification, or with shadow/blue-green evaluation of new versions against live traffic.
• Exposure to automotive, financial services, healthcare, or other regulated industries.