Lead Eval Engineer

The Lead Eval Engineer is the top individual contributor in the eval engineering function: a principal-level technical authority and a peer to the most senior engineers in the company. Where application architects define how agents are built, you define how they are proven. You set the technical vision for the measurement layer: the harness architecture, the acceptability frameworks, the failure-attribution methodology, the historical scoring backbone, and the autonomy trust gates that the entire engineering organization operates against.

Requirements:

• 10+ years of software engineering experience, with 4+ years of senior IC leadership in evaluation, measurement infrastructure, or AI quality systems.
• Demonstrated track record of setting technical direction adopted at organizational scale (standards, frameworks, or architectural patterns), not just tools.
• Deep, hands-on expertise in agentic AI evaluation: multi-step reasoning assessment, tool orchestration accuracy, judge design and calibration, adversarial evaluation, and production behavioral monitoring.
• Principal-level engineering depth in distributed systems, event-driven architectures, and multi-tenant platforms, sufficient to design measurement for the full complexity of a multi-service, policy-governed AI platform.
• Proven ability to define acceptability for nondeterministic systems and get an engineering organization to operate against it.
• Track record of influencing architectural decisions across organizational boundaries, shaping the scope rather than just executing within it.
• Full ownership experience of AI quality strategy in production at large scale: incident response, drift analysis, and model-change governance.
• Statistical fluency at depth: experimental design, sample size determination, confidence intervals, and significance testing, sufficient to design measurement systems that produce statistically defensible signals rather than metrics that look precise but cannot be trusted.
• Ability to translate evaluation evidence into business terms for VP/C-level stakeholders, turning score distributions and coverage gaps into "should we ship this? " and "what does this risk mean for our customers? ".