Director, Research - AI Evals

This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Director, Research - AI Evals based in United States.

This is a high-impact leadership opportunity for an experienced research professional passionate about defining and measuring quality in AI-powered products. The role sits at the intersection of AI, product strategy, and user experience, shaping how next-generation intelligent features are evaluated and improved before reaching millions of users. You will establish evaluation frameworks, quality standards, and scalable processes that directly influence product decisions across the organization. Working closely with Product, Design, Engineering, and Data Science teams, you will lead the development of trusted evaluation systems that combine human judgment with automated methods. This position offers the chance to build a strategic capability from the ground up while guiding a small team in a highly collaborative, innovation-driven environment.

Accountabilities:: Define and own the evaluation strategy for AI-powered experiences, establishing quality dimensions, success metrics, and decision frameworks.

Build and maintain evaluation methodologies, including rubrics, benchmark datasets, golden sets, and model-based assessment approaches.

Partner with engineering teams to implement scalable evaluation pipelines, regression testing systems, and reproducible quality measurement processes.

Develop dashboards, reporting mechanisms, and executive-ready insights that support product prioritization and go/no-go decisions.

Champion consistent AI quality standards across teams and embed evaluation practices into the product development lifecycle.

Lead and mentor a small team responsible for executing AI evaluation initiatives in collaboration with internal stakeholders, contractors, and AI-assisted workflows.

Influence organizational strategy by translating complex evaluation findings into actionable recommendations for senior leadership.

Requirements

10+ years of experience in product research, applied research, product development, or related disciplines, including at least 2 years of people management experience.

Proven hands-on expertise evaluating AI and LLM-powered products, including human evaluation programs and automated evaluation methodologies.

Strong understanding of rubric design, benchmark creation, inter-rater reliability, and model-based assessment techniques such as LLM-as-a-judge approaches.

Experience combining qualitative and quantitative research methods to assess model performance and user impact.

Demonstrated ability to identify critical risks, prioritize ambiguous quality questions, and design effective evaluation strategies.

Strong stakeholder management and communication skills, with experience influencing executive and cross-functional teams.

Familiarity with AI evaluation tooling and infrastructure such as Braintrust, LangSmith, DeepEval, or similar platforms is considered a strong asset.

Experience building new functions, practices, or research disciplines from the ground up is highly desirable.

Additional background in product design, product management, data science, engineering, or front-end development would be advantageous.

Benefits

Competitive compensation package with annual base salary ranging from $258,000 to $348,000 USD , adjusted based on location, experience, and scope.

Equity participation program, offering long-term growth and ownership opportunities.

Comprehensive health, dental, and vision coverage.

Retirement plans with company contributions.

Generous paid time off, company recharge days, and family-friendly leave policies, including parental and reproductive support programs.

Mental health and wellness benefits designed to support overall well-being.

Learning and development stipend to encourage continuous professional growth.

Work-from-home allowance and cell phone reimbursement to support remote productivity.

Flexible and collaborative work environment with opportunities to contribute to cutting-edge AI innovation.

How Jobgether works: We use an AI-powered matching process to ensure your application is reviewed quickly, objectively, and fairly against the role's core requirements. Our system identifies the top-fitting candidates, and this shortlist is then shared directly with the hiring company. The final decision and next steps (interviews, assessments) are managed by their internal team. We appreciate your interest and wish you the best!  Why Apply Through Jobgether?    Data Privacy Notice: By submitting your application, you acknowledge that Jobgether will process your personal data to evaluate your candidacy and share relevant information with the hiring employer. This processing is based on legitimate interest and pre-contractual measures under applicable data protection laws (including GDPR). You may exercise your rights (access, rectification, erasure, objection) at any time.     #LI-CL1