Senior AI Evaluation & RLHF Specialist

ROLE OVERVIEW & OBJECTIVE:

We are expanding our core evaluation and are looking to hire experienced AI Evaluation and Reinforcement Learning from Human

Feedback (RLHF) Specialists into our high-complexity alignment pipeline. This cohort directly powers reward model training, preference

modeling, complex multi-turn reasoning, and critique-and-rework loops for top-tier frontier models. We are sourcing experienced

practitioners from leading AI data platforms who require zero ramp-up time, demonstrate rigorous adherence to dynamic rubrics, and deliver

ironclad, defensible evaluation rationales.

KEY RESPONSIBILITIES & CORE WORKFLOWS:

• High-Precision Preference Data Generation: Author high-fidelity preference datasets and rank multi-turn model completions to

optimize reward model loss functions.

• Side-by-Side Evaluations: Execute deep SxS comparative evaluations across factuality, helpfulness, harmlessness, logical

coherence, and instruction-following.

• Improve Model output: Rewrite sub-optimal model outputs and break down complex chain-of-thought failures into structured,

corrective learning signals.

• Multi-Step Reasoning & Agentic Auditing: Evaluate step-by-step mathematical, logical, and tool-use trajectories generated by

autonomous AI agents.

• Golden Sets & Worked Examples: Construct reference 'golden responses' and edge-case worked examples to train and calibrate

downstream evaluation cohorts.

• Inter-Rater Reliability (IRR) Leadership: Participate in daily calibration huddles, driving consensus on ambiguous prompts and rapid

guideline iterations.

Mandatory Requirements:

• Direct AI Platform Experience: 2+ years of hands-on data

annotation, model evaluation, or RLHF preference scoring on

recognized platforms

• Complex Task Track Record: Demonstrable mastery in multidimensional rubric scoring, and chain-of-thought verification.
• Written Communication: English proficiency with proven

discipline in articulating clear, fact-backed, objective rationales

under tight SLAs.

• Operational Agility: High tolerance for rapidly evolving project

guidelines, shifting evaluation criteria, and fast-paced delivery

sprints.

Preferred Qualifications:

• Domain Specialization: Advanced educational background in

STEM, Law, Finance, Medicine, or Humanities.

• Technical Literacy: Basic Python scripting, prompt engineering

knowledge, or familiarity with loss functions and reward modeling

concepts.

• Leadership Experience: Prior experience mentoring junior raters,

conducting secondary QA audits, or serving as a task calibration

lead.