Senior AI Evaluation & RLHF Specialist

Senior AI Evaluation & RLHF Specialist

Experience: 2+ Years
Employment Type: Contract 3 Months

Language Requirement: C1+ English
Location: Work from Office

Job Description

We are hiring Senior AI Evaluation & RLHF Specialists to work on advanced AI/LLM evaluation and alignment projects. The role involves evaluating AI-generated responses, preference ranking, RLHF data generation, multi-turn reasoning evaluation, and improving model outputs.

Key Responsibilities

• Evaluate and rank AI/LLM responses using detailed evaluation rubrics.
• Perform Side-by-Side (SxS) evaluations for factuality, helpfulness, safety, reasoning, coherence, and instruction-following.
• Identify errors and provide clear, fact-based evaluation rationales.
• Rewrite and improve AI-generated responses.
• Evaluate mathematical, logical, reasoning, and tool-use outputs.
• Review multi-turn conversations and agentic AI workflows.
• Create golden responses and edge-case examples.
• Participate in calibration sessions and maintain high inter-rater reliability.
• Follow evolving project guidelines and meet quality and productivity SLAs.

Required Skills

• 2+ years of hands-on experience in AI data annotation, AI model evaluation, RLHF, preference scoring, or AI training data.
• Experience working on complex AI/LLM evaluation projects.
• Strong analytical and critical-thinking skills.
• Excellent written English communication (C1 or above).
• Ability to provide objective, evidence-based evaluation rationales.
• Strong attention to detail and ability to follow complex guidelines.
• Comfortable working in a fast-paced, SLA-driven environment.

Preferred Skills

• Experience with RLHF, Reward Models, Preference Modeling, Generative AI, LLMs, NLP, or Prompt Engineering.
• Basic Python/scripting knowledge.
• Experience in AI quality/annotation QA or calibration.
• Experience mentoring junior evaluators or conducting secondary QA.
• Background in STEM, Law, Finance, Medicine, Humanities, or related fields.