Technical AI Evaluation Specialist

Technical AI Evaluation Specialist Code & Software Engineering

Department: AI/LLM Operations
Experience: 2+ Years
Employment Type: Contract 3 Months
Location: Remote
Language Requirement: C1-equivalent English proficiency

Job Description

We are hiring Technical AI Evaluation Specialists to evaluate and benchmark advanced AI/LLM models focused on code generation, software engineering, technical reasoning, and UI development.

The role requires strong software development and coding knowledge to assess AI-generated code for correctness, performance, security, architecture, and UI fidelity. You will also perform side-by-side model evaluations and provide detailed, evidence-based technical feedback to support AI model improvement and RLHF workflows.

Key Responsibilities

• Evaluate AI-generated code for syntax, correctness, execution accuracy, performance, complexity, and coding best practices.
• Review code across multiple programming languages and frameworks.
• Perform Text-to-Code and Side-by-Side (SxS) evaluations using detailed technical rubrics.
• Evaluate Image-to-Code/UI generation against design mockups, wireframes, and screenshots.
• Assess front-end implementations for responsiveness, layout accuracy, functionality, and UI fidelity.
• Identify bugs, edge cases, logical errors, race conditions, memory issues, and security vulnerabilities.
• Review AI-generated solutions against data structures, algorithms, software architecture, and engineering best practices.
• Conduct structured code reviews and provide clear technical feedback.
• Write detailed, evidence-based rationales explaining evaluation scores, errors, and potential fixes.
• Identify model failure patterns and emerging quality issues.
• Participate in guideline calibration and evaluation benchmark development.
• Collaborate with research engineers, prompt engineers, and AI/ML teams to improve evaluation frameworks.
• Maintain high accuracy, consistency, and productivity while meeting project SLAs.

Required Qualifications

• 2+ years of professional software development experience OR a strong degree in Computer Science / Software Engineering with demonstrable coding proficiency.
• Strong programming and debugging skills.
• Hands-on experience with HTML5, CSS, React, JavaScript, or TypeScript.
• Experience translating UI designs, mockups, or screenshots into responsive front-end code.
• Strong understanding of code review, software testing, debugging, and software engineering practices.
• Prior experience in AI model evaluation, AI data annotation, RLHF, or technical evaluation is highly desirable.
• Strong written and verbal English communication with C1-equivalent proficiency.
• Ability to write clear, structured, technically defensible evaluation rationales.
• Strong analytical and problem-solving skills.
• Ability to understand and evaluate unfamiliar codebases and frameworks.