LLM Evaluation Expert – Computer Science | $32/hr Remote

Position: Computer Science - Domain Experts (Pay Per Task)

Type: Short-term Contract

Compensation: $250 per task upon approval

Estimate task duration: 6-8 hours per task

Location: Remote

Commitment: Up to 40 hours/week with 4 hours overlap with PST

Duration: 3 Months

Start Date: Immediate

Role Responsibilities

• Design advanced computer science problems that test AI model performance across conceptual and reasoning-heavy topics, including systems design, algorithms, cybersecurity, and other advanced CS areas
• Develop clear, step-by-step reference solutions with rigorous logic, well-structured explanations, and defensible reasoning
• Evaluate AI-generated responses for correctness, reasoning quality, completeness, clarity, and alignment with the intended problem requirements
• Identify flaws, gaps, edge cases, and near-miss reasoning patterns in AI outputs
• Collaborate with researchers to refine benchmarks across undergraduate, graduate, and PhD-level computer science topics
• Provide structured feedback to improve problem quality, benchmark reliability, and model evaluation standards
• Apply computer science concepts creatively to challenging reasoning and problem-solving tasks

Requirements

• Candidates with a Ph.D., completed or currently pursuing, in Computer Science or closely related fields are encouraged to apply
• Strong expertise in advanced computer science reasoning and problem-solving across domains such as:
• Algorithms, Systems Design, Cybersecurity, Databases, Networks, Theory of Computation, Operating Systems, and related CS fields
• Ability to design challenging conceptual problems that require deep understanding rather than straightforward recall or coding
• Strong analytical, critical thinking, and problem-solving skills
• Excellent written communication skills, with the ability to explain complex technical ideas clearly and logically

Application Process

• Apply/Easy Apply and check email for Interest Check Form
• Online Assessment
• Delivery Review / Interview
• Onboarding Process