Maths LLM Evaluation Expert

Responsibilities:

• Write scientific problem specifications consisting of one main problem and a minimum of 3 sub-problems, all logically connected and progressively building toward the main problem solution.
• Implement verified golden solutions in Python with complete unit test coverage.
• Design discriminative test cases that clearly differentiate correct from incorrect model outputs.
• Run QC validation checks on the Turing Central Task Platform (CTP), including Tier 1 structure checks and Tier 2 quality rubrics.
• Iterate on tasks based on QC feedback to meet Pass@K evaluation criteria across multiple LLM judges (GPT, Gemini, and Nemotron).
• Maintain high output quality with a low rework rate, targeting consistent L1 approval on first submission.
• Participate in sync calls for reviews, feedback sessions, and project standups during overlap hours.

Requirements:

• Master's or PhD in Mathematics. Strong Python programming skills with experience in scientific computing.
• Ability to write rigorous, well-posed scientific problems with clear constraints and expected outputs.
• Attention to detail; tasks must meet strict rubrics for well-posedness, test case discriminativeness, scientific correctness, and determinism.
• Prior experience in AI data annotation, research, or scientific writing.
• Familiarity with LLM evaluation frameworks or coding benchmarks.
• Experience with libraries such as NumPy, SciPy, SymPy, or domain-specific scientific tools.
• Published research or academic project experience in a STEM domain.