Agent Evaluation & Instrumentation Engineer

Serve as the technical evaluator within the team, responsible for the design, calibration, and adjudication of evaluation suites and gate thresholds across agent archetypes. Lead gate reviews under the Team Lead s authority, own the regression-pack methodology for AI/ML model changes, and act as the technical custodian of evaluation quality and drift hygiene. 1. Design and maintain offline evaluation suites (golden sets, regression packs, adversarial/safety probes) and the continuous-evaluation scoring pipeline across archetypes. 2. Conduct operability gate reviews: assess evidence packs, reproduce evaluation results, and recommend go/no-go with documented findings. 3. Own the model-update regression pack methodology and adjudicate regression runs against archived baselines with AIML Operations team. 4. Calibrate gate thresholds against production reality and maintain evaluation drift hygiene (golden-set rotation, hold-out sets, judge calibration). 5. Produce the monthly quality report per agent: eval trends, failure-mode taxonomy, and defect clusters with reproduction traces. 6. Mentor members of the team when needed and review their work for quality and consistency. Category Essential for this role Good to have Education and Qualifications Bachelor s or Master s degree in Computer Science or a related field Work Experience Adversarial testing / red-teaming Non-Technical / Soft Skills - Management presentation Other Task-Specific Knowledge Understanding of telco customer intents and journeys - Responsible-AI and safety evaluation practices