PUNE · FULLTIME
Agent Evaluation & Instrumentation Engineer
Umanist NA
Pune · onsite · Posted 8d ago
Sourced from
Not disclosed8–11 yrsfulltimePune
Your match
?
Sign in to see your match score, skill gaps & tailored resume.
Section · 01
About this role
Agent Evaluation & Instrumentation Engineer
Location: Pune | Experience: 8–11 Years | CTC: Up to 33 LPA(576)
Notice Period: Immediate–45 Days | Preference: Local Pune Candidates
Interview: 2 Technical Rounds + Client Round + HR
MANDATE CRITERIA – NON-NEGOTIABLE
Agentic AI | RAG | Telco AI | AI Red-Teaming | Responsible AI & Safety | Judge Calibration | Management Presentations
Ideal Profile: AI/ML Evaluation + Python + Data Analysis + LLM/Agent Quality with strong communication and client-facing skills.
Skills: deepeval,quality thresholds,llm/agent evaluation design,python,llm/ml evaluation,telco customer intents,regression packs,technical documentation,statistical rigour,tracing,model regression,evaluation drift,promptfoo,ml/data/software engineering,golden sets,failure modes
Location: Pune | Experience: 8–11 Years | CTC: Up to 33 LPA(576)
Notice Period: Immediate–45 Days | Preference: Local Pune Candidates
Interview: 2 Technical Rounds + Client Round + HR
MANDATE CRITERIA – NON-NEGOTIABLE
- 6+ years in ML/Data/Software Engineering with strong evaluation/quality focus
- Hands-on LLM/ML evaluation experience
- Strong Python skills
- Experience with Promptfoo / DeepEval / custom evaluation frameworks
- Strong understanding of LLM/Agent evaluation design & statistical rigour
- Hands-on data analysis & metric interpretation
- Experience with tracing & observability tools
- Strong technical documentation, analytical & stakeholder-management skills
- Understanding of Telco customer intents & customer journeys
- 2+ years stability in an organization
- Design and maintain LLM/Agent evaluation suites, golden sets, regression packs & adversarial tests
- Build continuous evaluation & scoring pipelines
- Conduct quality/operability gate reviews and reproduce evaluation results
- Analyse model regression, failure modes, defects & evaluation trends
- Calibrate quality thresholds, judges & evaluation datasets
- Monitor evaluation drift and maintain hold-out/golden sets
- Prepare technical quality reports & evaluation findings
- Collaborate with AI/ML, Engineering & Operations teams on quality improvements
- Mentor team members and communicate findings to technical/client stakeholders
Agentic AI | RAG | Telco AI | AI Red-Teaming | Responsible AI & Safety | Judge Calibration | Management Presentations
Ideal Profile: AI/ML Evaluation + Python + Data Analysis + LLM/Agent Quality with strong communication and client-facing skills.
Skills: deepeval,quality thresholds,llm/agent evaluation design,python,llm/ml evaluation,telco customer intents,regression packs,technical documentation,statistical rigour,tracing,model regression,evaluation drift,promptfoo,ml/data/software engineering,golden sets,failure modes
Sourced from foundit
Let the agent run this one for you.
Tailored resume, auto-apply, and referral lookup — in under 2 minutes.
Section · 02
Skills
LLM ML evaluationML Data Software EngineeringTracing observability toolsDeepEvalTechnical DocumentationPythonPromptfoodata analysisstatistical rigour