BENGALURU · FULLTIME
Senior AI Evals Engineer
RUBRIK INDIA
Bengaluru · onsite · Posted 2d ago
Sourced from
Undisclosed8–13 yrsfulltimeBengaluru
Your match
?
Sign in to see your match score, skill gaps & tailored resume.
Section · 01
About this role
Job Summary
Key Responsibilities
Required Qualifications
Education & Experience
Technical Skills
Tech Stack
Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.
The Senior AI Evals Test Engineer is a hands-on technical role responsible for defining, developing, and executing robust evaluation frameworks and test strategies for AI agents built using the Google Agent Development Kit (ADK) or similar sort of agentic frameworks. This role bridges AI development and reliable deployment - ensuring our agents are safe, ethical, effective, and meet high-quality performance standards. A significant focus of the role is on Responsible AI and Safety Evals, including adversarial testing, red-teaming, bias detection, and toxicity measurement. The engineer will design scalable evaluation pipelines, build synthetic test environments, and work closely with AI/ML Engineers, Researchers, and policy teams to ensure agents are production-ready and compliant with regulatory and ethical standards. The role is 70% Automation and 30% Manual Testing, and sits within the Applied AI: Test Engineering team as a key technical authority.
Key Responsibilities
- Design and build agent evaluation frameworks - create end-to-end infrastructure for evaluating LLM-based agents across correctness, latency, cost, safety, and task completion rate
- Build synthetic testing environments and simulation strategies to stress-test AI agents under diverse real-world conditions, with specific focus on agents built on Google ADK and related Google AI/ML services (e.g., Vertex AI, Gemini models)
- Build golden datasets and test harnesses - curate domain-specific evaluation sets for Rubriks product verticals (data security, SaaS protection, cyber recovery, compliance)
- Establish eval-as-a-service - build and enhance centralized evaluation platform that any IT team members can self-serve to test their agents against standardized and custom benchmarks
- Automate regression and drift detection - implement CI/CD-integrated eval pipelines that catch regressions on every model or prompt change before production deployment
- Define eval taxonomies and rubrics - establish structured scoring criteria for agentic workflows (tool use accuracy, multi-step reasoning, hallucination rate, policy adherence, error recovery)
- Instrument production observability - build real-time monitoring for agent quality in production: success rates, user corrections, fallback triggers, escalation patterns
- Benchmark and select models - run structured model comparisons (cost, quality, latency trade-offs) to inform model selection decisions across agent use cases
- Analyze and interpret evaluation results , providing clear, actionable quality and safety reports to stakeholders and development teams, with specific focus on risk metrics, trend analysis, and mitigation recommendations
- Define comprehensive end-to-end QA strategies - functional, integration, regression, and user acceptance testing (UAT) - tailored for conversational and goal-oriented AI agents with tool-use, memory, and multi-turn reasoning capabilities
- Develop and maintain detailed test artifacts (test plans, test cases, test scenarios, automated test suites) covering agent features, tool invocation, memory persistence, and reasoning accuracy across evolving model versions
- Identify, document, prioritize, and track defects , performance regressions, and alignment failures using Jira, collaborating closely with Applied AI Engineers to analyze root causes and validate fixes
- Evaluate new technologies and patterns (e.g., new tools for Agent Evals), run proofs-of-concept, and influence the platform roadmap
- Partner with policy and legal teams to ensure agent evaluations systematically cover regulatory compliance, fairness objectives, and emerging AI governance requirements
- Provide technical leadership to engineers , including solutioning, design reviews, code reviews, pair programming, and mentoring of junior and contract engineers
- Collaborate cross-functionally - work closely with ML engineers, product teams, platform engineers, and security to embed eval-driven quality gates into the agent development lifecycle
Required Qualifications
Education & Experience
- Bachelors or Masters degree in Computer Science, Information Systems, Engineering, or equivalent practical experience
- 8+ years of overall software Test engineering experience , with 5+ years focused on test engineering, QA automation, or developer tooling, and demonstrated ability to build evaluation systems across multiple AI/ML domains, conversational agents, or Large Language Models (LLMs)
- 2+ years working with ML/AI systems, building , testing, or evaluating models in production environments, including hands-on experience with LLM-based applications, prompt engineering, RAG pipelines, or agentic workflows
- 2+ years as a senior/lead, owning evaluation strategy, quality frameworks, and technical decisions for complex AI agent systems in an enterprise product environment
- 1+ years in designing or operating evaluation frameworks specifically for generative AI / LLM agents, including benchmarking, red-teaming, and production quality monitoring in the AI ADLC model
Technical Skills
- Deep hands-on experience with LLM evaluation - familiarity with frameworks and libraries such as LangSmith, DeepEval,LLM as a Judge, Promptfoo, RAGAS, Hugging Face evaluation tooling, custom eval harnesses
- Strong understanding of agentic AI patterns - tool use, function calling, multi-step planning, ReAct/CoT reasoning chains, retrieval-augmented generation (RAG)
- Production experience with eval pipelines - building automated, CI-integrated evaluation infrastructure (not just notebooks)
- Proficiency in Python - strong scripting and tooling skills; experience with pytest, async patterns, and data processing at scale
- Familiarity with Google Cloud Platform (GCP) or any Cloud Service relevant to AI/ML (e.g., Vertex AI) and experience integrating testing into MLOps workflows
- Statistical literacy - ability to design experiments, interpret eval metrics with appropriate confidence, and avoid common pitfalls (overfitting to benchmarks, metric gaming)
- Enterprise product sensibility - understanding of how AI quality translates to customer trust in a B2B/enterprise context (security, compliance, reliability SLAs)
- Excellent communication - ability to present eval results and quality trade-offs to engineering leadership, articulate risks and safety findings to non-technical audiences, and drive alignment on ship/no-ship decisions
- Hands-on experience with CI/CD pipelines and Git-based workflows , including automated testing, static analysis, deployment automation, and environment configuration for our DevTestOps
Tech Stack
- Languages: Python(Must), TypeScript, Java (Optional)
- Test Automation: Selenium, Playwright
- LLM Providers: Anthropic Claude, OpenAI, Azure OpenAI, open-source models
- Eval Tooling: DeepEvals(Must), LLM as a Judge (Must), Promptfoo/RAGAS/LangSmith/ Custom harnesses
- Agent Evals Frameworks/libraries: LangGraph, Claude Agent SDK, custom orchestrators
- Infrastructure: DevOps (Basic), Kubernetes, CI/CD (Jenkins/GitHub Actions)
Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.
Sourced from naukri
Let the agent run this one for you.
Tailored resume, auto-apply, and referral lookup — in under 2 minutes.
Section · 02
Skills
Test CasesSDKAgileScrumOpen sourcePythonSQLmanual testingSeleniumdata analysis
Section · Company
About RUBRIK INDIA
RUBRIK INDIA
Software Product
822
employees
2014
12 years old
Palo Alto,California
United States
₹27.9L PA avg
Avg at RUBRIK INDIA
About
Provides cloud data management solutions for hybrid cloud environments, focusing on data availability, recovery, and ransomware protection.
Industries
Software ProductInternet
Employee ratings
10 reviews
3.2/ 5
Culture
3.1
Career growth
2.7
Work-life
3.0
Employees rate it well for
Compensation & Benefits37Overall Company32
Find them on