AI Quality & Model Evaluation Lead

Job Title: AI Quality & Model Evaluation Lead
Department: Engineering
Location: Bengaluru
Employment Type: Full-Time
Experience: 5+ Years
Role Level: Technical Lead

Job Overview

We are looking for an experienced AI Quality & Model Evaluation Lead to drive quality assurance, healthcare audit validation, AI/ML evaluation, and production readiness for AI-powered healthcare solutions. The ideal candidate will bring strong expertise in software testing, healthcare revenue cycle management (RCM), statistical quality methodologies, AI/ML model evaluation, and Generative AI assessment.

The role requires a hands-on leader who can establish quality standards, define audit methodologies, evaluate AI model performance, perform healthcare coding audits, and provide independent Go/No-Go recommendations for production deployments.

Required Skill Set

• QA & Software Testing (Functional, Integration, Regression, API, Data Validation, Test Planning and Execution)
• AI/ML Model Evaluation (Model Accuracy, Precision, Recall, F1-Score, Sensitivity, Specificity, Benchmark and Ground-Truth Validation)
• LLM & Generative AI Evaluation (Response Accuracy, Relevance, Groundedness, Hallucination Detection, Factuality, Consistency, and Prompt Evaluation)
• Healthcare RCM (Medical Coding Validation, Clinical Documentation Review, AR Denial Management, Prior Authorization, and Audit Methodology)
• Statistical Sampling & Audit (Random/Stratified Sampling, Sample Size Determination, Quality Audits, Confidence Levels, and Error Analysis)
• Data Analysis & SQL (Data Extraction, Validation, Reconciliation, Quality Analysis, and Complex SQL Queries)
• Python (Data Analysis, Statistical Analysis, Test Automation, and Evaluation Scripts)
• LLM Evaluation Tools (DeepEval, Ragas, LangSmith, TruLens, Promptfoo, or Equivalent Frameworks)
• API Testing (Postman, REST APIs, Request/Response Validation, Authentication, and API Automation)
• Test Management & Defect Tracking (JIRA, TestRail, TestLink, Zephyr, Xray, Defect Lifecycle Management, and Reporting)
• QA Automation (Automation Frameworks, Regression Automation, Data Validation, and Test Execution Optimization)
• Quality Metrics & Reporting (Test Coverage, Defect Trends, Model Accuracy, Audit Results, Quality KPIs, and Dashboards)
• Go/No-Go & Risk Assessment (Release Readiness, Quality Risk Evaluation, Acceptance Criteria, and Data-Driven Decisions)
• Agile & QA Process (Sprint Planning, Estimation, QA Processes, Continuous Improvement, and Cross-Functional Collaboration)
• Leadership & Stakeholder Management (Team Mentoring, Task Planning, Communication, Issue Escalation, and Stakeholder Coordination)

Qualifications

• Bachelor's Degree in Computer Science, Statistics, Data Science, Health Informatics, Healthcare Management, Engineering, or a related field.
• Master's Degree is an added advantage.

Candidate Profile

5+ years of experience in Software QA, Healthcare Audit, or AI/ML Quality Evaluation
Strong hands-on experience in QA and Testing activities
Experience evaluating AI/ML Models and Generative AI/LLM Applications
Healthcare RCM, Medical Coding Validation, and Clinical Documentation Audit experience
Strong SQL, Data Analysis, and Python skills
Experience with LLM Evaluation Tools such as DeepEval, Ragas, or LangSmith
Ability to make independent Go/No-Go decisions based on quality, audit, and statistical evidence
Excellent leadership, stakeholder management, and communication skills