Manager -Foundation model (Model Evaluation and MLOps) || India Ai

About the Role:-

We are looking for a Manager Foundation Model (Model Evaluation & MLOps) to support the IndiaAI project. The role will focus on model evaluation and benchmarking, MLOps tracking, compute utilisation review, deployment readiness, technical monitoring and MIS.

The position will work closely with AI/ML teams, startups, technology partners, evaluation agencies and compute providers to track technical progress and ensure timely delivery of evaluation, benchmarking and deployment-readiness activities.

Key Responsibilities :-

• Coordinate model evaluation and benchmarking activities as per approved technical parameters and timelines.
• Track model performance metrics including accuracy, precision, recall, F1 score, BLEU, ROUGE, latency, throughput, hallucination rate, robustness, bias, toxicity, safety and multilingual capability.
• Monitor model development across training, fine-tuning, evaluation, validation, deployment readiness and post-deployment monitoring.
• Track MLOps activities including model versioning, experiment tracking, evaluation records, CI/CD readiness, API readiness, monitoring dashboards and retraining requirements.
• Review compute utilisation related to GPU, TPU, HPC and cloud infrastructure, including distributed training, storage, inference infrastructure and cost optimisation.
• Coordinate with startups, technology partners, evaluation agencies and compute providers for timely submission of technical reports and benchmarking results.
• Prepare technical MIS dashboards, evaluation summaries, compute review notes and project progress updates.
• Identify performance gaps, compute bottlenecks, deployment risks and MLOps-related dependencies.
• Support technical reviews and provide actionable insights to senior management on model performance, infrastructure utilisation and deployment readiness.

Required Experience & Skills

• Minimum 6 years of experience in AI/ML projects, model evaluation, MLOps, software systems, cloud projects, data platforms or technical programme management.
• Strong understanding of the AI/ML lifecycle, including model training, fine-tuning, evaluation, benchmarking, inference, deployment, monitoring and retraining.
• Understanding of model performance and evaluation metrics for machine learning and foundation/large language models.
• Familiarity with Git, MLflow, Weights & Biases, Docker, Kubernetes, CI/CD, APIs and cloud platforms.
• Understanding of GPU/TPU/HPC infrastructure, distributed training, inference infrastructure and compute utilisation.
• Ability to track technical deliverables, analyse performance gaps and prepare management-level technical reports and dashboards.
• Strong coordination and communication skills for working with startups, technology partners, evaluation agencies and technical teams.

Preferred Profile:-

Candidates with experience in foundation models, LLM evaluation, AI benchmarking, MLOps, model deployment, AI infrastructure, cloud/HPC environments or technical programme management will be preferred.

Key Competencies

Model Evaluation | AI/ML | Foundation Models | MLOps | Model Benchmarking | LLM Evaluation | Model Deployment | Model Monitoring | MLflow | Kubernetes | Docker | Cloud | GPU/TPU/HPC | CI/CD | Technical MIS | AI Infrastructure