Associate Vice Presidentsenior - Lead MLOps Engineer

Responsibilities:

• Own day-to-day operations for multiple AI services (LLM apps, ML models, RAG services, agents), meeting SLO/SLA targets for availability, latency, cost, and quality.
• Maintain operational documentation (runbooks, escalation matrices) and participate in the on-call schedule.
• Coordinate incident response including triage, mitigation, and root cause analysis.
• Manage releases and deployments (including canary deployments and blue/green rollouts) and make sure test and production environments remain aligned.
• Follow the company's formal change management process and maintain required documentation for audits.
• Design and maintain full observability across AI systems covering both service health metrics and model quality indicators.
• Track and act on KPIs such as latency, throughput, error rates, safety filter triggers, factual accuracy issues, user acceptance rates, and operational cost per 1K tokens.
• Monitor data and model drift and take action when behaviour changes.
• Build dashboards and alerts; adjust thresholds based on trends.
• Run A/B tests and use controlled side-by-side evaluations to validate improvements.
• Operate and improve continuous integration and continuous delivery workflows for machine learning and large language model components, including pipelines, model registries, and version control.
• Trigger retraining or prompt updates when quality drops, new data becomes available, or policies change.
• Coordinate any required data labelling tasks.
• Manage rollbacks and controlled promotions between models to ensure reliability and auditability.
• Maintain full traceability for decisions, model versions, and deployments.
• Ensure compliance with data protection rules, regional data storage requirements, and secure secret handling.
• Integrate AI-safety measures such as content filtering and misuse detection.
• Maintain documentation describing how each model works, how it was trained, and any risks or limitations.
• Collaborate with Security, Risk, and Governance teams to provide required evidence for audits and reviews.
• Work closely with testing teams on release preparation, environment planning, and resilience testing.
• Contribute AI-specific checks such as prompt quality regression tests, bias/safety evaluations, and retrieval quality assessments.
• Collaborate with data engineering, application engineering, and AI-engineering teams to optimise operations and control costs.
• Help build shared components, templates, and best practices that other AI teams can reuse.
• Ensure all operational and risk-related documentation is kept up to date.

Requirements:

• Experience in Site Reliability Engineering, DevOps, Test Operations, Platform Engineering, or MLOps ideally with responsibility for production systems.
• Hands-on experience with container orchestration platforms such as OpenShift or Kubernetes; CI/CD tools; and observability solutions (metrics, logs, traces, dashboards).
• Practical experience operating machine learning or large language model workloads: serving models, running pipelines, maintaining model registries, and evaluating performance.
• Strong understanding of evaluating ML/LLM quality and safety, detecting drift, running A/B tests, and managing controlled deployments.
• Proficiency in Python and familiarity with one ML platform (MLflow, Vertex AI, Kubeflow, etc. ).
• Ability to communicate clearly and produce high-quality documentation (runbooks, change requests, incident reports).
• Experience with retrieval-based AI systems (vector databases, retrieval pipelines), prompt engineering, or multi-step AI agents.
• Background in AI security or risk assessments (safety testing, privacy, auditability).
• Knowledge of enterprise change management processes and business continuity testing.