AI Safety & Evaluation Lead

As the AI Safety and Evaluation Lead at ConveGenius, you will own the process ensuring every AI system deployed is accurate, safe, unbiased, secure, and compliant. Given that ConveGenius serves children and underserved communities across India, this role carries direct responsibility for learner trust, content safety, and regulatory standing. A core dimension of this role is understanding student psychology, since our entire user base is school and college-going students, and AI outputs must be designed to handle sensitive or negative queries (e. g., distressing searches by students) in a safe, age- appropriate manner that does not surface harmful information.

Responsibilities:

• Design and own the AI evaluation framework: benchmark construction, automated and human-in- the-loop eval pipelines.
• Lead red-teaming and adversarial testing across deployed models: prompt injection, jailbreaking, edge case identification.
• Define AI safety standards, responsible AI policies, and child-safe content guardrails for deployment at scale.
• Build and maintain evaluation tooling: RAGAS, LangSmith, PromptBench, EleutherAI lm- evaluation-harness.
• Conduct systematic bias detection and fairness measurement across outputs, especially for regional and linguistic variation.
• Manage AI risk assessments and translate safety findings into actionable model or prompt changes.
• Establish AI governance documentation: model cards, audit trails, safety reports for internal and regulatory stakeholders.
• Embed evaluation and safety checkpoints into the model release pipeline in collaboration with engineering.
• Track and apply evolving regulatory requirements: EU AI Act, NIST AI RMF, India AI policy frameworks.
• Measure and report model accuracy, grounding fidelity, and response quality across deployed AI systems through systematic evaluation.
• Build and maintain knowledge of AI governance and compliance standards, operationalising responsible AI practices into daily engineering workflows.

Requirement:

• Strong experience in AI/LLM safety, evaluation, and risk assessment.
• Expertise in designing and executing model evaluation frameworks and test methodologies.
• Experience in red-teaming, adversarial testing, and identifying model vulnerabilities.
• Strong understanding of bias, fairness, hallucination, and content safety in AI systems.
• Experience measuring model accuracy, grounding, and response quality.
• Knowledge of AI governance, compliance, and responsible AI practices.
• Familiarity with frameworks such as NIST AI RMF and AI regulatory standards; understanding of student psychology and age-appropriate AI output design.
• Strong analytical and problem-solving skills with experience working on production AI systems.

Preferred Skills:

• AI regulations: EU AI Act, NIST AI RMF, DPDP Act (India) and their operational implications.
• Understanding of fine-tuning processes and how alignment, tax, and behaviour drift occur post-training.
• Explainable AI: SHAP, LIME, attention visualisation, interpretability for non-technical audiences.
• Child safety standards and age-appropriate AI design principles for deployment to minors.
• Cybersecurity fundamentals: threat modelling, adversarial ML, model extraction attack patterns.