Senior GenAI Consultant
Ernst Young · Mumbai · onsite · Posted 44d ago
Undisclosed3–8 yrsfulltimeSourced from
About a minute. You'll make an account on the way through.
About this role
The Opportunity
We are seeking an experienced AI QA Engineer with 59+ years of professional experience and deep expertise in AI testing, quality engineering, validation frameworks, and enterprise AI solution assurance. The ideal candidate should possess strong hands-on experience validating AI systems, Agentic AI workflows, RAG applications, conversational AI solutions, and enterprise-scale AI platforms.
The individual will play a key role in ensuring the quality, reliability, security, safety, and performance of AI systems while collaborating with AI engineers, architects, developers, and business stakeholders to establish robust quality assurance practices across AI and Generative AI implementations.
Important note: This role requires professional implementation experience in relevant AI technologies. Personal portfolio projects, cloned demos, certifications, or tutorial-based builds are helpful, but are not sufficient on their own.
Application guidance: Candidates should be able to clearly explain at least two relevant AI implementations, including the business use case, their direct contribution, the technical approach, and the outcome delivered.
Your key responsibilities - Design, develop, and execute comprehensive testing strategies for AI, GenAI, Agentic AI, and Agentic RAG systems.
- Validate Large Language Model outputs for accuracy, consistency, relevance, safety, and compliance requirements.
- Develop test frameworks and automation suites for AI services, copilots, AI assistants, and enterprise AI applications.
- Design and execute evaluation frameworks for RAG systems, retrieval pipelines, vector search solutions, and knowledge-based AI applications.
- Perform functional, integration, regression, performance, reliability, and scalability testing of AI-powered applications.
- Develop automated testing frameworks using Python to validate AI workflows, APIs, models, and orchestration pipelines.
- Validate prompt engineering approaches, agent workflows, tool usage patterns, and reasoning capabilities in Agentic AI systems.
- Execute AI model testing across conversational AI, multimodal AI, and intelligent automation solutions.
- Perform AI penetration testing, adversarial testing, prompt injection testing, jailbreak testing, and security validation activities.
- Assess AI solutions for bias, hallucinations, content quality, safety risks, and responsible AI compliance requirements.
- Validate integrations between AI systems, enterprise applications, APIs, databases, and cloud platforms.
- Utilize RAG evaluation frameworks such as Ragas to measure retrieval quality, answer faithfulness, relevance, and response effectiveness.
- Develop testing dashboards, quality metrics, defect reporting processes, and continuous quality improvement practices.
- Collaborate closely with AI engineers and architects to identify quality risks and improve operational reliability.
- Contribute to quality standards, governance controls, testing accelerators, reusable frameworks, and best practices.
- Stay current with advancements in AI testing, evaluation methodologies, Agentic AI, AI security, and emerging quality assurance technologies.
- 59+ years of total experience, including 3+ years of directly relevant hands-on experience in AI/ML, applied AI, or GenAI systems.
- Proven experience building production, pre-production, or enterprise pilot AI applications.
- Strong hands-on depth in at least 3 of the following areas, with meaningful ownership in at least 2:
- LLM application development.
- Retrieval pipelines.
- Agent or tool orchestration.
- Evaluation, guardrails, or quality systems.
- Model adaptation or fine-tuning.
- Cloud deployment or optimization of AI applications.
- Conversational AI.
- Multimodal AI.
- Backend engineering for AI systems.
- Strong software engineering foundation with ability to build reliable, maintainable AI services.
- Able to clearly explain direct contributions to delivered AI implementations.
- Bachelors/masters degree in computer science, Data Science, Engineering, or related field.
- Strong understanding of Large Language Models (LLMs), Generative AI, Agentic AI, and Agentic RAG systems.
- Hands-on experience validating and testing AI-powered applications and enterprise AI platforms.
- Experience developing AI testing strategies, validation frameworks, and automated test suites.
- Strong expertise in AI Model Testing, functional testing, regression testing, integration testing, and end-to-end quality validation.
- Experience performing AI penetration testing, adversarial testing, prompt injection testing, jailbreak testing, and vulnerability assessments.
- Strong understanding of prompt engineering validation, reasoning evaluation, and response quality assessment techniques.
- Experience implementing evaluation frameworks and quality measurements for Agentic AI workflows and AI agents.
- Hands-on experience with Ragas and similar evaluation frameworks for assessing retrieval quality, answer correctness, faithfulness, relevance, and grounding.
- Experience validating Agentic RAG systems, retrieval pipelines, and knowledge-based AI applications.
- Experience testing AI integrations across APIs, enterprise applications, databases, and cloud services.
- Strong understanding of AI observability, evaluation metrics, monitoring, and performance benchmarking.
- Experience working with Microsoft Azure AI Platform and Azure AI
Sourced from naukri
Skills
Automationai assistantsAzure Aiapisai servicesautomated testingai modelai technologiesai governanceAnalyticsai securityAI testingAPI TestingApplication DevelopmentAi PlatformAi SolutionsMicrosoft AzureArtificial IntelligenceAutomation Solutions
About Ernst & Young
Ernst Young
Management Consulting
44k+
employees
2002
24 years old
Gurgaon / Gurugram, Haryana
India
₹15.3L PA avg
Avg at Ernst & Young
About
At EY, our purpose is Building a better working world. The insights and quality services we provide help build trust and confidence in the capital markets and in economies the world over.
Industries
Management Consulting
Employee ratings
10 reviews
3.4/ 5
Culture
1.9
Career growth
2.3
Work-life
2.5
Employees rate it well for
Job Security36Skill Development35
Find them on