Software Engineer

We are looking an experienced AI Test Engineer to drive quality assurance for AI-powered applications, including LLM models, conversational bots, autonomous agents, and MCP-enabled systems. The ideal candidate will have strong expertise in Playwright with .NET, API/UI automation, AI/LLM validation, and CI/CD automation. The role involves building scalable test frameworks, validating AI responses, testing agent workflows, and ensuring the reliability, security, and performance of GenAI solutions.

Role :AI Test Engineer

Location: PAN India

Experience Range: 5 to 12years

Key Responsibilities:

• Design and execute test strategies for LLM-powered applications, AI chatbots, conversational AI, and autonomous agents.
• Develop and maintain Playwright (.NET) automation frameworks for UI and API testing.
• Validate AI model outputs for:
• Accuracy
• Relevance
• Hallucinations
• Consistency
• Safety and Guardrails
• Test MCP (Model Context Protocol) endpoints and MCP-enabled agent workflows.
• Automate API testing and integrate test suites into CI/CD pipelines.
• Create test datasets, evaluation metrics, and AI validation frameworks.
• Perform regression, functional, integration, and end-to-end testing for AI systems.
• Collaborate with developers, data scientists, and product teams to ensure quality delivery.
• Monitor and troubleshoot AI system failures, agent interactions, and model behavior.
• Implement quality gates within Azure DevOps/GitHub Actions/Jenkins pipelines.

Required Skills:

AI / GenAI Testing

• LLM Testing
• Prompt Validation
• AI Response Validation
• Conversational Bot Testing
• Autonomous Agent Testing
• MCP (Model Context Protocol)
• Hallucination Detection
• AI Safety & Guardrail Testing
• RAG Validation

Automation Testing

• Playwright with .NET (C#)
• UI Automation
• API Automation
• Postman / REST APIs
• Selenium (Good to Have)

Programming

• C#
• .NET Core / ASP.NET
• JavaScript / TypeScript (Good to Have)

DevOps & CI/CD

• Azure DevOps
• GitHub Actions
• Jenkins
• Docker (Preferred)

Other Skills

• Agile/Scrum
• SQL Databases
• Test Framework Design
• Version Control (Git)