Principal Software Engineer

Senior AI Test / Automation Engineer

Overview

Role: AI Test / Automation Engineer
Location: Bangalore India

Department: AI Engineering / Quality Assurance
Experience Level: Mid to Senior

We are looking for a highly motivated Senior AI Test / Automation Engineer to design and scale automated validation frameworks for AI/ML models, LLM-based applications, and agentic systems . This role is critical to ensure that AI solutions meet enterprise standards for quality, reliability, safety, and compliance before and after production deployment.

Key Responsibilities

• Build and maintain AI test automation frameworks for pre-qualification and continuous validation of models and agent workflows
• Develop comprehensive test suites , including:
  • Unit, integration, and end-to-end (E2E)
  • Functional, regression, performance, and safety testing
• Validate AI system behavior , including:
  • Non-deterministic LLM outputs
  • Hallucinations and edge cases
  • Multi-step agent decision-making
• Design and manage evaluation systems :
  • Golden datasets
  • Benchmarking pipelines (accuracy, latency, reliability)
• Automate testing within CI/CD pipelines for model updates, prompt changes, and tool integrations
• Implement observability and telemetry to enable traceability, monitoring, and audit readiness
• Collaborate cross-functionally with ML, MLOps, Product, and Security teams to define quality gates and release criteria
• Track and report quality KPIs , including test coverage, defect leakage, and system reliability
• Drive root-cause analysis and continuous improvement across the AI testing lifecycle

Required Skills

Core Engineering

• Strong programming skills in Python ; familiarity with Bash, TypeScript, or Go
• Experience with test automation frameworks such as PyTest, Playwright, Selenium, or Cypress
• Proficiency in CI/CD tools (GitHub Actions, Jenkins, GitLab CI)
• Experience with cloud platforms (AWS, Azure, GCP) and containers (Docker, Kubernetes)

AI / ML Agentic Systems

• Hands-on experience with LLM ecosystems (OpenAI, Anthropic, Bedrock)
• Familiarity with:
  • RAG architectures and vector databases (Pinecone, Weaviate)
  • Agent frameworks (LangChain, LlamaIndex, AutoGen)

AI Testing Techniques

• Experience with non-deterministic testing approaches (statistical assertions, tolerance thresholds)
• Knowledge of evaluation methods :
  • LLM-as-a-judge
  • BLEU, ROUGE, semantic similarity scoring
• Experience with prompt and agent regression testing
• Understanding of AI safety testing , including adversarial testing, bias/fairness validation, and jailbreak detection

Tooling (Preferred)

• AI testing observability tools: LangSmith, TruLens, Arize, Weights Biases
• Evaluation tools: DeepEval, Ragas, PromptFoo, Giskard
• Monitoring: Prometheus, Grafana, OpenTelemetry

Soft Skills

• Strong analytical and problem-solving skills
• Excellent communication and cross-functional collaboration
• Data-driven mindset with focus on quality KPIs
• Detail-oriented with a strong bias toward automation and scalability

Experience Requirements

• 7+ years in QA, SDET, or test automation engineering
• Proven experience building and scaling automation frameworks
• Hands-on experience with AI/ML systems or LLM-based applications

• Experience testing RAG pipelines or agentic workflows
• Owned end-to-end AI test strategy and architecture
• Defined quality metrics and release gates
• Delivered scalable validation pipelines for production AI systems
• Supported audit and compliance readiness

Preferred

• Experience in enterprise or regulated environments (SOC2, ISO 27001, etc. )
• Exposure to:
  • Shift-left testing practices
  • Production observability and monitoring
  • Chaos or resilience testing

Senior-Level Differentiators

Education

• Bachelor s or Master s degree in Computer Science, Software Engineering, or related field

Nice-to-have:

• ISTQB certification
• Cloud/ML certifications (AWS, Azure, GCP)
• AI testing certifications

What Success Looks Like

• AI systems that are accurate, reliable, and safe
• Fully automated test pipelines integrated into CI/CD
• Measurable improvements in defect leakage and model quality
• Strong observability and auditability across AI systems
• Scalable validation frameworks supporting rapid AI innovation