AI Test Engineer (LLM and AI QA)

**About the Role**

We're looking for an AI Tester to evaluate the quality, safety, and reliability of AI-powered products, including LLM applications, agents, and ML-driven features. You'll design test strategies for non-deterministic systems and help ensure our AI behaves accurately, safely, and consistently in production.

**Responsibilities**

- Design and execute test plans for LLM, RAG, and agentic workflows

- Build evaluation datasets (golden sets, edge cases, adversarial prompts)

- Assess output quality: accuracy, relevance, hallucination, tone, and consistency

- Perform red-teaming: prompt injection, jailbreaks, data leakage, and bias testing

- Validate tool-calling, multi-step agent behavior, and guardrails

- Automate regression and evaluation pipelines (e.g., LLM-as-judge, scoring harnesses)

- Track metrics such as latency, cost per request, and failure rates

- Log, triage, and reproduce defects with clear prompts, inputs, and model versions

- Partner with ML engineers, product owners, and developers to improve prompts and models

- Document findings and contribute to release go/no-go decisions

**Required Qualifications**

- 4+ years in QA, test automation, or data/ML evaluation

- Solid understanding of how LLMs work (prompts, context windows, temperature, embeddings)

- Experience testing APIs and writing automated tests (Python preferred)

- Strong analytical skills and attention to detail

- Clear written communication for documenting non-deterministic behavior

**Preferred Qualifications**

- Experience with evaluation tools (e.g.,LangSmith, Ragas, DeepEval)

- Familiarity with RAG pipelines, vector databases, and agent frameworks

- Knowledge of AI safety, responsible AI, and compliance practices

- Cloud platform experience (Azure)

- Background in test automation frameworks