AI Test Engineer

Must have 3+ years in software QA/testing, with hands-on senior-level experience evaluating LLM-based or agentic AI systems (not just traditional QA)

Direct experience with eval frameworks/tools LangSmith strongly preferred; also acceptable:

• Ragas, DeepEval, PromptFoo, Braintrust, Langfuse
• Strong understanding of RAG architecture and how to independently evaluate retrieval quality vs. generation quality
• Practical familiarity with MCP (Model Context Protocol) or comparable agent tool calling frameworks
• Strong Python skills for building test harnesses, scoring pipelines, and automation
• Experience designing evaluation approaches for non-deterministic systems (probabilistic scoring, not binary pass/fail)
• Comfortable working directly with client engineering teamsstrong communication, able to explain quality risk to both technical and business stakeholders;