AI or ML Engineer - Python, QA Testing

We are looking for an AI Quality Engineer to help ensure our agentic AI systems are accurate, reliable, secure, and safe. You will define and execute a quality strategy spanning manual testing, test automation, release evaluation, and production monitoring. You will build an automation framework for agent responses, retrieval, tool use, and multi-step workflows, and work with engineering and product teams to turn failures into lasting improvements.

• Define and execute a risk-based test strategy covering functional, API, integration, end-to-end, regression, performance, reliability, safety, and security testing for AI agents
• Build and maintain automated evaluation and test frameworks for prompts, model outputs, retrieval, tool calls, agent handoffs, and multi-step workflows
• Perform manual exploratory and scenario-based testing of new capabilities, ambiguous requests, edge cases, and real-world failure modes
• Create and maintain representative test datasets, expected outcomes, and evaluation rubrics. Use deterministic assertions, LLM-as-judge, and human review where each is appropriate
• Run quality checks for every release and for material prompt, model, tool, or data-source change. Integrate repeatable tests into CI/CD pipelines as release gates
• Monitor production quality using measures such as task completion, answer accuracy, groundedness, hallucinations, tool-call success, latency, cost, and user feedback
• Detect drift and quality regressions, set alert thresholds, investigate failures, and expand test coverage based on production findings
• Evaluate risks including prompt injection, sensitive-data exposure, bias, and unauthorized or unsafe agent actions
• Publish quality dashboards and release-readiness reports. Partner with engineering, product, and SRE teams on defect triage, root-cause analysis, and corrective actions
• Comply with all applicable Company policies, procedures, and business directives, changes including those relating to work location, team assignments, work schedules, and flexible work arrangements

• Bachelor's degree in Computer Science or a related field, or equivalent practical experience
• 5+ years of experience in software quality engineering or test automation, including 2+ years testing AI/ML, generative AI, or LLM-based systems
• Hands-on experience evaluating LLM applications with test datasets, rubrics, groundedness or relevance checks, and a combination of automated and human review
• Experience designing and executing manual and automated functional, API, integration, end-to-end, regression, and risk-based tests
• Experience integrating automated quality checks into CI/CD pipelines and testing APIs, databases, and distributed applications
• Experience using logs, traces, dashboards, alerts, and user feedback to investigate production quality issues
• Solid Python skills and experience building or extending test automation frameworks, such as Pytest or Playwright
• Working knowledge of QA methods, defect lifecycle management, Agile delivery, and the software development lifecycle
• Understanding of RAG, prompt engineering, tool use, and multi-step agent workflows
• Demonstrated solid analytical, problem-solving, communication, and collaboration skills

• Experience with evaluation or observability tools such as LangSmith, Langfuse, Arize, DeepEval, Ragas, Vertex AI evaluation, or OpenTelemetry
• Experience with GCP, Vertex AI, or Gemini; experience testing applications on AWS or Azure
• Experience with JavaScript/TypeScript or Java, Selenium, advanced Playwright framework development, or browser testing platforms such as BrowserStack or Sauce Labs
• Experience with adversarial testing, responsible AI, and the OWASP Top 10 for LLM Applications
• Experience with A/B testing or canary evaluation of prompt and model changes
• Experience testing conversational AI, multi-agent or MCP-enabled solutions, event-driven services, or API orchestration
• Experience using AI to assist with test generation, failure analysis, or quality reporting
• Experience with performance and load testing tools such as JMeter or LoadRunner, or operational tools such as Dynatrace or SonarQube
• Experience with cloud migration, data migration, or data reconciliation testing
• Healthcare or other regulated-industry experience, including knowledge of HIPAA, data privacy, and security requirements. Experience with claims, eligibility, prior authorization, provider, contact center, or digital health systems