SDET 2

We are seeking a proactive and detail-oriented SDET 2 to establish and scale the automated testing, evaluation, and quality infrastructure for our multi-agent procurement platform. In an agentic system, quality assurance extends beyond traditional API and UI testing to include non-deterministic AI evaluation, dynamic workflow verification, and reliability in multi-agent coordination.

You will design and build test harnesses to ensure that autonomous agents accurately plan, call external enterprise tools, adhere to safety guardrails, and execute high-stakes procurement workflows with deterministic consistency.

Responsibilities:

• Agentic and LLM Evaluation Infrastructure: Build and maintain automated evaluation (evals) frameworks (using tools like DeepEval, Ragas, Langfuse, or custom harnesses) to measure agent reasoning accuracy, tool-calling precision, hallucination rates, and prompt regressions.
• Workflow and State Machine Testing: Design automated test suites for complex, asynchronous, long-running workflows validating state persistence, checkpointing, retries, and Human-in-the-Loop (HITL) pause/resume logic across workflow engines (e. g., Temporal, Celery/Redis).
• End-to-End Automation: Develop and scale robust test automation frameworks across modern Python backend microservices, async APIs (FastAPI), web interfaces (Playwright/Cypress), and enterprise integration touchpoints (ERP mock layers).
• Synthetic Data and Scenario Generation: Generate synthetic enterprise procurement datasets, edge cases, and adversarial prompt injections to stress-test agent guardrails, schema enforcement (Pydantic), and fallback systems.
• Performance and Load Testing: Conduct latency benchmarking, rate-limit testing, and load testing across multi-agent graphs to identify bottlenecks in LLM token consumption and parallel tool execution.
• CI/CD Integration: Embed automated test gates and regression evals seamlessly into CI/CD pipelines (e. g., GitHub Actions, GitLab CI) to prevent agent drift and API regressions before deployment.

Requirements:

• Experience: 4 to 7 years of software testing and automation experience, with strong exposure to backend distributed systems and modern AI/LLM-enabled applications.
• Strong Python Proficiency: Deep hands-on coding expertise in Python (PyTest, AsyncIO, Requests/HTTPX) for building custom test frameworks and automation utilities.
• API and Integration Testing: Advanced knowledge of REST/gRPC API testing, schema validation, contract testing, and mocking asynchronous third-party enterprise services.
• LLM Quality Fluency: Practical understanding of LLM evaluation metrics (groundedness, context precision, answer relevancy, safety guardrails) and familiarity with agentic architectures (ReAct, multi-agent state graphs).
• UI Automation: Experience with modern browser automation frameworks (Playwright, Cypress, or Selenium).

Preferred Qualifications:

• Prior experience testing durable execution platforms (e. g., Temporal) or distributed state engines.
• Exposure to enterprise SaaS, fintech, or procurement and supply-chain domains.