Senior Automation Quality Analyst

Role & responsibilities

• E2E testing of multi-format document processing (PDFs, Word, PPT, images/screenshots) and parsing accuracy (OCR/Vision extraction).
• Performing standard test activities: acceptance criteria refinement, test scenario creation, execution, reporting, defect management, and regression testing.
• Validation of RAG retrieval quality, including hybrid search (semantic + BM25 keyword search), metadata filtering, and retrieval scoping.
• Verification of probabilistic AI outputs for answer grounding, hallucination risk, source references, and confidence scores.
• E2E testing of human-in-the-loop content workflows (AI draft generation, confidence score thresholds, reviewer approvals, and vector DB publishing sync).
• API and enterprise integration testing (portals, chatbots, ticketing systems) across deterministic and non-deterministic flows.
• Validation of data protection controls, including PII scanning and redaction.
• Support QA automation at both API and UI levels for critical data pipelines.

Preferred candidate profile

• Experience working with Azure-based systems is a strong plus.
• Experience with Boost.ai.
• Experience testing enterprise integrations (CRMs, SAP, SharePoint, Jira).
• Experience with UI automated testing.

Perks and benefits