Senior Analyst - AI QA

Shape the Future with Dun & Bradstreet

At Dun & Bradstreet, we believe data has the power to create a better tomorrow. As a global leader in business decisioning data and analytics, we help companies worldwide grow, manage risk, and innovate. For over 180 years, businesses have trusted us to turn uncertainty into opportunity. We're a diverse, global team that values creativity, collaboration, and bold ideas. Are you ready to make an impact and help shape what's next Join us! Explore opportunities at dnb.com/careers.

The Role: We are looking for a highly skilled AI Tool / Agent Testing Engineer to evaluate, validate, and ensure the reliability of AI agents, AI automation tools, and agentic workflows used across our analytics platform. This role blends test engineering, GenAI understanding, Python/PySpark proficiency, and agent development lifecycle knowledge.

You will work closely with Data Science, AI Engineering, and Platform teams to ensure that AI agents behave predictably, safely, and in alignment with business and compliance requirements.

Key Responsibilities:

1. Agent & AI Tool Testing

• Design and execute test strategies for LLM agents, multi agent workflows, and automation tools.

• Validate reasoning paths, tool calls, workflows, and guardrails.

• Assess regression, functionality, performance, safety, and hallucination risks.

2. Agent Development Lifecycle (ADLC)

• Partner with AI engineers on prompts, knowledge sources, skills, and tool integrations.

• Validate interoperability with APIs, databases, vector stores, and orchestration frameworks.

• Ensure accuracy, consistency, tool-call reliability, trace quality, and guardrail adherence.

3. GenAI & Workflow Validation

• Test RAG systems for grounding and factual correctness.

• Validate sequential, loop, and parallel agent workflows.

• Ensure compliance with AI governance and security standards.

4. Test Automation Frameworks

• Build Python/PySpark utilities to automate scenarios, input generation, metrics, and trace analysis.

• Develop reusable test harnesses for agent evaluation pipelines.

5. Documentation & Reporting

• Produce test plans, scenario libraries, coverage reports, and defect logs.

• Deliver insights to Data Science & Engineering teams to improve reliability.

• 5 - 8 years of overall experience in software engineering, data science, or AI/ML development, with at least 3+ years focused on AI/LLM/GenAI testing or agent-based systems.


• Python expertise in scripting, automation, and debugging.


• Strong PySpark experience in distributed testing, data validation, and pipeline testing.


• Hands-on knowledge of GenAI concepts, including LLMs, prompting, context management, RAG pipelines, agent tool-calling, and multi-agent orchestration.


• Experience with agent development and deployment frameworks such as LangGraph, AutoGen, CrewAI, Copilot Studio Agent SDK, and Vertex AI/OpenAI agent frameworks.


• Solid understanding of agent architecture covering skills, tools, connectors, memory, guardrails, and observability.


• Familiarity with modern GenAI/agent evaluation frameworks such as Langsmith evaluation, AutoGen agent-behavior assessment utilities etc. for benchmarking reliability, grounding, tool-use correctness, and multi-agent performance.


• Strong foundation in functional and regression testing, scenario and edge-case testing, LLM safety and hallucination testing, and workflow validation.


• Experience creating evaluation datasets and defining success criteria for AI behavior.


• Good understanding of API testing frameworks, data engineering concepts, cloud workflow execution (Azure/GCP/AWS), and CI/CD pipelines for test automation.

All Dun & Bradstreet job postings can be found at https://jobs.lever.co/dnb. Official communication from Dun & Bradstreet will come from an email address ending in @dnb.com.

Notice to Applicants: Please be advised that this job posting page is hosted and powered by Lever, a subsidiary of Employ Inc. Your use of this page is subject to Employ's Privacy Notice and Cookie Policy, which governs the processing of visitor data on this platform.

We may use artificial intelligence (AI) tools to support parts of the hiring process, such as reviewing applications, analyzing resumes, or assessing responses. These tools assist our recruitment team but do not replace human judgment. Final hiring decisions are ultimately made by humans. If you would like more information about how your data is processed, please visit https://bit.ly/3LMn4CQ.