AI QA Engineer

Design, develop, and maintain automated test frameworks using Python.
Perform API testing and data validation to ensure accuracy, consistency, and reliability.
Evaluate AI/ML models for functional accuracy, performance, and reliability.
Test Generative AI and LLM-based applications, including RAG pipelines.
Evaluate prompt effectiveness and response quality across different use cases.
Perform hallucination detection and groundedness testing to verify that AI-generated responses are supported by source information.
Validate Responsible AI requirements, including relevant safety, security, and risk controls.
Conduct security testing to identify vulnerabilities and potential risks in AI-powered applications.
Perform performance testing to assess response times, scalability, and system reliability.
Integrate automated tests into CI/CD pipelines to enable continuous quality validation.
Work with cloud AI platforms, including Azure AI Foundry, Azure OpenAI, and Azure Machine Learning, to validate AI solutions.
Collaborate with development, data science, and engineering teams to identify defects and improve overall AI solution quality.

Must-Have Skills:
Strong expertise in Python-based test automation.
Hands-on experience in API testing and data validation.
Experience in AI/ML model evaluation and testing.
Experience testing Generative AI and LLM-based applications.
Strong understanding of RAG testing and validation.
Experience in prompt and response quality evaluation.
Knowledge of hallucination detection and groundedness testing.
Experience in Responsible AI validation and security testing.
Experience in performance testing.
Experience integrating automated testing into CI/CD pipelines.
Hands-on experience with cloud AI platforms such as Azure AI Foundry, Azure OpenAI, and Azure Machine Learning.
Good-to-Have Skills:
Experience developing reusable Python-based test automation frameworks for AI applications.
Experience evaluating the quality, reliability, and consistency of LLM-generated responses.
Experience validating RAG retrieval quality and response relevance.
Familiarity with AI safety, responsible AI practices, and security validation techniques.
Experience automating AI quality checks within continuous integration and deployment workflows.
CGI is an equal opportunity employer. In addition, CGI is committed to providing accommodation for people with disabilities in accordance with applicable legislation. Please let us know if you require reasonable accommodation due to a disability during any aspect of the recruitment process, and we will work with you to address your needs.
Life at CGI:
It is rooted in ownership, teamwork, respect and belonging. Here, youll reach your full potential because

Come join our team, one of the largest IT and business consulting services firms in the world.

• Prompt Engineering
• Python
• Test automation
• Test data management
• Test Design

That same commitment to fairness extends to how we use technology. To support our recruitment team, AI tools may be used to help assess applications though they never replace human judgement. All hiring decisions remain entirely in the hands of our recruitment professionals.