AI Automation Engineer

Position Description:

We are seeking an Automation Quality Engineer with strong hands-on experience in Java and/or Python automation frameworks and practical exposure to AI-driven testing. The ideal candidate will play a key role in validating intelligent applications, ensuring the quality, reliability, and performance of AI-enabled systems. This role involves testing across traditional automation frameworks as well as advanced AI workflows, including Large Language Models (LLMs), Retrieval-Augmented Generation (RAG), Vector Databases, and OpenAI APIs.

Your future duties and responsibilities

• Design, develop, and maintain automation test frameworks using Java and/or Python.
• Implement UI, API, and backend automation using tools such as Selenium, Playwright, REST Assured, PyTest, or similar.
• Validate LLM-based applications, ensuring prompt accuracy, response quality, hallucination detection, and consistency.
• Test and evaluate RAG pipelines, including document ingestion, embeddings, retrieval accuracy, and response relevance.
• Work with Vector Databases (FAISS, Pinecone, Chroma, Weaviate) to validate indexing, retrieval, and performance.
• Integrate and test applications using OpenAI APIs or similar LLM providers.
• Leverage LangChain (or equivalent frameworks) for testing chains, agents, tools, and workflows.
• Create automated test cases for AI workflows, model outputs, prompt versions, and inference pipelines.
• Implement data-driven and AI-assisted testing strategies.
• Collaborate with Data Scientists, Developers, and Product teams to define quality metrics for AI systems.
• Participate in CI/CD pipelines to enable continuous testing and enforce quality gates.
• Ensure compliance with security, performance, and scalability standards.

Required qualifications to be successful in this role

• Strong programming skills in Python.
• Hands-on experience with automation testing frameworks (Selenium, Playwright, REST Assured, PyTest, etc.).
• Practical experience with LLMs (OpenAI GPT, Azure OpenAI, or similar).
• Solid understanding and exposure to RAG architectures.
• Experience working with Vector Databases for embeddings and semantic search.
• Hands-on usage of OpenAI APIs (chat/completions, embeddings, etc.).
• Experience with LangChain for prompt chaining, memory, and tool usage.
• Strong knowledge of API testing, SQL, and data validation.
• Familiarity with CI/CD tools (Jenkins, GitHub Actions, Azure DevOps).
• Good understanding of SDLC, STLC, and Agile methodologies.

Good-to-Have Skills

• Certification in LLMs and/or Python.
• Exposure to AI-assisted testing strategies and emerging AI QA tools.
• Experience with performance testing of AI-enabled systems.
• Knowledge of cloud-based AI platforms and orchestration frameworks