Senior SDET - AI

The candidate will have responsibilities across the following functions:

Generative AI and LLM Evaluation:

• Build automated testing suites to detect hallucinations, bias, toxicity, and prompt injection vulnerabilities across LLM-powered products.
• Implement automated evaluations for RAG systems measuring context relevance, groundedness, and answer faithfulness using frameworks like RAGAS or DeepEval.
• Design test beds to validate multi-agent workflows, tool-calling accuracy, multi-step reasoning, memory, and autonomous decision loops.
• Build and run automated conversation simulations scripted and synthetic user journeys to stress-test agent behaviour across intents, edge cases, and multi-turn dialogue flows.
• Create prompt regression frameworks to assess how changes in system prompts, temperature, and sampling parameters impact output consistency.

Data Quality Assurance:

• Statistically validate AI data output distributions, precision/recall, and error pattern analysis to catch silent data quality failures before production.
• Programmatically audit data ingestion, transformation, and feature store pipelines for schema drift and data corruption.
• Validate vector DB indexing, embedding semantic similarity accuracy, and retrieval latency.
• Verify quality, diversity, and privacy compliance of synthetic datasets used for model training and evaluation.

Classical ML and Deep Learning Validation:

• Maintain automated suites tracking ML metrics: precision, Recall, F1 ROC-AUC and deep learning loss curves across model versions.
• Implement continuous monitoring scripts to detect data and concept drift on live inference endpoints.

Automation Engineering and CI/CD:

• Build and maintain scalable test automation frameworks for APIs, backend services, and model endpoints.
• Embed AI evaluation and data QA suites into MLOps and CI/CD pipelines so quality failures block releases automatically.
• Define and track AI quality KPIs and communicate release readiness to engineering and product teams.

The core requirements for the job include the following:

Core Programming:

• Python expert level; test automation, eval pipelines, data analysis (Pandas, NumPy, Pytest).
• SQL data output validation, ground truth querying, pipeline data quality checks.

GenAI and Evaluation:

• RAGAS / TruLens / DeepEval / Promptflow, etc LLM evaluation frameworks for measuring faithfulness, hallucination rate, and task success.
• LangChain / LangSmith / LlamaIndex agent workflow testing, prompt tracing, and LLM response debugging.
• OpenAI / Anthropic / Hugging Face APIs: direct LLM endpoint testing and output consistency validation.
• Vector DBs: retrieval quality testing, embedding validation, and latency benchmarking.
• Pandas / NumPy, etc.statistical analysis for output validation and error pattern investigation, data profiling, schema validation, and pipeline integrity checks.

API and Automation:

• Pytest modular, reusable test framework for AI eval and automation suites.
• Postman / REST Assured / Requests API contract validation and service-level integration testing.

MLOps and CI/CD:

• MLflow tracking model versions and eval runs to detect regressions across updates.
• Docker / GitHub Actions / Jenkins containerised test environments and deployment pipeline automation.

Observability:

• Grafana / Kibana / OpenTelemetry monitoring AI system health, output drift, and distributed tracing across agent pipelines.

Preferred Qualifications:

• Cloud AI Services: AWS Bedrock, Azure OpenAI, or GCP Vertex AI for testing managed model endpoints and cloud-deployed agents.
• MLOps Platforms: MLflow, Kubeflow, Weights & Biases, or Feast feature stores for experiment tracking and model governance.
• ML Frameworks: Scikit-learn, TensorFlow, or PyTorch familiarity for understanding model internals and validating training pipelines.
• Infrastructure as Code: Docker, Kubernetes, Terraform for managing containerised test environments at scale.
• UI Automation: Playwright or Cypress for end-to-end conversational AI application testing.
• Performance Engineering: Locust or JMeter for load testing heavy AI inference endpoints under peak traffic.
• Statistical Hypothesis Testing: t-tests, confidence intervals, significance testing to distinguish real quality signal from noise.
• Synthetic Data Generation using LLMs to generate diverse test cases and evaluation datasets at scale.