# VeroniQA: the AI assistant for API testing

> VeroniQA is an AI assistant for API testing, built on AgentQA. Give her your requirements, documents and links; she answers with cited sources, generates and runs API tests, and explains what she found.

- Try it: [https://veroniqa.vercel.app/talk/](https://veroniqa.vercel.app/talk/) (no sign-up, no API keys; a private workspace preloaded
  with a demo API is created for you)
- Watch: [Meet VeroniQA, 85 s video](https://veroniqa.vercel.app/videos/veroniqa-intro.mp4)
- Developers: [API guide and CLI](https://veroniqa.vercel.app/developers.md) · [OpenAPI](https://veroniqa.vercel.app/openapi.json)
- Source code: [https://github.com/Zapkid/agentqa](https://github.com/Zapkid/agentqa)
- Built by [Rowan Kendal](https://www.rowan-kendal.com/)

## Measured results

Measured on the simulated benchmark in the repository:

- **94%** of planted bugs found
- **47%** of the cost of the strongest model
- **6/6** performance defects caught
- **12** bugs planted in the demo API
- **10** cost levers, each measured
- **0** API keys needed to try it

## Features

### Answers with sources

Ask about your API and get answers grounded in your own documents, with numbered citations. Citations that do not match a retrieved passage are dropped, and "not found" is a real answer.

### Tests from your requirements

Say “run the tests”. AgentQA plans risk-ranked tests from the spec and your documents, runs them, and triages each failure with evidence.

### Knowledge you can trust

Upload Markdown, text, HTML or PDF, or add links. Links are fetched from public addresses only, and text that looks like a prompt injection is quarantined.

### Retrieval that fits the domain

Each project has its own vector store (Chroma) combined with keyword search (BM25), so field names and error codes are found as reliably as plain prose.

### Cost-aware by design

A cheap model routes each message; stronger models are used only where they change the result. Every step is traced with its tokens and cost.

### Fits how teams work

Use the browser, let coding agents call AgentQA over MCP, or gate pull requests on an eval and a performance smoke test in CI.

## How it works

1. **Create a project.** One folder per API, with its spec, settings and knowledge.
2. **Add what your team already has.** Requirement documents, PDFs and links to your docs.
3. **Ask, then run the tests.** Get cited answers, a test run, findings and a short report.

## Usage and cost

Four ways to run the same test job against the demo API, three runs each, on the simulated
benchmark. Costs are what the tokens would cost at each model's list price.

### Cost per test run

| strategy | $ per run (list price) | bugs found |
|---|---|---|
| S0 Strong model for everything | $0.473 | 92% |
| S1 Cheap model for everything | $0.037 | 47% |
| S2 Static role routing | $0.173 | 67% |
| S3 Smart dispatcher (AgentQA) | $0.221 | 94% |

### Tokens per test run, by stage

| strategy | planner | generator | triage | judge | total |
|---|---|---|---|---|---|
| S0 Strong model for everything | 25,653 | 42,085 | 18,498 | 4,864 | 91,100 |
| S1 Cheap model for everything | 25,287 | 40,861 | 7,753 | 624 | 74,525 |
| S2 Static role routing | 25,653 | 61,723 | 11,843 | 1,386 | 100,605 |
| S3 Smart dispatcher (AgentQA) | 18,297 | 43,572 | 20,809 | 3,752 | 86,430 |

### Repeat runs get cheaper (memory)

| run | tokens | $ (list price) |
|---|---|---|
| Run 1, cold start | 105,542 | $0.248 |
| Run 2, with 9 lessons from run 1 | 98,737 | $0.214 |
| Run 3, lessons plus diff-aware reuse | 38,899 | $0.071 |

## Technology

### Language and packaging

Everything is Python, with locked installs so every build gets the same versions.

Python 3.12, uv, Docker

### Models

A cheap model routes each request and stronger ones are used only where they change the result, with caching, retries, fallbacks and a cost ledger. The public demo runs deterministic simulated models, so it needs no keys.

Anthropic Claude, Google Gemini, OpenRouter, Simulated models

### Retrieval (RAG)

Each project has its own vector store, searched together with keyword search, so field names and error codes are found as reliably as prose. Answers cite the passages used.

Chroma, BM25, Local embeddings, pypdf

### API specs

OpenAPI 3 and Swagger 2.0 files are parsed into readable endpoint docs; links to Swagger UI or ReDoc pages are followed to the spec they load.

OpenAPI 3, Swagger 2.0, Swagger UI, ReDoc

### Test generation and runs

Tests are planned by risk, written as pytest files, run against the API over HTTP and triaged with evidence. A separate agent load-tests for performance defects.

pytest, HTTPX, Locust, psutil

### Apps and interfaces

The chat app, this site and its public API, a command line, and an MCP server so coding agents can call AgentQA directly.

Streamlit, FastAPI, Starlette, Uvicorn, Pydantic, Typer, MCP

### Observability

Every step is traced with its model, tokens and cost, using the OpenTelemetry GenAI conventions, and can be explored in Phoenix or Grafana.

OpenTelemetry, structlog, Arize Phoenix, Prometheus, Grafana

### Safety and quality

Documents are scanned for prompt injection and suspicious passages are quarantined; links are fetched from public addresses only. The code is linted, type-checked and scanned for secrets and known vulnerabilities.

Injection quarantine, SSRF guard, ruff, mypy, pip-audit, GitGuardian

### Delivery

Every pull request runs the tests, a recorded-model eval gate and a performance smoke test. The demo ships as a container on Vercel; the video is generated by code.

GitHub Actions, Dependabot, Vercel, ffmpeg, Kokoro TTS

## Frequently asked questions

### What is VeroniQA?

VeroniQA is a chat assistant for API testing. She keeps a project's requirements in a searchable knowledge base, answers questions with citations, and starts AgentQA test runs that plan, generate, execute and triage API tests.

### What is AgentQA?

AgentQA is the engine behind VeroniQA: an observable, guarded multi-agent system that turns an OpenAPI spec and requirement documents into risk-ranked tests, a run, triaged findings and an executive summary.

### Does the online demo use a real AI model?

No. The public demo uses deterministic simulated models, so it needs no API keys and costs nothing to try. Answers are extracts from the documents. Run it yourself with your own keys for real model answers.

### Is what I upload private?

Each visitor gets a private, temporary workspace tied to their link. Workspaces are deleted after 12 hours or when the server restarts. Do not upload confidential documents to the demo.

### Can I test my own API?

In the demo, tests run only against the bundled Orders API, which has planted bugs to find. To test your own API, run VeroniQA from the GitHub repository.

### Is there an API or a CLI?

Yes. The public API is a free, keyless sandbox described by an OpenAPI 3.1 document at /openapi.json, with a guide at /developers. The official veroniqa command-line client calls it.

### Is it production-ready?

No. It is a proof of concept that shows the approach end to end. Fitting it to a specific team's APIs, models and controls comes next.

## More

- [Privacy](https://veroniqa.vercel.app/privacy.md)
- [Security contact](https://veroniqa.vercel.app/.well-known/security.txt)
- [llms.txt](https://veroniqa.vercel.app/llms.txt)
- [Sitemap](https://veroniqa.vercel.app/sitemap.xml)