Answers with sources
Ask about your API and get answers grounded in your own documents, with numbered citations. Citations that do not match a retrieved passage are dropped, and "not found" is a real answer.
Live demo · no sign-up, no API keys
VeroniQA The AI assistant for API testing, built on AgentQA
Give her your requirements, documents and API docs. She answers with sources, writes and runs the tests, and explains every failure with evidence.
- 94%of planted bugs found
- 47%of the cost of the strongest model
- 6/6performance defects caught
- 12bugs planted in the demo API
- 10cost levers, each measured
- 0API keys needed to try it
Measured on the simulated benchmark in the repository.
- Python 3.12
- FastAPI
- Streamlit
- Chroma vector store
- BM25 hybrid retrieval
- OpenAPI 3 · Swagger 2
- pytest
- OpenTelemetry
- Model Context Protocol
- Locust
- Claude
- Gemini
- OpenRouter
- Prompt-injection quarantine
- SSRF-safe links
- Replay-gated CI
Watch
Meet VeroniQA in 85 seconds
Knowledge in, cited answers out, safe links, and a full test run from one chat message.
Features
API testing, AI-augmented
One place for an API's requirements, questions and test runs, on top of an engine that is observable, guarded and measured.
Tests from your requirements
Say “run the tests”. AgentQA plans risk-ranked tests from the spec and your documents, runs them, and triages each failure with evidence.
Knowledge you can trust
Upload Markdown, text, HTML or PDF, or add links. Links are fetched from public addresses only, and text that looks like a prompt injection is quarantined.
Retrieval that fits the domain
Each project has its own vector store (Chroma) combined with keyword search (BM25), so field names and error codes are found as reliably as plain prose.
Cost-aware by design
A cheap model routes each message; stronger models are used only where they change the result. Every step is traced with its tokens and cost.
Fits how teams work
Use the browser, let coding agents call AgentQA over MCP, or gate pull requests on an eval and a performance smoke test in CI.
How it works
From documents to findings in three steps
The demo starts you with a sample Orders API that has planted bugs, so you can try every step right away.
Create a project
One folder per API, with its spec, settings and knowledge.
Add what your team already has
Requirement documents, PDFs and links to your docs.
Ask, then run the tests
Get cited answers, a test run, findings and a short report.
Results
Usage and cost, measured
Four ways to run the same test job against the demo API, three runs each. Costs are what the tokens would cost at each model's list price.
Cost per test run
The smart dispatcher finds the most bugs at 47% of the cost of using the strongest model for everything.
- S0 · Strong model for everything92% of bugs found$0.473
- S1 · Cheap model for everything47% of bugs found$0.037
- S2 · Static role routing67% of bugs found$0.173
- S3 · Smart dispatcher (AgentQA)94% of bugs found$0.221
Show as table
| Strategy | $ per run (list price) | Bugs found |
|---|---|---|
| S0 Strong model for everything | $0.473 | 92% |
| S1 Cheap model for everything | $0.037 | 47% |
| S2 Static role routing | $0.173 | 67% |
| S3 Smart dispatcher (AgentQA) | $0.221 | 94% |
Tokens per test run, by stage
The dispatcher uses about as many tokens as the strong model (86K vs 91K). It saves money by giving most of the work to cheaper models, not by doing less.
- Planner
- Generator
- Triage
- Judge
- S0 · Strong model for everything91K
- S1 · Cheap model for everything75K
- S2 · Static role routing101K
- S3 · Smart dispatcher (AgentQA)86K
Show as table
| Strategy | Planner | Generator | Triage | Judge | Total |
|---|---|---|---|---|---|
| S0 Strong model for everything | 25,653 | 42,085 | 18,498 | 4,864 | 91,100 |
| S1 Cheap model for everything | 25,287 | 40,861 | 7,753 | 624 | 74,525 |
| S2 Static role routing | 25,653 | 61,723 | 11,843 | 1,386 | 100,605 |
| S3 Smart dispatcher (AgentQA) | 18,297 | 43,572 | 20,809 | 3,752 | 86,430 |
Repeat runs get cheaper
The same suite run three times with memory: the third run cost 71% less than the first, at the same recall.
- Run 1 · cold start$0.248106K tokens
- Run 2 · with 9 lessons from run 1$0.21499K tokens
- Run 3 · lessons plus diff-aware reuse$0.07139K tokens
Show as table
| Run | Tokens | $ (list price) |
|---|---|---|
| Run 1, cold start | 105,542 | $0.248 |
| Run 2, with 9 lessons from run 1 | 98,737 | $0.214 |
| Run 3, lessons plus diff-aware reuse | 38,899 | $0.071 |
Measured on the simulated benchmark in the repository (models with declared error rates, ADR 0003); full tables in RESULTS.md.
Technology
What it is built with
The whole stack, from model routing to delivery. The source for every part is in the repository.
Language and packaging
Everything is Python, with locked installs so every build gets the same versions.
- Python 3.12
- uv
- Docker
Models
A cheap model routes each request and stronger ones are used only where they change the result, with caching, retries, fallbacks and a cost ledger. The public demo runs deterministic simulated models, so it needs no keys.
- Anthropic Claude
- Google Gemini
- OpenRouter
- Simulated models
Retrieval (RAG)
Each project has its own vector store, searched together with keyword search, so field names and error codes are found as reliably as prose. Answers cite the passages used.
- Chroma
- BM25
- Local embeddings
- pypdf
API specs
OpenAPI 3 and Swagger 2.0 files are parsed into readable endpoint docs; links to Swagger UI or ReDoc pages are followed to the spec they load.
- OpenAPI 3
- Swagger 2.0
- Swagger UI
- ReDoc
Test generation and runs
Tests are planned by risk, written as pytest files, run against the API over HTTP and triaged with evidence. A separate agent load-tests for performance defects.
- pytest
- HTTPX
- Locust
- psutil
Apps and interfaces
The chat app, this site and its public API, a command line, and an MCP server so coding agents can call AgentQA directly.
- Streamlit
- FastAPI
- Starlette
- Uvicorn
- Pydantic
- Typer
- MCP
Observability
Every step is traced with its model, tokens and cost, using the OpenTelemetry GenAI conventions, and can be explored in Phoenix or Grafana.
- OpenTelemetry
- structlog
- Arize Phoenix
- Prometheus
- Grafana
Safety and quality
Documents are scanned for prompt injection and suspicious passages are quarantined; links are fetched from public addresses only. The code is linted, type-checked and scanned for secrets and known vulnerabilities.
- Injection quarantine
- SSRF guard
- ruff
- mypy
- pip-audit
- GitGuardian
Delivery
Every pull request runs the tests, a recorded-model eval gate and a performance smoke test. The demo ships as a container on Vercel; the video is generated by code.
- GitHub Actions
- Dependabot
- Vercel
- ffmpeg
- Kokoro TTS
FAQ
Questions
What is VeroniQA?
VeroniQA is a chat assistant for API testing. She keeps a project's requirements in a searchable knowledge base, answers questions with citations, and starts AgentQA test runs that plan, generate, execute and triage API tests.
What is AgentQA?
AgentQA is the engine behind VeroniQA: an observable, guarded multi-agent system that turns an OpenAPI spec and requirement documents into risk-ranked tests, a run, triaged findings and an executive summary.
Does the online demo use a real AI model?
No. The public demo uses deterministic simulated models, so it needs no API keys and costs nothing to try. Answers are extracts from the documents. Run it yourself with your own keys for real model answers.
Is what I upload private?
Each visitor gets a private, temporary workspace tied to their link. Workspaces are deleted after 12 hours or when the server restarts. Do not upload confidential documents to the demo.
Can I test my own API?
In the demo, tests run only against the bundled Orders API, which has planted bugs to find. To test your own API, run VeroniQA from the GitHub repository.
Is there an API or a CLI?
Yes. The public API is a free, keyless sandbox described by an OpenAPI 3.1 document at /openapi.json, with a guide at /developers. The official veroniqa command-line client calls it.
Is it production-ready?
No. It is a proof of concept that shows the approach end to end. Fitting it to a specific team's APIs, models and controls comes next.
Ready to try it?
Your own private workspace, preloaded with the demo API. Nothing to install and nothing to pay.