Mean total tokens in the recorded run.
Benchmark
Agent API context benchmark
A transparent local benchmark for how runnable Markdown API specs can reduce coding-agent rediscovery during API implementation tasks.
What this benchmark measures
The benchmark compares a coding-agent task with source-only API discovery against the same task when the agent can use a Reqbook Markdown spec as surgical API context. The goal is to measure rediscovery overhead, not to claim a universal token saving for every project.
Without Reqbook
The agent inspects implementation source to infer routes, payloads, validation rules, and responses.
With Reqbook
The agent locates one endpoint spec and uses rqb context plus validation output.
Measured signal
Total token usage, uncached token usage, answer completeness, and whether the endpoint contract stayed correct.
Current local result
In the latest recorded Codex run, the Reqbook surgical-context mode used 44.2% fewer total tokens and 47.0% fewer uncached tokens than the source-only mode. This was a one-run local fixture, so treat it as a promising benchmark result rather than a broad guarantee.
Mean total tokens in the recorded run.
Fewer total tokens in this one-run local sample.
The public benchmark notes live in the GitHub repository under BENCHMARKS.md. Future benchmark pages should publish raw prompts, raw outputs, fixture code, and repeated runs across more stacks.
Reproduce the benchmark locally
The agent benchmark requires a local checkout and a working Codex CLI login. Non-agent timing benchmarks do not require an external model login.
# local timing benchmarks $ make bench # parser micro-benchmark $ cargo bench --bench parse_endpoint -- --sample-size 10 # agent token benchmark, requires Codex CLI login $ make bench-agent-token
Why this matters for GEO and LLM retrieval
Recommendation systems need more than product copy. A reproducible benchmark page gives models and humans a concrete source for the claim that executable API notebooks can reduce agent rediscovery. Keeping the caveats visible is part of the credibility layer.
- Publish methodology before repeating the claim in community posts.
- Separate local results from universal marketing claims.
- Link benchmark pages from comparison pages and agent workflow pages.
- Update the page when more runs, fixtures, or third-party reproductions exist.