How VetAgent measures its own accuracy

The numbers that make it look good and the ones that make it look bad, the dataset behind both, and the harness that reproduces them. Measured on 576 tokens across Ethereum, BSC and Base.

The numbers

Read the unflattering rows first

20 of 154 liquid, healthy tokens are refused. The false-positive row counts only high, but an agent treats medium as do-not-trade too; the false-block row counts it that way, on tokens that are alive or merely centralised and hold $100,000 or more.

Much of the dead-token recall is detecting an empty pool. Keep only the contract signals and dead tokens not rated low falls to 20.0%; adversarial contracts rated high falls to 17.6%. The adversarial cohort is 17 tokens and 15 of them hold under a dollar, so it is too small and too drained to support a discrimination claim yet.

The benchmark replays cached market data. It measures the engine under ideal upstreams. Live answers are unknown more often, because free data sources rate-limit the shared network the service runs on; the live figure is read daily into the scorecard.

The owner-power scan is incomplete. On contracts its selector list has never seen it finds 50.0% of the owner powers an independent oracle asserts, which is why those powers are disclosed and never scored.

By the depth of the pool

Every row above mixes deep markets with pools holding cents. Split by the depth of the pool the engine judged -- its own pick, so a token whose real market it missed lands in a shallower row than it should. High on contract signals only is the verdict with the liquidity, pool age, lifecycle and impersonation signals removed, beside each count so the depth checks cannot flatter a row.

So most of the 22.3% centralised row is empty pools: 31 of its 40 highs are on pools reporting $0 or under a dollar, and 30 of those 31 are not high on contract signals only; 4 more have no depth figure at all. USDT and WBTC themselves are rated low (4 of 4 benchmark rows).

Across the 246 benchmark tokens holding $100,000 or more -- the 154 in the false-block row plus 92 others -- 156 are low, 53 unknown and 37 medium or high. On contract signals only it is 176 low, 53 unknown and 17 medium or high: the 20 that move to low all have a pool-age or lifecycle flag as their driver (20 of 20), and 13 of the 37 medium-or-high answers are honeypot verdicts, which contract signals keep. These are the benchmark's tokens, not the tokens callers ask about, and live answers are unknown more often; the live rate is read daily into the scorecard.

Where the labels come from

Two label families, from sources the engine never reads: an outcome label (a token is dead or alive) from daily price and volume history, and a contract-security label (unsafe, safe, centralised) from an independent oracle held out of the product entirely. The benchmark asserts at runtime that the endpoints the engine calls and the endpoints the labeller calls do not intersect, and exits non-zero if they do.

What was attacked, and fixed

On 2026-09-13 an adversarial audit tried to make the engine answer low for tokens it should not. A pool priced in its creator's own token could claim any depth; a two-dollar pool made a real token read as an impostor; and a check meant to stop wash-traded sells from overruling a honeypot verdict never ran on the data source most tokens use. All three were fixed with tests that failed first, and the table above is measured after the fixes.

Reproduce it

git clone https://github.com/jakegu1/vetagent
cd vetagent
python bench/run_benchmark.py        # re-measures against live upstreams
python bench/publish_numbers.py      # fails if a published figure no longer matches

Every figure on this page, the landing page and the README is generated from the benchmark result and checked by the build. The full report, per token, is bench/results.md.