When two agents produce equally correct work, the one that consumes five times the tokens is not equally capable in practice. Resources shape latency, reliability, and whether the system can be deployed at all.
Quality is not free—but spend alone proves nothing
Longer reasoning can improve outcomes. It can also signal thrashing: repeated file reads, abandoned branches, and tool calls that do not reduce uncertainty. Cost must therefore sit beside quality, never replace it.
An efficient wrong answer is still wrong. An expensive correct answer may still be unusable.
The useful question is how much verified progress a system produces per unit of constrained resource. That resource might be dollars, wall time, tokens, human interventions, or all four.
Tokens expose the shape of the work
Processed tokens measure more than verbosity. They capture repeated context, cached reads, reasoning output, and the size of the model’s working loop. We break them out because a single total can hide radically different execution patterns.
In our first audit pilot, one agent processed 813,424 tokens and the other 3,949,038. The smaller run also found more canonical issues. On that task, extra processing did not buy extra quality.
Wall time is the user’s actual wait
Provider compute time and end-to-end time are different. Tool latency, orchestration, retries, and serialization all live in the gap. We prefer wall time for the user-facing comparison and preserve provider timing as diagnostic context.
How we report cost
Subscription products, API calls, and provider telemetry are not interchangeable. We label cost as provider reported, API equivalent, or estimated—and never imply that an estimate is a literal invoice.
This is not bookkeeping. It is how a benchmark becomes a deployment decision instead of a demo score.