Independent agent research
We test frontier coding agents on real engineering tasks—and publish the methods, costs, failures, and evidence behind every result.
Current campaign
Controlled security campaign v1In progress and paused at a clean 11-block boundary: 22 terminal assignments, 50 pending, quality ungraded.Campaign 01 · Smart-contract security
Twelve public EVMbench audits, three replicates per system, and a paired checkpoint that reports operational status without turning partial data into a winner.
Terminal assignments
Matched audit blocks
Official grades
The controls, disclosures, and uncomfortable edge cases we think every serious coding-agent comparison should publish.
Why tokens, turns, and elapsed time belong next to quality scores—and what each one reveals about an agent.
Tasks come from production repositories with authentic constraints, ambiguity, and failure modes.
We preregister settings, isolate runs, preserve artifacts, and separate pilot evidence from controlled claims.
Quality sits beside time, tokens, cost, tool use, and variance. A single score never tells the whole story.