Independent agent research
We test frontier coding agents on real engineering tasks—and publish the methods, costs, failures, and evidence behind every result.
Current campaign
Controlled security campaign v1Frozen and manually reconstructed: 72 terminal assignments, 70 complete reports, and 72 scored runs. Official API-backed EVMbench grading remains unrun.Campaign 01 · Smart-contract security
Twelve public EVMbench audits, three replicates per system, and all 72 assignments frozen. GPT-5.6 Sol led the reconstructed reference scores; Grok surfaced more novel-valid findings and recorded lower cost.
Terminal assignments
Complete reports
Manual run scores
The controls, disclosures, and uncomfortable edge cases we think every serious coding-agent comparison should publish.
Why tokens, turns, and elapsed time belong next to quality scores—and what each one reveals about an agent.
Tasks come from production repositories with authentic constraints, ambiguity, and failure modes.
We preregister settings, isolate runs, preserve artifacts, and separate pilot evidence from controlled claims.
Quality sits beside time, tokens, cost, tool use, and variance. A single score never tells the whole story.