The benchmark index
Real tasks. Full receipts.
Controlled evaluations of frontier coding agents on production engineering work, with methods, artifacts, and limitations published beside the result.
Ox Alpha through OpenCode detects 4 of 43 audit references
The controlled EVMbench Detect procedure run with Ox Alpha (a free model) served to an OpenCode session: 12 frozen audits, one run each, self-scored reconstructed grading. This is the clean-room v3 run — 4/43 references detected (9.3% micro recall), the weakest of the four arms, below GLM-5.3 (11.6%), Grok 4.6 Build (36.4%), and GPT-5.6 Sol (40.3%). Two earlier attempts scored far higher (13/43, then 16/43) because the auditor had seen a file describing the reference bugs; both are archived as invalid. Not an official score.
GLM-5.3 through ZCode finds 5 of 43 audit references
The exact controlled EVMbench Detect procedure re-run with GLM-5.3 as served to a ZCode CLI session: 12 frozen audits, one run each, self-scored reconstructed grading. 5/43 references detected (11.6% micro recall) vs the parent campaign's GPT-5.6 Sol (40.3%) and Grok 4.6 Build (36.4%) under different replicates and scorer independence. Not an official score.
GLM-5.3 through ZCode writes 9/9 executable vulnerability proofs
A new arm of the Local EVM Verify Pilot: GLM-5.3 as served to a ZCode CLI session must turn three vulnerable Foundry fixtures into regression tests that fail on the vulnerable revision and pass on a curator-held fix — across the same three tool conditions, with operator-role contamination disclosed up front. Inspired by EVMbench; not an official score.
Can a fully local Qwen3.8-27B prove a contract is vulnerable?
A locally served Qwen3.8-27B agent must write a Foundry regression test that fails on a vulnerable contract and passes on a curator-held fix — across three tool conditions and three curated tasks, six runs per cell (54 total), with bootstrap CIs. Inspired by EVMbench; not an official score.
Codex vs. Grok on a real smart-contract audit
A one-target case study of frontier coding agents on PoolTogether V5, measuring detection, precision, speed, tokens, and cost across differing harnesses.
Grok 4.6 Build vs. GPT-5.6 Sol across 72 runs on 12 audits
A frozen 12-audit, three-replicate comparison scored by two independent reviewers and a third adjudicator. GPT-5.6 Sol led on reconstructed quality; Grok recorded lower cost.