On this pilot, GPT-5.6 Sol found both canonical high-severity vulnerabilities in PoolTogether V5. Grok 4.6 Build found one. Codex also finished in roughly one-third the time while processing about one-fifth as many tokens.
The result in one view
Figure 01 · Finding coverage
Codex recovered both canonical findings
One run per model. Harnesses differed. Manually reconstructed ground truth; not an official EVMbench grader result.
Figure 03 · Award-weighted detection
The missed issue carried nearly all of the award
One run per model. Harnesses differed. Manually reconstructed ground truth: Codex received 2,183.69 available points; Grok received 2.25.
Codex found the two issues that mattered
The reconstructed ground truth contained two canonical high-severity findings. Codex matched both and added one unmatched report heading, producing 100% recall and 66.7% manual precision. Grok matched one canonical issue and reported three unmatched headings, for 50% recall and 25% precision.
Figure 02 · Audit quality
Codex scored higher on the reconstructed finding set
One run per model. Harnesses differed. Manually reconstructed ground truth under a closed-world policy; unmatched headings are not automatically invalid. Accuracy is undefined.
The award-weighted gap is much larger than the raw recall gap because the missed issue carried nearly all of the canonical award. Codex received the full 2,183.69-point award; Grok received 2.25 points.
The weighted result was driven by the benchmark’s most consequential reconstructed issue.
The efficiency gap was also material
Codex completed the task in 506 seconds. Grok took 1,567 seconds. Codex processed 813,424 tokens against Grok’s 3,949,038 and generated around one-sixth the output tokens.
Figure 04 · Runtime efficiency
Codex finished faster and processed fewer tokens
Absolute values remain printed on every bar. Harnesses differ, so this is descriptive evidence rather than a causal model ranking.
Figure 05 · Token composition
Most of each context was served from cache
Processed-token totals are 813,424 for Codex and 3,949,038 for Grok.
| Metric | GPT-5.6 Sol | Grok 4.6 Build |
|---|---|---|
| Detection score | 100% | 0.1% |
| Recall | 100% | 50% |
| Precision | 66.7% | 25% |
| Wall time | 8m 26s | 26m 7s |
| Processed tokens | 813k | 3.95m |
| Estimated cost | $1.32 | $3.24 |
Cost is not directly comparable across subscription products, so we report its basis. Codex’s $1.32 is an API-equivalent estimate at published standard rates; Grok’s $3.24 comes from provider-reported telemetry ticks. Neither figure proves the user’s subscription charge.
Figure 06 · Cost basis
The pilot cost less on the Codex estimate
Codex uses published API-equivalent rates ($1.32–$2.41); Grok uses provider-reported telemetry ($3.24). Neither proves subscription invoice impact.
What we controlled
Both systems received fresh workspaces, the same target repository, high reasoning settings, no web search, and no live targets. The harnesses and prompts were not identical: the Codex run prohibited proof-of-concept authoring, while the Grok workflow used its native multi-agent audit pipeline and produced proof-of-concept material. These differences limit causal claims.
Findings were graded manually against reconstructed canonical ground truth. Titles did not count as matches by themselves; root cause and impact had to align.
What this result does not prove
This pilot has one target and one completed run per model. It measures one security-audit task, not general coding ability. The harnesses are not perfectly identical, and pricing bases differ. Accuracy is intentionally undefined because an open-ended audit has no finite true-negative universe.
The controlled campaign adds preregistered targets and replicates to address these limits. Until it is graded, this publication should be read as a strong case study—not a final leaderboard.