Grok 4.6 Build vs. GPT-5.6 Sol across 72 runs on 12 audits

The final result plate

GPT-5.6 Sol led the reconstructed reference scores

Across 36 scheduled runs per system, GPT-5.6 Sol recovered 52 of 129 repeated reference opportunities. Grok recovered 47 of 129. That produced 40.31% versus 36.43% micro recall and 46.06% versus 41.81% macro audit recall.

GPT-5.6 Sol also led on the reconstructed award-weighted score, closed-world precision, and F1. The margins were consistent rather than overwhelming: 6.814% versus 4.779% award-weighted recall, 50.49% versus 45.63% precision, and 44.83% versus 40.52% F1.

Grok stayed competitive beyond the reference set

Closed-world precision treats a finding as false positive when it does not match the reconstructed reference set. That is useful for comparability, but it can penalize genuinely new issues. The expert-validity review credits findings judged valid even when they were absent from the reference set.

Grok produced 51 novel-valid findings, compared with 49 for GPT-5.6 Sol. Expert-validity precision was high for both systems: 96.12% for Grok and 98.06% for GPT-5.6 Sol. The reference-score gap therefore does not mean Grok mostly generated bad findings.

Two reviews, then adjudication

The reconstructed scoring layer used two independent Codex reviewer passes over the frozen outputs. They initially agreed on 257 of 258 reference-detection decisions and 195 of 211 finding-disposition decisions. A third adjudication pass resolved the remaining 1 reference disagreement and 16 finding disagreements.

The reference layer measures whether each known vulnerability was detected. The separate finding-disposition layer distinguishes reference matches, novel-valid findings, invalid findings, partial matches, duplicates, and out-of-scope reports. This prevents official-style recall and manual precision from being blended into one opaque number.

The quality lead is not conclusive at 12 audits

The preregistered paired analysis resampled 12 audit clusters and the three paired replicate blocks inside each audit. On the reconstructed primary endpoint, the GPT-minus-Grok effect was 8.83 percentage points, with a 95% hierarchical-bootstrap interval from -6.88 to 25.48 points. The exact sign-flip p-value was 0.1484.

The interval crosses zero. The observed aggregate metrics favor GPT-5.6 Sol, but this 12-audit sample does not support a decisive population-level quality claim under the reconstructed scoring layer.

The primary paired estimator is computed inside matched audit–replicate blocks; it is not the same estimand as subtracting the two repeated-award aggregates printed above.

The campaign design

The frozen protocol uses 12 public smart-contract audits and three replicates per system–target pair. The inferential cluster count is 12 audits, not 72 independent observations. The earlier PoolTogether pilot is excluded.

Operationally, the systems made different tradeoffs

Across the 35 complete reports per system, GPT-5.6 Sol had a 2.37× shorter median wall time: 4m 24s versus 10m 25s. It also processed 19.6% fewer tokens per completed report on average: 99.9k versus 124.2k.

Grok recorded the lower numeric cost: $0.41 per complete report and $14.31 total, compared with GPT-5.6 Sol at $0.75 per report and $26.08 total. That apparent 45.1% gap is not a like-for-like price comparison. Grok uses provider-reported telemetry; GPT-5.6 Sol uses an API-equivalent estimate.

Across the 34 paired blocks where both systems completed, the geometric-mean GPT/Grok ratio was 0.414 for wall time, 0.778 for processed tokens, and 1.804 for recorded cost. The 95% ratio intervals were 0.347–0.498, 0.697–0.848, and 1.552–2.104, respectively.

GPT-5.6 Sol returned sooner and used fewer processed tokens. Grok recorded lower numeric cost while staying within roughly four to five points on the core reconstructed recall and F1 measures. The price bases are different, so this is an operational tradeoff rather than a direct invoice comparison.

The two non-successes stay visible

The 30-minute Grok timeout and the GPT-5.6 Sol provider failure are preserved in the frozen ledger. Report-level efficiency aggregates use the 35 complete reports per system; completion rates use all 36 scheduled assignments per system. Keeping both denominators visible prevents a successful-only analysis from hiding reliability outcomes.

Frozen before ground truth

All 72 assignment ledgers were terminal before the report set was frozen. The lock records 72 run outcomes and 70 report hashes. Ground truth remained sealed during collection, all 36 paired blocks are complete, and there are zero unpaired blocks.

Source and frozen artifacts

The public GitHub research repository contains the frozen protocol and schedule, terminal run ledgers and reports, reconstructed scoring decisions, CSV/JSON results, statistical inference, chart generators, and this publication’s source. Credentials, provider transcripts, build caches, cloned upstream repositories, and regenerable source bundles are intentionally excluded.

What remains unresolved

The official API-backed EVMbench judge has not run. The award-weighted values here are manual reconstructions from the frozen reference material, not official Detect scores. Conventional accuracy and false-positive rate are undefined because an open-ended audit has no finite true-negative universe.

The reconstructed inference uses 12 audit clusters and 50,000 hierarchical-bootstrap iterations. It does not make the manual reconstruction official, remove correlated reviewer bias, or guarantee exact cloud-inference reproducibility.

GPT-5.6 Sol led this reconstructed scoring layer. Grok remained close, surfaced more novel-valid findings, and recorded lower cost.