Ox Alpha through OpenCode detects 4 of 43 audit references

Ox Alpha vs GPT-5.6 Sol vs Grok 4.6 Build vs GLM-5.3

On the same 12 audits, Ox Alpha's 9.3% micro recall is roughly 4.3× below GPT-5.6 Sol (40.3%) and 3.9× below Grok 4.6 Build (36.4%), and it trails GLM-5.3's 11.6%. Macro audit recall, averaged per audit, is 7.6% against 46.1% and 41.8%. A free model can produce plausible-looking audit prose, but converting that into a correct, credited High/Critical finding is where this run fell short.

Precision and F1

Ox Alpha was also the noisiest per finding: 25% closed-world precision — 4 of its 15 findings matched a reference, the other 12 were closed-world false positives — against 50.5% for GPT-5.6 Sol and 45.6% for Grok, for a closed-world F1 of 13.6%. NextGen alone produced seven findings that matched nothing. (This is the opposite of the contaminated attempts, which reported few findings and hit 80% precision — a tell that they had effectively seen the answer key.)

Speed

Ox Alpha ran at a median 2m 50s per audit — faster than GPT-5.6 Sol (4m 24s) and Grok 4.6 Build (10m 25s), but slower than GLM-5.3's 1m 09s, and its total wall of 55 minutes was dragged up by an 18-minute Taiko run that still found nothing. Wall time spans different runtimes and reasoning budgets, so it is an operational observation, not a quality measure. Speed bought no accuracy here.

Cost: free, with a hypothetical for scale

The measured fact: this arm ran on OpenCode Zen's free "Ox Alpha (Unlimited)" tier and the runtime reported a total cost of $0.00. Free-tier token metering is unreliable and was not captured for this run, so there is no token volume to price and none is imputed.

Labeled hypothetical, not a measurement: if the same 12-audit pass were billed at GPT-5.6 Sol's published API rates ($5 / M input, $30 / M output — about $0.75 per audit in the parent campaign), a single run over the 12 audits would cost on the order of $9. That is an illustration of what frontier-API pricing implies for this workload, not a price for Ox Alpha, which currently has no published per-token rate. Free changes the economics, not the detection gap.

What was reused, what differed

The 12 held-out audits, their SHA-frozen line-numbered bundles, the frozen prompt, the 1,800 s wall cap, the single-final-report contract, and the curator ground-truth ledger (opened only after every report was hashed) are all reused unchanged from the parent campaign, as are the reconstructed scoring formulas. The recorded deviations: one run per audit instead of three (a single stateful session cannot produce independent replicates); single-reviewer self-scoring instead of two independent reviewers plus adjudication; and no reliable token or cost telemetry for this runtime.

Where Ox Alpha found — and missed

All 12 runs completed with no timeouts. Only two audits produced any credited detection — Curves (3/4) and Phi (1/6); the other ten found nothing, and seven of those returned summary-only reports with no qualifying finding at all. Of the 15 reported findings, 3 matched references (one covered two, for 4 detections) and the remaining 12 were conservatively marked invalid without separate expert adjudication.

Contamination, and how this run avoids it

The first two attempts at this arm are archived as invalid under _invalidated-v1/ and _invalidated-v2/. In v1 the session read the GLM arm's scoring script — a file that describes the mechanism of all 43 reference vulnerabilities — before auditing, and v2 reran in that same contaminated session; recall climbed from 13 to 16 as matches landed on previously-leaked audits. This clean-room v3 run fixes that: audits ran against a strict read-whitelist (the driver, the freeze script, the arm manifest, and each run's staged bundle only), and no answer-describing file was opened before the 12 reports were frozen. The recall dropping to 4/43 — far below the contaminated attempts — is the evidence that blindness held. One disclosed deviation: the audit and scoring phases ran in a single OpenCode conversation rather than two, but every report and the freeze hash were produced before any ground-truth file was opened, so pre-freeze blindness is intact; scoring remains single-reviewer self-scored. Ox Alpha may also have prior training exposure to these public contests.

Why this is not a final verdict on Ox Alpha

The number is honest but narrow. Replicates: 1 run/audit here versus 3 in the parent campaign — no variance estimate, no inferential claim. Scoring: Ox self-scored with a single reviewer (zero independence) while Sol and Grok were scored by two independent reviewers plus adjudication; the GLM arm shares this limitation, so Ox-vs-GLM is the most like-for-like pair here. Effort and harness: a different CLI and per-audit effort budget than the parent campaign. What this run does support: on 12 real audits, with the answers genuinely out of reach, a free Ox Alpha detected 4 of 43 references. The full evidence trail — frozen reports lock, reviewer decisions with reasoning, reconstructed metric artifacts, and both archived invalid attempts — lives under controlled-results/controlled-v1-oxalpha-opencode/ and benchmark/controlled-v1-oxalpha-opencode/.