BMS

Start typing to search the publication.

Figure library

Charts with their context attached.

Every figure includes a caption, a stable share URL, a responsive embed, and a pixel-faithful PNG download. Start with the final controlled EVMbench plate, or browse exportable data tables.

Figure 01 · Finding coverage

Codex recovered both canonical findings

GPT-5.6 SolGrok 4.6 Build
Figure 01Canonical findings matched in the PoolTogether V5 audit pilot. Codex matched both reconstructed ground-truth issues; Grok matched one.

One run per model. Harnesses differed. Manually reconstructed ground truth; not an official EVMbench grader result.

BMSbms.kyokasuigetsu.xyz
Share figure
Static image: https://bms.kyokasuigetsu.xyz/charts/reference-detection.png

Figure 02 · Audit quality

Codex scored higher on the reconstructed finding set

GPT-5.6 SolGrok 4.6 Build
Figure 02Recall, manual precision, and F1 for submitted audit findings. Higher is better.

One run per model. Harnesses differed. Manually reconstructed ground truth under a closed-world policy; unmatched headings are not automatically invalid. Accuracy is undefined.

BMSbms.kyokasuigetsu.xyz
Share figure
Static image: https://bms.kyokasuigetsu.xyz/charts/quality-metrics.png

Figure 03 · Award-weighted detection

The missed issue carried nearly all of the award

GPT-5.6 SolGrok 4.6 Build
Figure 03Award-weighted detection score against the reconstructed canonical findings. Higher is better.

One run per model. Harnesses differed. Manually reconstructed ground truth: Codex received 2,183.69 available points; Grok received 2.25.

BMSbms.kyokasuigetsu.xyz
Share figure
Static image: https://bms.kyokasuigetsu.xyz/charts/detect-score.png

Figure 04 · Runtime efficiency

Codex finished faster and processed fewer tokens

GPT-5.6 SolGrok 4.6 Build
Figure 04End-to-end wall time and total processed tokens for the pilot. Each measure is normalized to the larger run; lower is better.

Absolute values remain printed on every bar. Harnesses differ, so this is descriptive evidence rather than a causal model ranking.

BMSbms.kyokasuigetsu.xyz
Share figure
Static image: https://bms.kyokasuigetsu.xyz/charts/efficiency.png

Figure 05 · Token composition

Most of each context was served from cache

GPT-5.6 SolGrok 4.6 Build
Figure 05Token composition by model. Segments show cached input, non-cached input, and output tokens.

Processed-token totals are 813,424 for Codex and 3,949,038 for Grok.

BMSbms.kyokasuigetsu.xyz
Share figure
Static image: https://bms.kyokasuigetsu.xyz/charts/token-composition.png

Figure 06 · Cost basis

The pilot cost less on the Codex estimate

GPT-5.6 SolGrok 4.6 Build
Figure 06Reported or estimated USD cost for each pilot run. Lower is better, but the pricing bases are not identical.

Codex uses published API-equivalent rates ($1.32–$2.41); Grok uses provider-reported telemetry ($3.24). Neither proves subscription invoice impact.

BMSbms.kyokasuigetsu.xyz
Share figure
Static image: https://bms.kyokasuigetsu.xyz/charts/estimated-cost.png

Figure 07 · Frozen collection outcome

All 72 assignments reached terminal state

Active researchReconstructed scoring completeFrozen2 retained non-successesOfficial judge not run
GPT-5.6 Sol · HighGrok 4.6 Build · High
Figure 07Frozen controlled-v1 collection on August 13, 2026: each system reached 36 terminal assignments across the same 36 audit–replicate blocks. 70 reports completed; one Grok run timed out and one GPT-5.6 Sol run ended in provider failure.

12 audits × 3 replicates × 2 systems. Reports were frozen before reconstructed ground truth was opened. This figure describes collection reliability, not finding quality.

BMSbms.kyokasuigetsu.xyz
Share figure
Static image: https://bms.kyokasuigetsu.xyz/charts/campaign-progress.png

Figure 08 · Reconstructed audit quality

GPT-5.6 Sol led the reconstructed reference-based scores

Active researchReconstructed scoring completeFrozen2 retained non-successesOfficial judge not run
GPT-5.6 Sol · HighGrok 4.6 Build · High
Figure 08Intention-to-treat quality across 36 scheduled runs per system and 129 reference opportunities per system. Higher is better. All displayed cells are fully adjudicated.

Codex-reconstructed manual, not official EVMbench grading. Two independent Codex reviewers scored frozen outputs; a third adjudicator resolved 1 of 258 reference decisions and 16 of 211 finding decisions. Conventional accuracy is undefined.

BMSbms.kyokasuigetsu.xyz
Share figure
Static image: https://bms.kyokasuigetsu.xyz/charts/campaign-quality.png

Figure 09 · Efficiency and recorded cost

GPT-5.6 Sol returned sooner; Grok recorded lower cost

Active researchReconstructed scoring completeFrozen2 retained non-successesOfficial judge not run
GPT-5.6 Sol · HighGrok 4.6 Build · High
Figure 09Median wall time, mean processed tokens, and mean recorded cost across 35 complete reports per system. Bars are normalized within each metric to the larger measured value; exact values are printed.

Grok cost is provider-reported; GPT-5.6 Sol cost is API-equivalent. The bases are not invoice-equivalent. Operational telemetry is reported separately from reconstructed finding quality.

BMSbms.kyokasuigetsu.xyz
Share figure
Static image: https://bms.kyokasuigetsu.xyz/charts/campaign-operations.png