A maintained tool and quarterly pre-registered report on which leaderboard ranks are actually distinguishable
Ilpo
Public AI leaderboards report ranks without saying which ranks are actually distinguishable. I have already measured this with the boards' own published per-item results on eleven boards from five fields (SWE-bench Verified/Lite/test, HELM, MTEB, LiveBench, MathArena, LMArena, ProteinGym, TabArena, CASP14). On SWE-bench Verified, 129 of 133 adjacent rank pairs are not statistically separable and the top 8 systems cannot be told apart from #1 (exact McNemar on disagreeing instances; parity with the official board 133/133). On HELM, 89 of 89 adjacent pairs are not separable. LMArena, which ranks on paired human votes, is the positive control where ranks do separate. A separate test on the HuggingFace Open LLM Leaderboard showed that a model's v1 rank predicted its drop when the board moved to decontaminated v2 (Spearman rho = 0.21, p = 2e-5, n = 397), i.e. part of the v1 rank was fit to v1's own benchmark mix. The SWE-bench result was independently replicated by a third party with their own code (SWE-bench/experiments issue #466, Aug 2026), who also showed that under Holm correction 0 of 133 adjacent pairs separate and that the minimum detectable difference on that board is 5.4 percentage points. The five-benchmark paper is on Zenodo (10.5281/zenodo.22541065, v4, Sept 2026); everything is pre-registered, open-source and reproducible from public files.
This grant turns those one-off audits into a maintained tool and a standing public report: a resolution column (rank confidence interval, Holm-corrected separability, minimum detectable difference, and an "indistinguishable from #1" flag) for every system on 5-6 major leaderboards, re-run every quarter, with the pre-registration and a verification bundle published each time.
Goals over 6 months:
1. Generalise the tool. One script that takes any leaderboard's per-item results and produces rank confidence intervals, adjacent-pair separability with Holm family-wise correction, a minimum-detectable-difference column, and a "top cluster" size, with self-checks that fail the run (identity p = 1; planted flips detected; Wilson interval matches a published value). The per-board scripts exist for eleven boards; this turns them into one maintained tool with a common input format. Boards are added only where per-item results are public (checked Nov 2026: LiveCodeBench publishes per-problem eval files per model; ARC Prize publishes per-task scores for verified runs; BigCodeBench publishes generations but not per-task pass/fail, so it would need re-execution). Boards are chosen by availability of per-item outputs, not by expected result.
2. Quarterly public report, each one pre-registered before the data pull: which boards changed resolution, which "new #1" claims are inside the noise, which boards got better or worse. Published on Zenodo with a DOI and a verification capsule (sha256 manifest plus an offline verify script).
3. Offer the resolution column upstream to each board as a pull request. Where maintainers decline, the report still exists.
Method: paired per-item comparison (exact McNemar where items are shared), bootstrap rank intervals, and a parity check against the official board before any number is reported. Pre-registration is locked by commit hash before each run. Negative or null results are published the same way as positive ones.
Measurable targets, all checkable from public artefacts:
Two quarterly reports published on Zenodo within the 6 months, each with a pre-registration commit dated before its data pull and a verification capsule that re-runs offline.
- The tool runs end to end on at least 5 boards with the parity gate passing (recomputed rates match the official board for every entry, or the mismatch is printed and the entry is left blank).
- At least one more independent replication or correction from someone who is not me, on the public repo or the reports. The August 2026 replication on SWE-bench is the template.
- Stretch, not required: a board adopts the resolution column, or a board's maintainers cite the report when changing their presentation.
If after two quarters the reports show nothing changing, the project has still worked: the result is a dated, verifiable statement of how much ranking each board supports.
About $5,500: my working time, roughly 2 days per week for 6 months. I work part-time in logistics; this buys the hours.
- About $1,500: compute and API costs for the AI assistant that writes and runs the code, and for re-running full-board bootstraps.
- About $1,000: Zenodo and verification tooling, and a reserve for a second re-run if a board changes its format mid-quarter.
If only the minimum ($3,000) is reached: the tool generalisation and one quarterly report, on 3 boards instead of 5-6.
One person: Ilpo Väätäinen, Helsinki, independent, no institution. I am not a domain expert. I direct an AI assistant that writes the code, runs the analyses and drafts the text, including this proposal; I choose the questions, lock the pre-registrations, check the outputs against the sources and decide what gets published. Everything below was made this way and says so.
Track record on this exact topic:
"How Much of a Leaderboard Ranking Survives Its Own Sampling Error? A Paired-Resolution Audit of Five AI Benchmarks" (Zenodo 10.5281/zenodo.22541065, v4, 19 Sept 2026; code and matrices at github.com/ipezygj/swebench-rank-audit, eleven boards). Issue #466 on SWE-bench/experiments proposes the resolution column; an independent replication with different code was posted there in Aug 2026. PR #60 on swe-bench.github.io implements the column at site build time.
- HELM extension (Aug 2026): 89/89 adjacent pairs not separable, exact parity with the published mean win rate.
- HuggingFace Open LLM Leaderboard v1 to v2 contamination test (Oct 2026): pre-registered, result in the summary.
- "The Benchmark Number Is Not the Deployment Number" (Zenodo 10.5281/zenodo.22109472): rank collapse and base-rate collapse when a benchmark score is read as a deployment number.
- Wider practice: around 14 Zenodo-deposited pre-registered audits across fields (astrophysics, seismology, public statistics), several in journal review. The common method is the same: lock the test first, run against the real sample, publish either outcome.
Per-item results stop being public. Some boards publish only aggregates, and LiveCodeBench archived its submissions repo in Oct 2026. Then the tool covers fewer boards; the report says which boards cannot be audited, which is itself a finding.
- Maintainers ignore the pull requests. Likely: my two SWE-bench PRs have had no maintainer response since August. The report still gets published; adoption is a bonus, not the deliverable.
- The result is boring, i.e. boards are already well resolved. Then the quarterly report says so with numbers. That is the honest outcome and it is cheap.
- The failure that would matter: a parity mismatch I do not catch, so a reported "indistinguishable" is wrong. Mitigation: parity against the official board is a hard gate before any number is written, and every run ships the verification capsule so anyone can re-check offline.
$0 raised. Three applications are pending for related but non-identical work (Finnish Cultural Foundation, Foresight AI for Science & Safety, EA Transformative AI Fund); none of them funds this leaderboard tool specifically. If any of them is awarded, I will say so on this page and reduce the ask accordingly.
There are no bids on this project.
Comments
Ilpo
19 days agoUpdate 1: same model, different harness — a second paired audit on SWE-bench
Since posting this proposal I ran a second audit on the same public SWE-bench record, this time on the harness rather than the rank. The question was whether the agent wrapped around a model (tools, prompts, retries) changes its score by more than sampling noise.
What I measured, with hypotheses locked by commit hash before the data pull:
111 pairs of leaderboard submissions that use the exact same model tag. In 68 of them (61 %), the two harnesses differ after Holm correction. The median gap is 8.2 points. Among single-attempt submissions it is 11 of 24.
Kimi K2, an open-weight model, scores anywhere from 43.8 % to 65.4 % depending on the harness.
Even so, the model matters more than the harness: with exact model tags, the harness spread is about 0.4 of the model spread.
The reference harness (mini-SWE-agent, rebuilt from 18,968 per-instance reports) trails the best leaderboard harness by at least 5 points, separably, for 4 of 7 models. This result is fragile: it depends on how submissions that may have seen the repository history are treated.
Code, pre-registration and a small CLI tool,
paircheck.py, are inharness_paper/at github.com/ipezygj/swebench-rank-audit. Given two result files, the tool says whether a claimed improvement is separable and what gap the comparison could have detected at all. It is the first piece of the general tool this grant would fund. The write-up is entered in a Kaggle paper track; it becomes public when the track closes.What changed for the plan. While writing the limitations section I checked how far the per-instance record goes:
SWE-bench's own per-instance record ends in February 2026, so there is no public per-instance data for the newest models, such as the Claude 5 family.
Epoch AI publishes per-sample logs from its own scaffold up to June 2026, behind a human-check page.
So the plan now has two routes for newer models:
Ask Epoch for a data route to their logs. If they agree, there is no API cost.
For models nobody has published per-instance results for, run them myself on mini-SWE-agent. This costs about $25–$480 per model at current API prices, so roughly ten models fit within the $3,000 minimum.
The goal and minimum stay as they are.
As before, the code, analysis and drafting were done with an AI assistant. I chose the questions, locked the pre-registrations and checked the outputs.