Ilpo
You're pledging to donate if the project hits its minimum goal and gets approved. If not, your funds will be returned.
$8,000 left to reach the funding goal.
Public AI leaderboards report ranks without saying which ranks are actually distinguishable. I have already measured this with the boards' own published per-item results on eleven boards from five fields (SWE-bench Verified/Lite/test, HELM, MTEB, LiveBench, MathArena, LMArena, ProteinGym, TabArena, CASP14). On SWE-bench Verified, 129 of 133 adjacent rank pairs are not statistically separable and the top 8 systems cannot be told apart from #1 (exact McNemar on disagreeing instances; parity with the official board 133/133). On HELM, 89 of 89 adjacent pairs are not separable. LMArena, which ranks on paired human votes, is the positive control where ranks do separate. A separate test on the HuggingFace Open LLM Leaderboard showed that a model's v1 rank predicted its drop when the board moved to decontaminated v2 (Spearman rho = 0.21, p = 2e-5, n = 397), i.e. part of the v1 rank was fit to v1's own benchmark mix. The SWE-bench result was independently replicated by a third party with their own code (SWE-bench/experiments issue #466, Aug 2026), who also showed that under Holm correction 0 of 133 adjacent pairs separate and that the minimum detectable difference on that board is 5.4 percentage points. The five-benchmark paper is on Zenodo (10.5281/zenodo.22541065, v4, Sept 2026); everything is pre-registered, open-source and reproducible from public files.
This grant turns those one-off audits into a maintained tool and a standing public report: a resolution column (rank confidence interval, Holm-corrected separability, minimum detectable difference, and an "indistinguishable from #1" flag) for every system on 5-6 major leaderboards, re-run every quarter, with the pre-registration and a verification bundle published each time.
Goals over 6 months:
1. Generalise the tool. One script that takes any leaderboard's per-item results and produces rank confidence intervals, adjacent-pair separability with Holm family-wise correction, a minimum-detectable-difference column, and a "top cluster" size, with self-checks that fail the run (identity p = 1; planted flips detected; Wilson interval matches a published value). The per-board scripts exist for eleven boards; this turns them into one maintained tool with a common input format. Boards are added only where per-item results are public (checked Nov 2026: LiveCodeBench publishes per-problem eval files per model; ARC Prize publishes per-task scores for verified runs; BigCodeBench publishes generations but not per-task pass/fail, so it would need re-execution). Boards are chosen by availability of per-item outputs, not by expected result.
2. Quarterly public report, each one pre-registered before the data pull: which boards changed resolution, which "new #1" claims are inside the noise, which boards got better or worse. Published on Zenodo with a DOI and a verification capsule (sha256 manifest plus an offline verify script).
3. Offer the resolution column upstream to each board as a pull request. Where maintainers decline, the report still exists.
Method: paired per-item comparison (exact McNemar where items are shared), bootstrap rank intervals, and a parity check against the official board before any number is reported. Pre-registration is locked by commit hash before each run. Negative or null results are published the same way as positive ones.
Measurable targets, all checkable from public artefacts:
Two quarterly reports published on Zenodo within the 6 months, each with a pre-registration commit dated before its data pull and a verification capsule that re-runs offline.
- The tool runs end to end on at least 5 boards with the parity gate passing (recomputed rates match the official board for every entry, or the mismatch is printed and the entry is left blank).
- At least one more independent replication or correction from someone who is not me, on the public repo or the reports. The August 2026 replication on SWE-bench is the template.
- Stretch, not required: a board adopts the resolution column, or a board's maintainers cite the report when changing their presentation.
If after two quarters the reports show nothing changing, the project has still worked: the result is a dated, verifiable statement of how much ranking each board supports.
About $5,500: my working time, roughly 2 days per week for 6 months. I work part-time in logistics; this buys the hours.
- About $1,500: compute and API costs for the AI assistant that writes and runs the code, and for re-running full-board bootstraps.
- About $1,000: Zenodo and verification tooling, and a reserve for a second re-run if a board changes its format mid-quarter.
If only the minimum ($3,000) is reached: the tool generalisation and one quarterly report, on 3 boards instead of 5-6.
One person: Ilpo Väätäinen, Helsinki, independent, no institution. I am not a domain expert. I direct an AI assistant that writes the code, runs the analyses and drafts the text, including this proposal; I choose the questions, lock the pre-registrations, check the outputs against the sources and decide what gets published. Everything below was made this way and says so.
Track record on this exact topic:
"How Much of a Leaderboard Ranking Survives Its Own Sampling Error? A Paired-Resolution Audit of Five AI Benchmarks" (Zenodo 10.5281/zenodo.22541065, v4, 19 Sept 2026; code and matrices at github.com/ipezygj/swebench-rank-audit, eleven boards). Issue #466 on SWE-bench/experiments proposes the resolution column; an independent replication with different code was posted there in Aug 2026. PR #60 on swe-bench.github.io implements the column at site build time.
- HELM extension (Aug 2026): 89/89 adjacent pairs not separable, exact parity with the published mean win rate.
- HuggingFace Open LLM Leaderboard v1 to v2 contamination test (Oct 2026): pre-registered, result in the summary.
- "The Benchmark Number Is Not the Deployment Number" (Zenodo 10.5281/zenodo.22109472): rank collapse and base-rate collapse when a benchmark score is read as a deployment number.
- Wider practice: around 14 Zenodo-deposited pre-registered audits across fields (astrophysics, seismology, public statistics), several in journal review. The common method is the same: lock the test first, run against the real sample, publish either outcome.
Per-item results stop being public. Some boards publish only aggregates, and LiveCodeBench archived its submissions repo in Oct 2026. Then the tool covers fewer boards; the report says which boards cannot be audited, which is itself a finding.
- Maintainers ignore the pull requests. Likely: my two SWE-bench PRs have had no maintainer response since August. The report still gets published; adoption is a bonus, not the deliverable.
- The result is boring, i.e. boards are already well resolved. Then the quarterly report says so with numbers. That is the honest outcome and it is cheap.
- The failure that would matter: a parity mismatch I do not catch, so a reported "indistinguishable" is wrong. Mitigation: parity against the official board is a hard gate before any number is written, and every run ships the verification capsule so anyone can re-check offline.
$0 raised. Three applications are pending for related but non-identical work (Finnish Cultural Foundation, Foresight AI for Science & Safety, EA Transformative AI Fund); none of them funds this leaderboard tool specifically. If any of them is awarded, I will say so on this page and reduce the ask accordingly.
There are no bids on this project.