You're pledging to donate if the project hits its minimum goal and gets approved. If not, your funds will be returned.
$24,000 left to reach the funding goal.
AI coding benchmarks score models by running auto-generated or hand-written tests. When those tests are weak, wrong programs pass, and reported scores go up. Shuna Audit checks the graders themselves. So far, using mutation testing, differential fuzzing, EvalPlus cross-checks and an adversarial prosecutor/defender review, we've confirmed 191 flawed tasks across 9 benchmarks, including HumanEval 20, MBPP 53, HumanEval+ 2, MBPP+ 11, HumanEvalNext 7 and EvoEval 31 (the per-benchmark breakdown for the rest is under responsible-disclosure embargo until Nov 1). We then measured the effect on 6 open coding models (1.3B to 7B parameters) in a Kaggle run completed Oct 20, 2026. On the confirmed tasks in the 5 benchmarks we ran, the official tests accepted 80 model solutions that our strict checks reject (each re-confirmed in 3 sequential re-runs; 5 of an initial 85 turned out to be timing flakes and were dropped). That inflates scores by up to 3.5 points: 1.6–3.5 points on MBPP for every model, up to 1.2 on HumanEval, and none on HumanEvalNext. We've sent responsible disclosure to affected benchmark authors with a 14-day notice (until Nov 1) before publishing details.
Goal: make "how trustworthy is this benchmark's grader?" a number anyone can look up.
Audit four benchmarks that labs still report: SWE-bench Verified (do the FAIL_TO_PASS tests still pass on mutated gold patches? We'll start with a 50-task sample), LiveCodeBench, CRUXEval and BigCodeBench-Hard.
Publish an open grader-quality leaderboard listing confirmed flawed tasks per benchmark, a witness input and wrong program for each one, corrected tests, and score-impact estimates.
Release all code and data under an open license, with a one-command rerun so anyone can reproduce or dispute any item.
Disclose responsibly: give maintainers 14 days' notice before we publish anything about their benchmark.
Method (already working on 9 benchmarks): we generate wrong-but-plausible variants of the reference solutions (mutation testing), run them against the official tests and against stronger oracles (differential fuzzing, EvalPlus), and then have one model argue each candidate is a real flaw while another argues it isn't. A task counts as confirmed only if it survives that review AND is backed by an oracle or by documented manual review.
Model API costs (mutation generation, prosecutor/defender review): $7,000
Cloud compute for running SWE-bench containers and test suites: $5,000
Paid expert spot-review of borderline findings (contract engineer, ~40 h): $4,000
Founder time (part-time stipend, 6 months): $6,000
Hosting, domain, leaderboard site: $500
Contingency: $1,500
Total: $24,000
At the $10k minimum, we'd cover API and compute plus a small expert-review budget, audit SWE-bench Verified (sample) and LiveCodeBench only, and drop the stipend.
Milestones (6 months)
Month 1: publish the existing 9-benchmark audit and data (after disclosure windows close), plus the Kaggle inflation results (already measured).
Month 2: SWE-bench Verified test-strength results for a 50-task sample.
Month 3: LiveCodeBench and CRUXEval audits.
Month 4: BigCodeBench-Hard audit; leaderboard v1 goes live.
Months 5–6: full SWE-bench Verified run if the sample warrants it; notify maintainers; write up the results; final report on Manifund.
Solo founder: BliZi. BliZi is a US-based independent founder and digital artist who runs Eidolon, an online art business. BliZi has no formal engineering background and founded Shuna to make AI evaluation more trustworthy, directing the work while AI tools carry out the technical auditing.
To be upfront: I'm a non-technical founder. The audit pipeline was built and is run with AI coding agents, and I direct and review the work. That's why every finding has to survive an adversarial check plus oracle or manual confirmation, and why every item ships with a reproducible witness that anyone can check. The ~190 confirmed findings above are the track record. Part of the budget pays for independent expert review so that the results don't depend only on AI judgment.
False positives. Some "flaws" might be acceptable test behavior. Mitigation: a strict confirmation rule, public witnesses, and the expert review budget. When we reran the stricter check, we already downgraded some earlier counts on two benchmarks.
SWE-bench is harder to audit (repo-level tests, containers). We might only finish the sample. That's still useful.
Maintainers dispute findings. Every item is public and reproducible, so any dispute can be settled in the open.
If the project fails, the cost is a few months of API spend, and the code and data are still released.
$0. The project has been self-funded at roughly zero cost so far.
There are no bids on this project.