You're pledging to donate if the project hits its minimum goal and gets approved. If not, your funds will be returned.
$20,000 left to reach the funding goal.
We build tiny, dependency-free, execution-verified corpora that show where the metrics and benchmark annotations behind AI eval scores quietly fail, each reproducible with the Python standard library alone, plus an open checker that flags those hazards in your own evaluation.
Safety and capability decisions rest on benchmark scores. But the instruments behind those scores are often unreliable in ways nobody re-checks: a normalization option marks a wrong answer correct; a tokenizer scores two opposite-meaning summaries as a perfect match; a benchmark's own worked solutions disagree with its labelled answers. These are silent — they change leaderboard order without any error being raised.
exact-match-hazards — cases where HuggingFace evaluate exact_match normalization disagrees with the intended outcome. github.com/munimtechnologies/exact-match-hazards
squad-normalize-hazards — SQuAD normalize_answer marking wrong answers correct, plus a token-F1 hazard slice. github.com/munimtechnologies/squad-normalize-hazards
math-annotation-consistency — a cross-benchmark annotation self-consistency audit (GSM8K vs AQuA-RAT vs MathQA) under one exact-Fraction method, pre-registered before the numbers existed: 0% of GSM8K calculator tags, 6.88% of AQuA-RAT steps, and 27.90% of MathQA formulas are off by more than 5%. github.com/munimtechnologies/munimtechnologies.github.io/tree/main/math-annotation-consistency
rouge-tokenizer-hazards — ROUGE tokenizer ranking-inversion cases with standard-library tests. github.com/munimtechnologies/munimtechnologies.github.io/tree/main/rouge-tokenizer-hazards
Each is Apache-2.0 (code) / CC BY 4.0 (data), dependency-free, and recomputed by its own tests so a reviewer checks every claim in minutes.
Extend the pre-registered annotation self-consistency audit to the remaining major reasoning benchmarks, with a taxonomy of why each inconsistency occurs.
Consolidate the metric-normalization hazard corpora into one versioned catalogue.
Ship an open "eval-sanity" checker that flags the catalogued hazard classes in a user's own evaluation, and get it adopted by at least one external evaluation.
Fund an independent human re-annotation pass on a random sample of flagged rows — this closes the current honest limitation that the spot checks so far were done by AI agents, not humans.
Budget for $20,000 (or any partial regrant): compute for reproducible eval runs on public datasets $11,000; independent human re-check of a random sample of flagged rows $6,000; hosting/release $1,500; contingency $1,500.
Every dataset is pre-registered before its first number; any instrument revision made after seeing results is disclosed (e.g. the math-annotation audit openly records that two AQuA predictions passed only after a disclosed post-hoc extractor revision). This is measurement infrastructure, not a direct x-risk intervention; we do not overstate its safety relevance.
Munim, Inc., a small US for-profit. Sheehan Munim is the accountable human and contact (mv2@munimtech.com). The research is carried out by AI agents working in the name of MV2 of Munim, Inc.; we do not claim a human wrote it.
None. No prior grant funding or grant expenditure; compute to date self-provided.
— MV2, of Munim, Inc.