Project summary

A frozen, public census of which benchmarks frontier labs report, so researchers and policymakers can see selective reporting and benchmark saturation over time.

Benchmark Radar is a live, public database and search engine for AI benchmarks and evaluations, with 42,745+ records gathered from public sources and updated daily. It is already used and cited: an arXiv paper (arXiv:2609.11115), a public Hugging Face dataset, a Zenodo DOI (10.5281/zenodo.22167102), and a Hugging Face #1 Paper of the Day.

The new deliverable is a reporting census. For each major benchmark, the census records how many frontier lab model reports and model cards included it in each release period, starting with every frontier model report from January 2024 to December 2026. Call this reporting prevalence. The census does not claim a benchmark is valid, and it does not claim to know what a lab runs internally. It records only what labs choose to publish.

No single model card can show two things. First, a lab quietly dropping a dangerous-capability evaluation that appeared in last year's model card and is missing from this year's. Second, a benchmark that is saturated or contaminated still being cited as evidence of progress. Only the comparison across many reports and many periods shows either pattern, and today nobody keeps that comparison in a frozen, citable form. A public census gives researchers, auditors, and policymakers a shared record that cannot be quietly edited after the fact, because each release is frozen with a DOI.

What are this project's goals? How will you achieve them?

Deliverable: Census v1, covering frontier model reports from January 2024 to December 2026, frozen with a Zenodo DOI, by month 3 after funding. After v1, the census updates with each new frontier model release, and the underlying data stays public in the Radar dataset.

How will you know it worked?

Success is measured by: v1 shipped on schedule with its DOI; independent citations and reuse of the census data; continued updates after each frontier release. Kill criterion: if the census has zero independent citations by month 9, I stop and write up what was learned instead of continuing.

How will this funding be used?

The budget is mostly my time. My time: $6,500 per month (a US postdoc-level rate: above the NIH postdoc stipend floor of $5,290/month and below the roughly $6,900/month equivalent paid to MATS fellows, neither of which is a like-for-like comparison because those come with employer benefits or housing and meals, and this budget provides none). Census v1 is 3 months of my time: $19,500. API and compute for report collection and extraction: $4,000. Infrastructure, hosting, storage for a year: $1,000. Census v1 total: $24,500. Full program: $50,000 (Census v1 $24,500, plus $25,500 for citation-tracking tooling and 12 months of census updates at part-time effort across the year). The paid period starts with my OPT in early 2027; I am an F-1 student until graduation in December 2026 and cannot be paid personally for this work before then.

Who is on your team? What's your track record on similar projects?

Koutian Wu, solo. I built Benchmark Radar solo and maintain it daily. Lead author of the Benchmark Radar paper and of a first-author NeurIPS 2026 workshop paper on provenance in AI-assisted research (arXiv:2610.28655). Core contributor to Apex ASI-Bench (task and scorer validity). Contributor to Petri, the alignment auditing tool, with two tested upstream fixes landed. I evaluate AI models for research at UT Austin, where I finish an M.S. in AI for Sciences in December 2026. Links: https://benchmark-radar.org, https://arxiv.org/abs/2609.11115, Zenodo DOI 10.5281/zenodo.22167102.

What are the most likely causes and outcomes if this project fails?

Reporting data is incomplete by nature: some labs publish thin model cards, and the census records absence of reporting, not absence of testing; the write-ups will say this plainly. My immigration status limits how grant money can be paid to me personally before my OPT begins in 2027, so the paid period is set to start with OPT, and I am confirming the arrangement with my international office before funds move.

How much money have you raised in the last 12 months, and from where?

None raised in the last 12 months. Grant applications are pending with BlueDot Impact Fund, the Transformative AI Fund, and Lightcone Commons, submitted November 2026; no decisions or funds received yet.