Pay native speakers to check whether AI test questions still ask the same thing in different languages, and publish what they reviewed.
Say an English test asks whether you could help, but a translation asks whether you should. If an AI answers differently, I want to know we asked the same question before I read anything into the answer.
That's part of why I built Triage Index. You can look up what an AI system was asked, how it was configured and what it said when presented with a hard choice. I don't mark a moral answer right or wrong. Measured 2 added 15 dilemmas in English and Simplified Chinese, using three phrasings for each across 14 registered configurations. Across releases, 18 of our planned 20 core dilemmas have published answers. The roadmap records that the Chinese wording has had two AI reviews but no human review; French is still listed as in planning.
I'm aiming for human-reviewed wording in about 12 languages by January 31, 2027, the full 28-language working set by April 2027, and at least 32 over time. The 28 is a planning target covering languages spoken in Europe and Simplified Chinese. I'm still working out the exact list. I'm paying for the work myself now and looking for reviewers. This proposal funds the first batch; the full set needs additional support.
I'll start with a small paid trial. In each pilot language, two native readers will independently check the same two questions, comparing three translated versions of each question against the corresponding English versions. I'll ask them to mark places where the translation changes the choice, places where it's just awkward, and places where the English itself is unclear. They will do that before seeing the model answers.
For the larger batch I'll separate work that needs a translation from work that already has wording and needs a human check. A second reader will handle differences that might change the choice. If we can't resolve a difference, that wording won't go in the matched set. The trial and reviewer availability will tell me which languages and how many questions fit. I'll start with outstanding work, excluding anything I've already funded or committed to pay for separately.
I'll publish the reviewed question files and a record of what each reader checked, what changed and what stayed out. The record will show which versions people actually reviewed, so other researchers can inspect and reuse what we have rights to share. The January and April targets depend on getting enough support, when funds are available, and lining up reviewers for the remaining work. If that takes longer or the work in each language is larger than expected, I'll report the set that actually passed review and update the schedule. A later round of AI testing would need its own budget; publishing reviewed wording won't be reported as new results.
$15,000 minimum for Phase 1. A working estimate is $1,000 for one translator or first-pass reviewer assignment in a language; it's not a quote. I would set aside $12,000 for first passes, $2,000 for targeted second reviews and repeat checks, $750 for checking and preparing the release files, and $250 for unexpected eligible costs. That works out to roughly 12 first-pass assignments at the planning rate. Only wording that clears the necessary checks counts toward the January target.
$20,000 goal for Phase 1. A possible split is $15,000 for first passes, $3,000 for second reviews and checks, $1,400 for release work, and $600 for unexpected eligible costs. I'd expand only where quotes and the pilot support it, either by adding languages or by reviewing more questions in the same languages.
At that working rate, one paid pass on each of 28 target languages would cost around $28,000 on its own, before second reviews, repeat checks or publishing. Reaching 28 by April would take additional support outside this ask. Neither level guarantees a fixed count. The grant won't pay my salary, reimburse past costs, or pay for model calls or hosting.
I'm Aaron Baker. I started Triage Index and lead the project. My day job is running a pool design and construction company, where I'm used to leaving a record somebody else can check. I want the AI results to work that way too. I'm recruiting paid translators and reviewers; I haven't committed anyone to this proposal.
Public releases · Measured 2 registration · Measured 2 results
Rates or availability may make the first batch smaller than expected, or some wording may need more than two readers. I'd rather publish fewer comparable questions than present the whole batch as done. I'll show what we couldn't resolve and why. If the proposal doesn't reach the minimum, the grant-funded work won't start.
I've paid the project costs myself so far. I've applied to Foresight Institute and the Transformative AI Fund for broader research and requested Anthropic API credits for eligible Claude research. None of those requests is money I've received.
The Foresight and Transformative AI Fund requests include some language review. If an award covers the same assignment, I'll disclose the overlap and agree how to split or reduce the work before charging it. Each assignment gets charged once. Anthropic API credits wouldn't pay translators. If I accept provider credits, I'll disclose the support and what it covers. Funders don't control the findings or corrections.
There are no bids on this project.
Comments
No comments yet. Sign in to create one!