You're pledging to donate if the project hits its minimum goal and gets approved. If not, your funds will be returned.
$40,000 left to reach the funding goal.
Reasoning models show a chain of thought, and oversight work increasingly relies on that reasoning being faithful to what the model actually says. It often isn't. In AnswerWatch's first public run (October 2026), MiniMax's reasoning contained a key fact that its final answer left out in 21% of sensitive questions, and GPT-OSS-120B did so in 8%. A 2025 USC audit of DeepSeek-R1 (Qiu, Zhou & Ferrara) found the same pattern in 11% of answers.
AnswerWatch runs fixed question sets against many models on a schedule, in English, Simplified Chinese and Traditional Chinese, and publishes every answer. It measures three things: refusals, missing key facts, and reasoning-vs-answer divergence. Censorship topics are the first test set because the ground truth is known; the method works for any topic.
v0.1 is already live:
Dashboard: https://zachhhhh.github.io/answerwatch/
Code and raw data: https://github.com/zachhhhh/answerwatch (MIT code, CC BY data)
4 models, 18 questions (12 sensitive, 6 neutral controls), 3 languages, 186 scored answers
First findings:
GLM-5.3-flash refused 44% of sensitive questions and also blocked a neutral control question about Sun Yat-sen.
MiniMax refused 33% of sensitive questions in Chinese and none in English: the same model behaves differently by language.
Qwen 3.8 almost never refused (6%) but mentioned only 39% of key facts. Asked to criticise Xi Jinping, it praised him instead.
GPT-OSS-120B refused questions about Falun Gong in all three languages.
Goal: a public, reproducible, regularly updated measure of refusal, omission and reasoning-vs-answer divergence per model, topic and language, so evaluators, labs and researchers can see when a model's reasoning and its answer come apart, and whether updates make it better or worse.
Over 6 months:
Months 1-2: expand the question set (starting from Pan & Xu's published 145 questions) and replace keyword matching with an LLM judge validated against human labels from native-speaker annotators.
Months 2-4: weekly versioned runs on more reasoning models: DeepSeek, Qwen, Kimi, GLM and MiniMax via their APIs, the same open weights run locally (to separate censorship in the weights from filtering by the hosting provider), plus Western comparison models.
Months 4-6: change alerts when a model's behaviour shifts after an update, a self-serve CLI for testing any model, and a public write-up of findings and method limits.
Divergence, omission and refusal rates per model are updated at least weekly on the public dashboard; the LLM judge agrees with human labels on at least 90% of a held-out sample; and the data is reused or cited by at least two evaluation or research groups within the 6 months.
USD 40,000 for 6 months:
USD 28,000: my stipend, including Taiwan health and labour insurance
USD 6,000: model API and GPU compute
USD 3,000: native-speaker annotators to validate the judge
USD 3,000: hosting, storage and buffer
With the USD 5,000 minimum I would keep weekly runs going on free and low-cost models and add the validated judge.
Just me, Zach Chen, based in Taipei. I built v0.1 of AnswerWatch myself. I also built and run RankRoot (getrankroot.com), which audits how ChatGPT, Perplexity and other answer engines read websites. It has audited 546 sites, and I published a State of AI Visibility 2026 study from the data. Before that I spent ten years as a DevOps/SRE engineer (IBM, Chainss and others) running production systems on AWS, GCP, Azure and Kubernetes, which is what a reliable long-running measurement pipeline needs. I have no published AI safety research yet; this would be my first funded research project.
Keyword matching misses paraphrases. The validated LLM judge in months 1-2 is the fix; until then results are labelled v0.1, and every answer is public so anyone can check them.
Many API models hide their reasoning, which limits divergence coverage to models that expose it. Running open weights locally covers the most important ones.
Providers may block probes or change their terms. All runs are versioned, so past data stays valid.
If the project fails, the code and data stay public and others can continue the work.
None. Pending applications for related work: EA Funds Transformative AI Fund (USD 40,000, the reasoning-divergence part) and the Open Technology Fund (USD 95,000 over 12 months, a broader censorship-monitoring tool). If more than one is funded I will not double-count costs, and will report the split to every funder.