A disclosure standard for AI literature-review agents, an open checker, and a pilot test of whether the disclosures improve readers' decisions.
Researchers now ask AI agents to write their literature reviews, and the reason is simple: they will not read the forty papers themselves. I run one of these agents on pubscope.org, and I built a citation checker for it that confirms each cited paper exists and says what the review claims it says.
Watching people use the two together changed my mind about what the checker is worth. A review can pass every citation check and still leave the reader with the wrong answer. It drops the trials that disagree. It sounds surer than the evidence. It reports a partial result as settled. None of that is visible to someone who has not read the sources, and the green ticks from my checker make the review look more trustworthy than it is.
So the question for this project is what an agent has to tell a reader who cannot check its work. I want to answer it in writing, turn the answer into a short standard with an open checker, and then test whether it helps readers decide better.
First, a definition, in a short paper. "Keep the reader informed" sounds obvious until you try to say what it means for a review of forty papers. I will compare my answer with three simpler ones (show everything, give a confidence score, check the citations) and use real agent-written reviews to show where each one breaks. My starting list asks the agent to say what it left out and why, where the evidence was thin, where its confidence runs ahead of the evidence, and which finding would change its conclusion. I expect the list to change while I write.
Second, the standard and a checker. The standard is a fixed block of statements the agent writes next to its answer. The checker is code that scores that block: are the statements there, are they specific, and do they match the sources the agent actually used? I will then ask an agent to pass the checker while keeping a wrong conclusion, to see how cheaply the standard can be gamed.
Third, a pilot where the right answers are known. The MedEvidence set (arXiv 2505.22787) pairs 100 findings of systematic reviews with the studies behind them. The agent writes each review from the open-access studies only. Two kinds of reader then answer the clinical question from the review.
Five paid expert readers, each with a graduate degree and research experience, answer 20 questions each: ten with the disclosure block and ten without, in a balanced order, 100 decisions in all. Each answer is multiple choice with a confidence rating. They are told beforehand that some reviews may be incomplete, and they see the correct answers at the end.
LLMs from several companies read the same reviews and answer all 100 questions, with and without the block. This is the cheap, large sample. The experts are the check on it: if the block moves the models but not the people, or the other way round, that disagreement is itself a result. The pilot should give an effect size, not a verdict.
Before writing this I looked for prior work. Omission detectors exist, but they need the source documents, which is exactly what the reader lacks (CARE, arXiv 2606.08969; arXiv 2510.07926). Reliance studies exist, but on classification tasks (arXiv 2512.08424, 2605.18036). A recent debate paper shows that even a correct verdict leaves the agent free to choose which true claims to present (arXiv 2609.29189). I found nothing on what a research agent owes a reader who cannot check it, and no test of whether such disclosures change decisions.
Everything is public: the paper (preprint first), the standard, the checker code and the anonymised reader data.
The pilot reports the change in correct decisions with and without the block, with a confidence interval, separately for the expert and the LLM readers, and whether the two groups agree. The gaming test reports how far an agent with a wrong conclusion gets through the checker.
If the block adds nothing a good review does not already give, I will publish that as well.
USD 8,800 over six months, starting when funding is confirmed:
My time, 240 hours at USD 25: 6,000
API costs for writing the reviews, running the checker and the LLM readers: 1,500
Five expert readers, about three hours each at USD 150 per reader: 750
Hosting and open release: 200
Contingency: 350
At the USD 3,600 minimum I will do the definition and the checker (120 hours for 3,000, API 400, release 200) and run the pilot only if the rest is raised.
It is me, plus five paid expert readers for the pilot.
I built pubscope.org on my own in 2026. Researchers use it to choose journals, to check reference lists against Crossref, OpenAlex, DataCite and Semantic Scholar, and to write literature reviews with verified citations. More than 800 researchers have signed up since May.
The citation checker has a published validation on a sealed set of 1,320 references (https://pubscope.org/citation-check/validation). I also measured publication times for 7,162 journals from Crossref dates and published the study (TR-2026-01, https://doi.org/10.13140/RG.2.2.29171.28961).
I have a PhD in computer engineering and 11 SCI-indexed papers; three from 2026 are on LLM agents and how to evaluate them (https://scholar.google.com/citations?user=VP9ceqAAAAAJ). I work as a senior research engineer on LLM agents at a large technology company.
The definition may not survive contact with real reviews: too vague to check, or too long for anyone to read. Then the paper says why, which still helps people who design agent outputs.
Five readers cannot detect a small effect. I report their result with its interval and use it mainly to see whether the LLM readers point the same way. If a reader drops out, someone from the same network takes the same questions.
The checker may turn out to be easy to game. The paper will then say which parts of the standard need a human to judge them.
Whatever happens, the paper, code and data are released.
No grants, donations or investment. The only support so far is in credits: USD 1,000 in API credits from Claude for Startups and USD 2,000 in Google Cloud credits from Google for Startups.
There are no bids on this project.
Comments
No comments yet. Sign in to create one!