Project summary

More and more researchers ask AI agents to write literature reviews for them. They do this because they will not read the papers themselves. I build such a tool, and I also built a citation checker for it. It tells you whether each cited paper exists and whether it says what the review claims. After watching people use it, I think this check gives readers a sense of control that they do not really have.

A review can pass every citation check and still mislead. It can leave out the studies that disagree. It can sound more certain than the evidence is. It can present a partial result as if it were settled. The reader has no way to see any of this.

In this project I want to work out what a research agent should tell a reader who cannot check its work, turn the answer into a standard with an open checker, and test whether it actually helps readers make better decisions.

What are this project's goals? How will you achieve them?

The first goal is a definition. I will write a short paper on what "keeping the reader informed" should mean for a research agent. I will compare my answer with three simpler options (full transparency, a confidence score alone, and citation checking alone) and show with real agent-written reviews where each of them fails. My starting list has four items. The agent should say what it left out and why, where the evidence was thin, where its confidence goes beyond the evidence, and which finding would change its conclusion. I expect this list to change while I write the paper.

The second goal is a standard and an open checker. The standard is a fixed set of statements that the agent gives next to its answer. The checker is code that scores an output on whether these statements are present, specific, and consistent with the sources the agent used. I will also ask an agent to pass the checker while keeping a wrong conclusion. This shows how easy it is to game the standard.

The third goal is a small pilot with known answers. I will use the MedEvidence set (arXiv 2505.22787), which pairs 100 findings of systematic reviews with the studies behind them. The agent writes each review from the open-access studies only, and then two groups read the reviews.

The first group is five paid expert readers with a graduate degree and research experience, from my own research network. Each of them answers 20 questions, ten with the disclosures and ten without, in a balanced order. This gives 100 decisions. Each answer is multiple choice with a confidence rating. Readers know in advance that some reviews may be incomplete, and they see the correct answers at the end.

The second group is several LLMs from different companies. They read the same reviews with and without the disclosures and answer all 100 questions. This gives a much larger sample for little cost.

I use the expert group to check the LLM group. If the disclosures change the answers of the models but not of the experts (or the reverse), I will report this as a result. The pilot should give an effect size, not a final verdict.

I looked for similar work before writing this. There are tools that find omissions in a summary, but they need the source documents, which the reader does not have (CARE, arXiv 2606.08969, and arXiv 2510.07926). There are studies on how people rely on AI explanations, but they use classification tasks (arXiv 2512.08424 and 2605.18036). A recent paper on debate shows that even a correct verdict lets an agent choose which true claims it presents (arXiv 2609.29189). I did not find any work on what a research agent owes a reader who cannot check it, or any test of whether such disclosures change decisions.

How will you know it worked?

The paper, the standard, the checker code and the anonymised reader data will be public, with a preprint first.

The pilot will give the difference in correct decisions with and without disclosures, with a confidence interval, separately for the LLM readers and for the experts. It will also say whether the two groups agree. The gaming test will show how far an agent can get through the checker with a wrong conclusion.

If the disclosures add nothing that a good review does not already give, I will publish that too. It is a real finding.

How will this funding be used?

USD 8,800 over six months, starting when the funding is confirmed:

• My time, 240 hours at USD 25: 6,000

• API costs for writing the reviews, running the checker and the LLM readers: 1,500

• Five expert readers, about three hours each, USD 150 per reader: 750

• Hosting and open release: 200

• Contingency: 350

With the USD 3,600 minimum I will do the first two goals (120 hours of my time for 3,000, API 400, release 200). I will run the pilot only if the rest is raised.

Who is on your team? What's your track record on similar projects?

I work alone, and I will pay five expert readers for the pilot.

I built pubscope.org by myself in 2026. Researchers use it to choose journals, to check reference lists against Crossref, OpenAlex, DataCite and Semantic Scholar, and to write literature reviews in which every citation is verified. Since May, 804 researchers from 90 countries have signed up.

The reference checker has a published validation (https://pubscope.org/citation-check/validation). I also published a study of publication times in 7,162 journals, measured from Crossref dates (TR-2026-01, https://doi.org/10.13140/RG.2.2.29171.28961).

I have a PhD in computer engineering and 11 SCI-indexed papers. Three of them, from 2026, are on LLM agents and their evaluation (https://scholar.google.com/citations?user=VP9ceqAAAAAJ). I work full-time as a senior research engineer on LLM agents at a large technology company, and I will do this project in my own time.

What are the most likely causes and outcomes if this project fails?

The definition may not hold up. The list could be too vague to check or too long to read. In that case the paper will explain why, and this is still useful for people who design agent outputs.

Five readers are too few to see a small effect. I know this. I will report the expert results with their interval and use them mainly to see if the LLM readers point the same way. If a reader drops out, another person from the same network takes the same questions.

The checker may be easy to game. That is also a result, and the paper will say which parts of the standard need a human.

In every case I will release the paper, the code and the data.

How much money have you raised in the last 12 months, and from where?

None in grants, donations or investment. The only support so far is USD 1,000 in API credits from Claude for Startups.