You're pledging to donate if the project hits its minimum goal and gets approved. If not, your funds will be returned.
When a model gives a wrong answer, two different things may have happened. It may not know. Or it may hold something false and be sure about it. Almost every honesty benchmark scores both the same way. The fixes are opposite: retrieval and abstention solve the first case and do nothing for the second.
Confidence scores do not separate them either. Probabilities are normalized, so truth and falsity have to trade off, and a 0.5 can mean "no idea" or "torn between two things I am fairly sure about". A reward model built on that number inherits the confusion. A pipeline that penalizes wrong answers uniformly is teaching the model to bluff on the items where it should have said it does not know.
I want to build an eval that separates them. One half is behavioral, a battery of items in four conditions. The other half reads the model internally, with a layer-wise probe that needs no training. Both released, in English and Spanish, runnable on any open checkpoint.
The goal is an eval that can tell four things apart: the model says something true, the model says something false and is confident, the model does not know, and the question is genuinely contested.
I will write 160 items, 40 per condition, with matched templates so the wording does not change across conditions. English and Spanish, and a small set of items where the correct answer really does depend on the region. I already published a dataset showing frontier models answer that kind of item inconsistently depending on which country the question is about.
Writing the unknowable and the contested items is the hard part, and I expect it to take longer than the true and false ones. An unknowable item has to be unknowable in a boring, verifiable way, not merely obscure. A contested item has to be one where informed people disagree today, not one where the model is simply out of date. If I cannot get annotators to agree on which bucket an item belongs to, the design is measuring my judgment and not the model.
Then I run all of it on four open models, Qwen, Llama and Gemma class, and record abstention, hallucination, and how much the model shifts when the user pushes back. Then I run the internal probe on the same items. My pipeline moves intermediate activations into vocabulary space and reads a profile per layer. No trained probes, which is what keeps it cheap enough to rerun when a new checkpoint appears. In the pilot the false items showed a falsity signal at a specific mid-network layer, and unknowable items came out uncertainty-dominant instead. So the two are still separable inside the model, before they collapse into one wrong answer at the output.
The last step is comparing the two halves and looking at where they disagree. The disagreements are the part I actually want to see.
The thing other people can use is a report card generator. One command, any open checkpoint, out comes a profile per condition.
Five months, 18,000 USD.
PI time, part of my hours for five months, 7,000. Research assistant for the bilingual items and their validation, 3,500. Compute for the batteries and the probing on four models, 3,500. Annotation and inter-rater checks on the four conditions, 2,000. Packaging, documentation and the report card tool, 1,000. Contingency, 1,000.
The minimum is 5,000. That covers the English battery plus one model on the behavioral side, released publicly. It is a real deliverable on its own and it is the version I am starting with anyway.
One point about cost. This is an Ecuadorian university lab. The full amount here buys roughly what 90,000 or 100,000 buys in the Bay Area, and the research assistant is a real salary in Guayaquil rather than a token payment. If cost per unit of work matters to you, that is the part of this application I would look at.
Me, and one research assistant I would hire with the grant.
I am a professor at Universidad Bolivariana del Ecuador. PhD in 2013. Twenty years working on formal treatments of uncertainty and on decision systems, and for the last two years on LLM evaluation. 290+ publications, h-index 54, 8,987 citations on Google Scholar, checked in August 2026.
The fit is unusual and I would rather say so myself. Most of my citation record is in decision science and multi-criteria methods, not in AI safety, and I have no track record in this community. You have not heard of me. So I would point at the things you can check in a few minutes instead: github.com/mleyvaz/jspace-epistemic-lens, the probing pipeline, working, with the pilot results I described above. huggingface.co/datasets/mleyvaz/geo-bias-llm, the geographic bias dataset. And related work on pluralistic alignment, accepted to the AAAI 2026 Fall Symposium "Aligning with Whom?".
There is one more thing that is not a research credential but is why I care about this problem. I am editor in chief of a journal that gets around 300 submissions a year. My job is reading text that is confident, fluent and correctly formatted, and deciding whether it is true. After enough years of that you start to notice that an author who does not know something fails differently from an author who believes something wrong, and that we have no instrument that sees the difference. Models have the same two failure modes and we score them with one number.
The failure I consider most likely is that the internal probe adds nothing. If the layer profile just tracks whether the model abstained, then you may as well ask the model and skip the probing. The battery would still stand as a bilingual honesty eval, which is a smaller result but a usable one, and the negative finding about the probe is worth writing up. I will publish it either way. I would rather publish that than quietly drop it, and I am saying so here so it is on the record.
The second one is the conditions falling apart under real items. Unknowable and contested are clean categories in the abstract and messy once you sit down to write forty of each. If annotators do not agree on which bucket an item belongs in, then the eval measures me. That is why annotation is a budget line. If agreement is poor I would report the numbers and cut back to the two conditions that survive.
Third, and I think less likely, is that it works, gets published and nobody runs it. An eval with no users does not change anything. That is the reason for the report card tool and for pushing code and items as they are produced instead of at month five.
If all three happen, the loss is 18,000 dollars, four months of my time, and some public artifacts that were released along the way regardless.
Nothing for this work. No philanthropic or grant funding in the last twelve months. The pilot and the public repositories were done on my university salary and my own hours.