You're pledging to donate if the project hits its minimum goal and gets approved. If not, your funds will be returned.
$10,000 left to reach the funding goal.
I created Lemma, an open-source agent that verifies papers claim by claim. When I tested the same claims with different model backends, Lemma agreed with itself just 46% of the time. This project is about that number, and how to interpret it.
Ag Lemma takes a claim, encodes it in a short number script, runs the script in a sandbox, says supported or falsified only if the internal checks pass, and says inconclusive otherwise.
Since a verdict is only worth anything if someone else can verify, I made the entire audit tamper-proof. All logs are hash-chained, and all files are included in a manifest, which collapses an audit into a single 256-bit 'evidence head'. Trust is thrown out the window: the verification process is to re-run the evidence, as each sealed workdir includes all scripts and data behind the verdict, and a single command recomputes the head.
It must be verifiable; but it must also be unstable. But neither of these is good enough. That is the task of this project.
Goals and how I'll reach them
Anchor evidence heads somewhere I don't have access to. Seal and verify are sufficient. I tested four tampering types, and a falsely extended record was only detected because of an external anchor. That's the limit, and I say so in the writeup. Next I'll put heads into a transparency log, or timestamped commits.
Score and stabilize verdict variation I'll perform k audits per claim on multiple model backends to track where verdicts switch and why. I've identified one to date - "falsified" verdicts were due to the fit window of the audit script, not the claim in the paper. Then I'll introduce consensus mode, where each verdict that remains "supported" will provide a stability score to prevent one fortunate run from seeming confident.
Lemma verify passes on all sealed workdir in corpus.
The tamper demo is still reproducible by someone other than me.
Agreement with error bars for at least three backends and about ten papers.
A no: even if the instability is simply high-frequency noise with no structure, I'll report that.
It is infrastructure, and it is free. The money that is raised here goes toward the measurement: approximately 60% to API compute (the consensus sweeps are run on each claim numerous times per backend), 30% toward part-time engineering (anchoring, consensus mode, verdict-confidence field), and 10% toward writeups.
It's just me, a solo independent builder. Lemma today: 10 finished sealed paper audits, an automated judge, sandboxed execution, a reviewer-reference mechanism, and a Hermes plugin in the plugin catalog. A pre-release version was successful in an ICML agent-reproductions challenge from start to finish. A 7 page writeup of what it does and where it doesn't can be found in the repo (sprints/epistemics-2026/report.pdf).
Likely causes and outcomes if this fails
The most probable failure mode is that this instability is fast and implementation noise and there is no learnable pattern, in which case consensus mode isn't useful. If that's not the case then the output is the deliverable: proof that one shot agent verdicts can't be trusted. The other failure mode is if no one implements sealing.
Funding in the last 12 months
None. I've paid for everything myself, including the API usage behind the corpus.
There are no bids on this project.