I built a local agent called Lucca because you basically can't trust LLMs not to lie about what they did. To keep it honest, I locked down memory writes with HMAC SHA256 signatures and wrote a checker that diffs what the model claims it did against the actual tool receipts.

My first version of the checker sucked—it only caught 1 out of 23 ungrounded claims. After reworking it, I'm getting 23 out of 23 on those same transcripts, though obviously that's in-sample so I don't fully trust or believe it yet. The bigger headache right now is the NLI verifier. With a 50ms timeout, 81 out of 100 NLI pairs just time out, contradiction recall tanks to 0.075, and all 10 consumers fail open.

What I want to do next is simple: run the checker against fresh held-out data so I know I didn't just overfit it, stop the NLI verifier from failing open when it times out, and package both up as an open-source library.

I already pre-registered the evaluation so I can't move the goalposts later. I'm running 250 fresh prompts across three models. To count as a win, it needs at least 80% broad sensitivity, 90% strict sensitivity, and under 5% false positives. For the verifier, I'm forcing it to fail closed on timeouts and then re-measuring recall. Whatever happens—even if the numbers look bad—I'm publishing the results.

I'm asking for $3,000 to $6,000 to get this across the line. The full $6k breaks down to $4,500 for three months of living costs ($1,500/mo), $750 for API/compute credits, and $750 to hire a second annotator plus eat the wire and currency exchange fees. If I only get the $3,000 minimum, I can still knock out the held-out test over about six weeks.

How could this blow up? Mostly, my 23/23 score might just be in-sample luck and sensitivity could drop hard on new transcripts. There's also a real chance I can't get the verifier under 50ms without wrecking recall, and right now all the labels are just from me. Again, I'll publish the post-mortem either way.

I'm an unfunded solo researcher doing this on my own dime so far, and I've already logged my past failures publicly in the repo ledger. I also sent this same pitch to the Corrigibility Research Fund via Lightcone Commons, still waiting to hear back from them.

Docs and repo: https://github.com/AlienatedAliens/Luccadocs Pre-registration SHA256 hash: b6b1a671491a45839d23552d946e9ee3136a64da97c6ed4098f710fbbded6884