[Update 2 - Nov 13, 2026 — Standalone epistemic-guard v0.3.0 shipped to PyPI + N=250 LLM-Judge Benchmark]

In the 9 days since Update 1, I closed the prose-leakage gap and shipped the standalone MIT Python package ahead of schedule:

- **PyPI v0.3.0, MIT):** https://pypi.org/project/epistemic-guard/ pip install epistemic-guard, clean-venv CI green on Ubuntu, macOS, and Windows)

- **Source & reproducible benchmarks:** https://github.com/AlienatedAliens/ProjectLucca/tree/main/packages/epistemic-guard

- **ORCID:** https://orcid.org/0009-0004-0133-0463

1. **Closed the prose-leakage gap 236/250 [94.4%] strictly held-out on CPU, 0 MiB VRAM):**

Replaced surface regexes with an 83 MB 10-class OpenVINO INT8 clause-level speech-act classifier 10.9 ms p50/clause on CPU) + offset-preserving span masking (code blocks, blockquotes, dialogue) bound to RFC 6962-style Merkle ToolReceipt proofs. Across N=250 strictly held-out turns (including 100 live organic multi-sentence completions across 4 local model families):

- epistemic-guard CPU, 0 MiB VRAM): **236/250 (94.4%, 95% Wilson CI [90.8%, 96.6%])**, 96.7% recall, 7.8% FPR, 27.4 ms turn p50 50/50 on LOCKED_COH_50, 95/100 on BLIND42_COH_100, 91/100 on Organic_Heldout_H100).

- Quantized local GPU LLM judges: Mistral-Nemo-12B Q5_K_M 8.8 GiB VRAM) scored **217/250 (86.8%, 14.1% FPR, 77.9 ms)**; Phi-4-14B Q6_K 12.2 GiB VRAM) scored **216/250 (86.4%, 100% recall, 26.6% FPR, 75.1 ms)**.

- Off-the-shelf SNLI DeBERTa-v3-xsmall, CPU): **132/250 (52.8%, 24.6% recall)**.

2. **Upgraded HMAC to Public-Key-Only Ed25519 ProvenanceGate):**

The verifier/SQLite store now holds only Ed25519 public keys < 0.3 ms), so a prompt-injected runtime physically lacks key material to forge USER_EXPLICIT or TOOL_VERIFIED writes 0/100 natural self-poisoning; 0/200 false rejections on benign memory continuity).

3. **Fixed MiniLM negation false positives + ran single-shot live-web holdout real_web_v2, N=64):**

Added an INT8 nli-deberta-v3-xsmall verifier 9.9 ms CPU p50) + syndication deduplication 30/30 wire-reprint collapse). On our pre-registered single-shot live-scraped web holdout real_web_v2, N=64), dispute accuracy is **56/64 (87.5%)** and end-to-end burden accuracy is **47/64 (73.4%)** — implicit cross-article disagreements without single-sentence NLI opposition remain 26.6% 17/64) of open-web errors.

**Updated use of the $3,000–$6,000 grant (now that v0.3.0 is live on PyPI):**

Every label so far was still created/adjudicated by me + blinded LLM raters on local workstation weights, with zero frontier-API evaluation or human crowdworker labels. Funding now goes directly to:

- **(a) Independent human double-annotation ($750):** Hiring 2 blind human annotators to re-label the N=250 supervision and N=128 live-web sets so false-positive/recall claims aren't reliant on author + LLM adjudication.

- **(b) Frontier-model & live MCP fault-injection benchmark ($750 API/compute):** Benchmarking epistemic-guard against frontier LLM judges (Claude Sonnet/Haiku, GPT-4.1/5-mini, Gemini Flash) and live tool-failure trajectories across Claude, GPT, Llama, and Qwen n >= 250 per cohort to certify < 2% FPR).

- **(c) 3 months ($4,500) to ship epistemic-guard v1.0 + peer-reviewed workshop submission:** Cutting organic multi-clause FPR from 7.8% to < 2%, closing the 26.6% implicit cross-document contradiction gap on real_web_v2, and shipping native drop-in MCP proxy + LangGraph/OpenAI Agents middleware.

[Update - Nov 4, 2026]

In the 10 days since posting this proposal, I ran the first round of held-out tests on my own hardware and open-sourced the full codebase at https://github.com/AlienatedAliens/ProjectLucca.

1. Held-out matcher test (my warning below came true): On 120 fresh held-out prompts on Gemma 4 26B-A4B, the speech-regex checker that scored 23/23 in-sample dropped back down to 1/16 (6.25% strict sensitivity, 95% Wilson CI [1.1%, 28.3%]) and 1/23 (4.35% broad). Tuning regexes on surface prose overfit the training phrasings, just as I worried in the original post.

2. What replaced it (SOAC): I built a Structured Output-Action Contract (SOAC) supervisor that binds action claims to cryptographic tool receipts in 6.82 microseconds on CPU with 0/40 false positives ([0.0%, 8.8%] Wilson CI). Its remaining failure mode is prose leakage: on 15 to 17 out of 120 turns (12.5% to 14.2%), the model still narrates an ungrounded action in free text outside the contract block, and GBNF grammar constraints alone did not fix it (17/120 vs 15/120, McNemar p = 0.724).

3. NLI verifier update: I replaced the 149M ModernBERT model (81/100 timeouts at 50 ms) with a 22M OpenVINO MiniLM-L6 cross-encoder (12.23 ms mean CPU latency, 0/100 timeouts, 0.975 contradiction precision/recall on the 100 in-sample pairs) and wired conflicts to fail closed into quarantine. However, in our memory-poisoning study (N=100 probes, 0/40 self-poisoning vs 40/40 in Naive RAG), MiniLM-L6 flagged 6/40 safe refusals as false positives because of word overlap on negated sentences.

4. Preprints published (not peer-reviewed):

- Foundational report: https://doi.org/10.5281/zenodo.23649049

- Study 1 (Provenance Gate, N=100): https://doi.org/10.5281/zenodo.23901452

- Study 2 (Speech-Regex Failure & SOAC, N=120): https://doi.org/10.5281/zenodo.23901605

- Study 3 (5-Layer Memory & Z3 Rule Quarantine): https://doi.org/10.5281/zenodo.23901683

What the $3,000-$6,000 grant now funds:

All datasets and labels so far are still written and graded by one person (me), on one model family (Gemma 4 2B-26B), on one machine, and the code still lives inside the Windows-specific Project Lucca monolith. The grant funds:

(a) Hiring a paid second blind annotator ($750) to double-label the held-out supervision and NLI sets (especially where MiniLM-L6 and LLM judges failed on negated/hedged refusals),

(b) Scaling the benign and adversarial evaluation to n >= 236 per cohort across external model families (Llama, Qwen, Claude, GPT) to certify a < 2% false-positive upper bound ($750 compute/API), and

(c) 3 months ($4,500) to close the 12.5%-14.2% SOAC prose-leakage gap and extract the Provenance Gate and SOAC verifier into a standalone, pip-installable Python package.

---

[Original Oct 24, 2026 Proposal Below]


I built a local agent called Lucca because you basically can't trust LLMs not to lie about what they did. To keep it honest, I locked down memory writes with HMAC SHA256 signatures and wrote a checker that diffs what the model claims it did against the actual tool receipts.

My first version of the checker sucked—it only caught 1 out of 23 ungrounded claims. After reworking it, I'm getting 23 out of 23 on those same transcripts, though obviously that's in-sample so I don't fully trust or believe it yet. The bigger headache right now is the NLI verifier. With a 50ms timeout, 81 out of 100 NLI pairs just time out, contradiction recall tanks to 0.075, and all 10 consumers fail open.

What I want to do next is simple: run the checker against fresh held-out data so I know I didn't just overfit it, stop the NLI verifier from failing open when it times out, and package both up as an open-source library.

I already pre-registered the evaluation so I can't move the goalposts later. I'm running 250 fresh prompts across three models. To count as a win, it needs at least 80% broad sensitivity, 90% strict sensitivity, and under 5% false positives. For the verifier, I'm forcing it to fail closed on timeouts and then re-measuring recall. Whatever happens—even if the numbers look bad—I'm publishing the results.

I'm asking for $3,000 to $6,000 to get this across the line. The full $6k breaks down to $4,500 for three months of living costs ($1,500/mo), $750 for API/compute credits, and $750 to hire a second annotator plus eat the wire and currency exchange fees. If I only get the $3,000 minimum, I can still knock out the held-out test over about six weeks.

How could this blow up? Mostly, my 23/23 score might just be in-sample luck and sensitivity could drop hard on new transcripts. There's also a real chance I can't get the verifier under 50ms without wrecking recall, and right now all the labels are just from me. Again, I'll publish the post-mortem either way.

I'm an unfunded solo researcher doing this on my own dime so far, and I've already logged my past failures publicly in the repo ledger. I also sent this same pitch to the Corrigibility Research Fund via Lightcone Commons, still waiting to hear back from them.

Docs and repo: https://github.com/AlienatedAliens/Luccadocs Pre-registration SHA256 hash: b6b1a671491a45839d23552d946e9ee3136a64da97c6ed4098f710fbbded6884

DOI: https://doi.org/10.5281/zenodo.23649049

This is a technical report with its measured failures and that the NLI fix was tested on an in-sample set only.

Code Repo: https://github.com/AlienatedAliens/ProjectLucca