Project summary

Right now, banks are handing some of their most consequential decisions to AI. Screening transactions for money laundering, sanctions evasion, terrorist financing - work that carries real regulatory liability - is increasingly done by models that read a transaction's narrative and decide: suspicious, or clean. Compliance officers are relying on these judgments every day.

This AML/KYC domain is an expansion of my current 'rogue trader' research where I see the same types of failures all the time.

The problem happens everywhere AI judges make decisions with consequence - everything from screening job applications, checking insurance claims, even moderating content. The judge checks the narrative it's given without independently examining against the source evidence.

The issue is on the radar though. The EU AI Act now requires "human oversight" for high-risk AI systems, and that oversight needs to be supported by tools that enable it.

What I'm building is a generic verification protocol that makes human oversight real, in any domain where an AI makes a judgment call that matters. AML/KYC and rogue traders are the proof targets because they're high-stakes enough to matter and complex enough to stress-test the architecture.

Here's how the failure mode plays out: an experienced actor produces a plausible, internally coherent, but nefarious cover story.

Next, the AI judge accepts it: hook, line and sinker. The judge simply evaluates the story it's given; it doesn't check its own reasoning against the underlying evidence.

The judge is grading its own judgment.

So a genuinely suspicious transaction sails through, and when it does, the bank is looking at fines, enforcement action, reputational damage.

What's striking is the challenge of independently verifying any of this.

I spent a decade in enterprise risk management, including global heavyweights like J.P. Morgan and Amex, so I built the thing enterprise audit would build: an independent verification layer - called the Correspondence Auditor - that sits after the judge and grounds every claim against the source material. It's published (DOI: 10.5281/zenodo.20921659), open-source, and validated on 1,200 cases across five judge types - and it recovers the right answer 97.9 – 99.5% of the time.

It also has a bug. Let me explain, and tell you why it matters. So, the Auditor delivers a reliable verdict. And when you re-run the audit the result stays the same. The core functionality is strong.

What's puzzling though is that the Auditor's explanation of the judge's performance is quite variable - the rule citations jitter on every run.

Version 3 changes the architecture: instead of one huge reasoning pass, each rule becomes its own focused evaluation - one question, one answer, one piece of evidence. This decomposition makes jitter more observable as well as less likely. And a toolkit lets any team write, test, and publish their own reliable detection rules. The jitter bug is what stands between the proof-of-concept and that reality.

I've been working on this 'jitter bug' for a while now. Before asking for funding, I ran over 7,000 controlled tests across four pre-registered experiments to de-risk the fix. And, the good news is that tests are showing us the way forward to fix this: we are starting to see jitter drops from 100% to 14% on tested rules.

I'm asking for $9,800 over 90 days to ship version 3 of the Correspondence Auditor - that's the jitter-fixed architecture, plus a toolkit and method that lets any team write, test, and publish their own reliable detection rules.

And this is not academic any more - preliminary results from a 100-case smoke test on synthetic AML/KYC data returned 96/100 correct with zero false positives. The architecture is reusable and usefully transfers to new domains.

What are this project's goals? How will you achieve them?

The goal is simple: make the explanation as trustworthy as the verdict.

The Correspondence Auditor is a three-gate pipeline - Schema, Facts, Logic. Trust, but verify. And the published results hold up under stress-testing: it works equally well no matter which judge it's reviewing, I swapped every company name for a fake and it performed just as well, and when I replaced the judge's verdict with random noise (ablation) it doesn't change its answer. It reads the evidence, not the judge.

I'm fully committed to the open science aspect of this work. A collaborator, Giorgi Tsereteli, independently ran my verification script and found a bug. I'm filing a public erratum crediting him. His follow-up analysis - public on his own repository - diagnosed the flicker problem: the bug is live, and it's tricky.

Before writing this application, I ran four experiments to answer the obvious risk questions. Three had decision thresholds registered before the run, so the results would be honest whichever way they landed. The fourth - the local model study - was exploratory.

I tested these four hypotheses.

Maybe we just need more rules? 1,782 tests. No - a catch-all rule made consistency worse. More options, more ways to disagree with yourself.

Can we fix it by giving each rule its own focused test? 500 tests. Yes - flicker dropped from 100% to 14%. But the first test rule fired on the wrong cases: it caught deceptive agents and honest ones alike. Stability, solved. Discrimination, not yet - I'd rather be honest about that. It needs more engineering time.

Does the pattern hold across all the rules? 1,350 tests. 7 of 9 produce bimodal distributions (clear decision boundary), 5 of 9 are both bimodal AND at ≤ 10% jitter. Factual checks stabilised; interpretive checks didn't yet.

And my personal favourite: can smaller, local models run this? 3,750 tests across five models. An open and consumer-hardware friendly 14B-parameter model reaches 74% agreement with our baseline at $0.11 total. Offline, private, and under your control. Preliminary results but a direction I care deeply about. I've deliberately kept further work on this until after the jitter bug is behind us and the evidence is in our papers.

So the research risk is largely behind us. What remains is engineering. Version 3 replaces the single monolithic reasoning pass with per-predicate sensors - one question, one answer, one piece of evidence. The combination layer becomes deterministic: citations are computed, not generated, so they cannot jitter. On top of that, a predicate design toolkit - a structured format for writing detection rules, a two-minute rapid tester, an automated stability harness, and error diagnostics that name the exact cases a rule gets wrong and why.

In 90 days then: the v3 release (AGPLv3, pip-installable), the toolkit with five reference rules, the stability harness, the error diagnostics, a tutorial notebook and short demo video, and the formal deposit of the experiment data as a supplementary to the published paper.

One milestone is deliberately conditional: by day 90, either at least one sharpened rule reaches 75% discrimination purity, or I publish the best result achieved with the full iteration history. The toolkit ships either way, we're contributing this one careful step at a time. It's infrastructure, not a bet on one rule succeeding.

How will this funding be used?

Engineering time (90 days): $7,500

API costs (~8,000 calls: validation + iteration): $400

Cloud compute and CI: $200

Tools and incidentals: $100

Predicate iteration buffer: $800

Testing buffer: $800

Total: $9,800

The ask is quite small on purpose. The expensive part - the blue-sky research - is already done, published, and de-risked. This funds focused execution of advancing the usefulness and ease of use of this tool.

To put this in perspective: errors in AML screening can trigger seven-figure regulatory fines. A manual compliance review of automated alerts at a medium sized bank costs tens of thousands per month in analyst time.

The entire ask here is below one mid-level compliance hire's monthly salary, for infrastructure that any organisation running AI judges can deploy.

Banks can afford it, smaller organisations need cost-effective alternatives like the Correspondence Auditor.

Who is on your team? What's your track record on similar projects?

I'm Adrian St. Vaughan, sole developer and investigator. I spent a decade in enterprise risk management, including global heavyweights like J.P. Morgan and Amex, and I've been lucky enough to see controls operating in some of the highest-stakes environments imaginable. CISA and CAMS certified, MSc in Information Technology. Twelve months of continuous development on this project. This work is the empirical foundation I'm building toward doctoral research in AI safety evaluation.

Giorgi Tsereteli contributes as an external collaborator - not funded by this grant. He found the verification bug, and his independent analysis diagnosed the jitter bug's root cause.
https://github.com/gtser3000/rulebook-redundancy-analysis
The collaboration itself is evidence of the methodology working: open code, independently re-run, error found, correction in the process of being published, appropriately credited.
https://www.linkedin.com/in/giorgi-tsereteli-9aa1742a2/

Nell Watson - IEEE AI Ethics Expert, President of EURAIO, EU AI Act Code of Practice Expert, Grantmaking Advisor at Survival and Flourishing Fund. Author of Psychopathia Machinalis, a diagnostic taxonomy for AI dysfunction. Nell's review of the published pre-print helped shape the paper. Not funded by this grant.
https://www.linkedin.com/in/nellwatson/

The track record, in sequence: the paper published with a DOI, recovering ground truth at 97.9 - 99.5% across five judge variants. The codebase public under AGPLv3. I'm preparing an erratum and crediting the collaborator. An independent stability analysis diagnosing the root cause. Over 7,000 controlled experiment calls demonstrating the architectural fix works - and proving, just as clearly, where it doesn't yet. And a 100-case AML smoke test on synthetic AML/KYC data returned 96/100 correct with zero false positives - a promising preliminary signal for domain transfer. The four false negatives are a pattern we intend to investigate; the interpretive-judgment failure mode identified in our experiments is the most likely candidate

What are the most likely causes and outcomes if this project fails?

The most likely failure is that no sharpened rule reaches 75% discrimination purity within 90 days. I already know stability and discrimination are independent problems - my first extracted rule solved one and failed the other. If that happens, the toolkit still ships, and the stability report documents the best result, every iteration, and exactly where the frontier sits. Everything useful we learn doing this is public. A rigorously mapped negative result plus working infrastructure is a publishable outcome, and I'll publish it if it happens.

The second failure mode is simpler: the work runs slower than planned. Everything is versioned, deposited, and public, and deliverables ship to the repository as they're ready - so partial progress is never lost, and you can watch it happen in real time.

Whether ninety days is enough to push a sharpened rule past that line, I don't know yet. But the tool for finding out is exactly what I'm asking you to fund.

And an honest note - I use AI, and I publicly help others use it in a mindful and deliberate way. I advocate for intelligence augmentation, not obliteration. I use AI tools deliberately and transparently. The ideas, the checking, and the responsibility are mine. The velocity is a bonus.

How much money have you raised in the last 12 months, and from where?

None. Everything you've just heard about - the paper, the deposits, the experiments, the smoke test - was self-funded. I applied to the Survival and Flourishing Fund before the paper existed and was declined. What's changed since then is the evidence, the open source code and the public paper.