Note: I write in English with AI help and fix the text myself. The data, the code, the eval runs and the corrections log are mine; everything is in the repository and can be checked.
I worked 23 years in bank credit in Adana, Turkey. I still do; I run a branch. In my spare time I took the full 2025 US mortgage file (HMDA, 1,187,606 FHA credit decisions) and published what is in it, lender by lender, at financeratecalc.com.
The problem I ran into was not the data. It was what happens to a number after it leaves the page. I wrote “22.1% of decisioned FHA applications were denied in 2025”. An AI engine gave it back as “about a quarter of mortgage applicants get rejected”. Same number. Wrong sentence. The population, the program and the year were gone. No accuracy benchmark catches that, because 22.1 is still 22.1.
So I measure it myself. Every figure on the site carries a contract (what it means, what it excludes, how it may be restated) and a receipt: a short hash next to the number. Anyone can check a receipt and get one of four answers: current, stale (with the correction that replaced it), edited, or never issued by me. On 21 October I published the receipt format as a standard that does not depend on my site, Receipt 1.0, with a 12-test conformance suite. My own site passes 12/12. A second publisher needs one manifest file and one command.
On top of that sit an eval (Inspect AI, 12 fixed questions, four conditions: no tools, web search, my MCP server, a receipt-aware client) and an RL environment (verifiers). The rewards need no second model. Is the figure there. Is the receipt real. Was anything invented. Was a red line crossed.
What I have so far, one frontier model, 12 questions, 3 runs: the number was right 27 of 27 times. Full contract compliance was 1 to 11 of 36, depending on what the tool handed the model. The receipt survived 0 of 36 in plain prose and 23 of 36 when the client treated it as part of the figure. In 2 of 36 the model forged a receipt although every tool had given it a real one. The RL environment runs end to end: first rollout mean reward 0.75, 9 of 9 receipts valid, 0 forged.
Two things happened since I wrote the first version of this page, and both belong here.
On 14 October I had to reverse two of my own case files. I had accused an AI assistant of inventing a correction history. It had not. It had read my own log more carefully than I had. I withdrew the accusation in public, voided two benchmark questions for all eight systems, and re-checked every other accusation on the site with the method that found the mistake. All of it is in the corrections log, dated.
Between 16 and 19 October, GPTBot and Applebot followed receipts on my pages and verified nine of them on their own. That is a check, not a citation. It is also the first time anyone outside the project used the verify endpoint.
Everything is CC BY 4.0: the spec, the verifier, the tasks, every run log with its grader named, and my own errors. So far I have logged 16 error families on my own pages, with the same codes I use for the models. The claim snapshot is signed into the Sigstore Rekor log and the figures are rebuilt from the CFPB file in GitHub Actions with build attestations. Nobody has to trust my laptop. Including me.
Goals
Grow the suite from 12 to about 200 questions. The contracted claims already have machine-generated answer keys; this is mostly run time.
Add a second, independent grader from another provider. One grader is the weakest point of the pipeline and the write-up says so.
Fine-tune a small open-weights model on the deterministic rewards and test whether “carry the receipt, never forge it” transfers to claims it has not seen. If it does not, I publish that.
Get one outside publisher to run the Receipt 1.0 suite against its own site and say so in public, within 90 days. If nobody does, I report zero.
Use of funds
Minimum $6,000: about $2,500 API credit across two providers (200 questions × 3 runs × 4 conditions × 2 graders), about $1,500 compute for one small fine-tune, the rest a stipend for six months of evenings and weekends. I have a full-time job.
Full $18,000: a second outside publisher, a third grader, and part-time help with outreach for the 90-day adoption test.
Team and track record
Only me. Bank branch manager, 23 years on the credit side. Four SSRN working papers on this record (7156938, 7309319, 7341481, 7423798), a benchmark dataset on Hugging Face, an MCP server in the official registry. No funding to date. No lender money, no lead sales, no ads. The corrections log is the part of this work I am proudest of, and the 14 October entry is the part I am least comfortable with. Both stay.
If it fails
If the second grader disagrees with the first on most answers, my measurement is grader noise and I say so. If the fine-tuned model carries receipts only on claims it saw in training, the environment teaches memorisation, not behaviour. If no publisher runs the suite within 90 days, the format is a spec nobody enforces. All three go on the null-results page with a date, like every other log here.
Funding in the last 12 months: $0.
Links
Project and open data: financeratecalc.com
Receipt 1.0: financeratecalc.com/spec/receipt-1.0.md · manifest: financeratecalc.com/.well-known/receipt.json
Hugging Face dataset: huggingface.co/datasets/ziyayetis/fha-denial-reconciliation-2025
SSRN: ssrn.com/author=8328657
MCP server (official registry): registry.modelcontextprotocol.io, io.github.financeratecalc/frc-mcp; code: github.com/financeratecalc/financeratecalc.github.io/tree/main/worker
Corrections log: financeratecalc.com/corrections.html