6 months of compute and time to test, publish and open-source a calibrated verification protocol for AI-generated science
Igor
I'm Igor. I'm a lawyer, not a scientist by training, and since August I've been working on one question: when an AI research agent says it discovered something, how do we know it didn't just fool itself?
SDCP is my answer so far. Before each run the agent writes down what it expects and seals it with a hash, so it can't adjust the story afterwards. It also runs on "empty" worlds with no real pattern, which shows how often it sees patterns that aren't there. A separate reviewer, not the agent, opens the sealed results.
Since October I've been testing it against things I don't control: an AI forecasting bot on Metaculus, where reality decides who was right, and next, NASA solar-wind data.
I write all the code with sufficient help of AI, but I edit them myself to my vision.
Project page: https://claude.ai/artifact/UYFp2CXdF78dLEPTU4H6VU
1. Blinded real-data test: a structure is deliberately removed from the engine's vocabulary; false-discovery controls and success criteria are frozen and hashed before the run; the independent evaluator opens the result, and it is published either way.
2. Public verification protocol: a short, reusable specification.
3. One worked audit of a public "AI discovered X" claim using the protocol.
4. Open-source release of the tooling: data-custody kit, one-shot sealed executor, fail-closed evaluator with exact CRPS scoring.
Funding goal $5,000: $2,400 for 6 months of AI model usage (generator and evaluator models), $2,100 partial stipend (includes income tax), $500 (10%) buffer. The minimum on this page covers compute only; any amount above it extends the time I can work on this full time.
Just me, Igor Trofimenko, an independent, self-funded researcher, working with AI models: one generates, a separate reviewer session audits and can stop the work; I act as the authorizer. Until 11 October the reviewer was a model from a second company; since then it is a separate Claude session, the same model family, so its independence is weaker. Since August 2026 SDCP has produced calibrated false-discovery rates on null worlds, blinded runs opened only by the evaluator, an append-only record of my own withdrawn novelty claims and rejected engine versions, and a first pre-registered prediction on public NHANES data confirmed in an independent later cohort (preliminary; the effect is probably already known). No new discovery yet.
Most likely failure: the blinded test comes back negative. That is still a useful published result, because it measures the limits of an AI discovery engine under honest controls. Second: the protocol gets little uptake outside the project; the tooling and protocol stay public either way.
None received so far. Emergent Ventures (applied 2026-10-08, $12,000) declined on 2026-10-12. Pending: EA Funds Transformative AI Fund (applied 2026-10-08, $13,200). If more than one comes through, I will adjust amounts to avoid double funding.
Comments
Igor
24 days agoUpdate, 25 October: SDCP is now being checked against reality, plus two corrections
Since the proposal I have moved part of the work from my own synthetic worlds to tests I do not control.
1. Live forecasting under a pre-registered rule. Since 18 October an AI forecasting bot that follows the SDCP rules (every forecast logged before the outcome, nothing rewritten) has posted 69 forecasts in the Metaculus FutureEval Fall 2026 and MiniBench tournaments, and 13 in Market Pulse. Reality resolves them, not me. On 23 October, before any outcome was visible, I locked a calibration test (M1): measure whether the bot's 80% intervals are too narrow on resolved questions, freeze a correction, and test it only on new questions. The lock includes my own predictions (for example, P = 0.70 that the bot is overconfident). A separate review session read the evaluation code and found seven issues; all were fixed and recorded before any result was opened. First read-out: 2 November. It will be published whatever it shows.
2. Real physics data next. A shadow tournament on NASA ACE / OMNI solar-wind data is being designed: 72-hour forecasts sealed by hash before the data arrives, scored the same way as the FDL Benchlab Solar Wind Prediction 2026 tournament.
Two corrections to the proposal:
- Evaluator independence is weaker than I wrote. The evaluator from a second AI company has not been available since 11 October. Reviews are now done by a separate Claude session, the same model family. I report this as a real weakness, not a detail.
- Funding: Emergent Ventures declined on 12 October. EA Funds is still pending.
The minimum ($2,400) still covers six months of model usage for the generator and the reviewer.