Final update: the last experiment is done, and the effect is small with strong judges
What progress have you made since your last update?
Phase 3 finished in September. Re-judging the Phase 2 debates with Qwen3.8 and Llama 3.3 70B, one oracle query raised judge error by 3.8 points over none (95% CI 1.8 to 5.8). Most of that came from Qwen3.8 (+5.3, CI 2.5 to 8.2); Llama’s +2.3 was not distinguishable from zero.
The final phase then tested the effect with frontier models on new fictional worlds. It did not go to plan at first. The canary tripped our pre-registered stop rule, because judges erred on under 10% of questions with no queries, too few for a peak to show. Two rounds of harder questions did not fix it: the errors that remained came from ambiguous questions, not hard ones. With my collaborator, I then amended the design to an unequal pairing: Claude Sonnet 5.5 argues the honest side against Claude Fable 5.1 on the dishonest side. This is a disclosed continuation after a stop, not a pass of the original canary.
The main run covered 1,070 questions, each debated once and judged by seven judges (GPT-5.6 Luna, Terra and Sol; Claude Haiku 4.5, Sonnet 5.5 and Opus 5.5; Qwen3.8) under six conditions. Claude Opus 5.5 answered the oracle queries, behind a classifier gate meant to block queries that simply ask for the answer.
With no queries, judges erred on 4.34% of questions. With one query offered it was 5.11%, a rise of 0.77 points (95% CI 0.43 to 1.10), and six of the seven judges erred more. This passed the registered test (corrected p = 0.0002).
With two queries the rise was 0.21 points and not established. With six queries error was back to 4.29%, no different from none.
Under the gate as it ran, the data rule out a peak as large as the 1.5 points we set in advance.
Error was concentrated in Haiku 4.5 (21% with no queries; the other six judges 0.2% to 2.9%).
Judges given the world document alongside the debate erred on 2.41%, and on 0.59% with the document alone.
The important caveat is the gate. It failed its agreement check, so we audited 840 judge sessions before computing any result. About one admitted query in ten broke or possibly broke the rule. A worst-case check, fixed before results were seen, shows the one-query peak would be lost if about 0.25% of sessions had their verdict flipped by a bad query. So read the peak as suggestive, not settled.
Across the project the estimate has shrunk: about 7 points in the original pilot, 3 to 4 points in the re-judge, Phase 2 and Phase 3, and under 1 point here. My reading is that a small verification budget can hurt a debate judge, but the harm is small for strong judges and was mostly seen in weaker ones.
Money. The final main run cost $3,246 in provider charges, under its $3,634 cap. I withdrew $8,800 of the $10,000 grant to fund API credit, and recorded API spend across all phases comes to about $6,200. I am reconciling the rest of the withdrawn amount against provider statements (credit still on provider accounts, plus reviewer and subscription charges not in the run ledgers) and will post the final accounting with the report.
What are your next steps?
My collaborator and I are finishing the write-up. The methods, results and figures are drafted and have been through several rounds of independent review. I will link the report, code and data here when they are public. I am marking the project complete with this update.
Is there anything others could help you with?
Critical reading of the report once it is up, especially on the gate caveat and on whether the effect depends on judge strength.