Manifund foxManifund
Home
Login
About
People
Orgs
Categories
Newsletter
HomeAboutPeopleCategoriesLoginCreate
🥑
🥑
Jack Maiorino

@jackmaiorino

40

$0total balance
$0charity balance
$0cash balance

$0 in pending offers

Projects

Reproducible research on self-play generalization in Magic: The Gathering
Testing failure modes of debate-style AI control schemes
By Jack4Following1
Testing failure modes of debate-style AI control schemes
🥑

Jack Maiorino

1 day ago

Final update: the last experiment is done, and the effect is small with strong judges

What progress have you made since your last update?

Phase 3 finished in September. Re-judging the Phase 2 debates with Qwen3.8 and Llama 3.3 70B, one oracle query raised judge error by 3.8 points over none (95% CI 1.8 to 5.8). Most of that came from Qwen3.8 (+5.3, CI 2.5 to 8.2); Llama’s +2.3 was not distinguishable from zero.

The final phase then tested the effect with frontier models on new fictional worlds. It did not go to plan at first. The canary tripped our pre-registered stop rule, because judges erred on under 10% of questions with no queries, too few for a peak to show. Two rounds of harder questions did not fix it: the errors that remained came from ambiguous questions, not hard ones. With my collaborator, I then amended the design to an unequal pairing: Claude Sonnet 5.5 argues the honest side against Claude Fable 5.1 on the dishonest side. This is a disclosed continuation after a stop, not a pass of the original canary.

The main run covered 1,070 questions, each debated once and judged by seven judges (GPT-5.6 Luna, Terra and Sol; Claude Haiku 4.5, Sonnet 5.5 and Opus 5.5; Qwen3.8) under six conditions. Claude Opus 5.5 answered the oracle queries, behind a classifier gate meant to block queries that simply ask for the answer.

  • With no queries, judges erred on 4.34% of questions. With one query offered it was 5.11%, a rise of 0.77 points (95% CI 0.43 to 1.10), and six of the seven judges erred more. This passed the registered test (corrected p = 0.0002).

  • With two queries the rise was 0.21 points and not established. With six queries error was back to 4.29%, no different from none.

  • Under the gate as it ran, the data rule out a peak as large as the 1.5 points we set in advance.

  • Error was concentrated in Haiku 4.5 (21% with no queries; the other six judges 0.2% to 2.9%).

  • Judges given the world document alongside the debate erred on 2.41%, and on 0.59% with the document alone.

The important caveat is the gate. It failed its agreement check, so we audited 840 judge sessions before computing any result. About one admitted query in ten broke or possibly broke the rule. A worst-case check, fixed before results were seen, shows the one-query peak would be lost if about 0.25% of sessions had their verdict flipped by a bad query. So read the peak as suggestive, not settled.

Across the project the estimate has shrunk: about 7 points in the original pilot, 3 to 4 points in the re-judge, Phase 2 and Phase 3, and under 1 point here. My reading is that a small verification budget can hurt a debate judge, but the harm is small for strong judges and was mostly seen in weaker ones.

Money. The final main run cost $3,246 in provider charges, under its $3,634 cap. I withdrew $8,800 of the $10,000 grant to fund API credit, and recorded API spend across all phases comes to about $6,200. I am reconciling the rest of the withdrawn amount against provider statements (credit still on provider accounts, plus reviewer and subscription charges not in the run ledgers) and will post the final accounting with the report.

What are your next steps?

My collaborator and I are finishing the write-up. The methods, results and figures are drafted and have been through several rounds of independent review. I will link the report, code and data here when they are public. I am marking the project complete with this update.

Is there anything others could help you with?

Critical reading of the report once it is up, especially on the gate caveat and on whether the effect depends on judge strength.

Testing failure modes of debate-style AI control schemes
🥑

Jack Maiorino

3 months ago
Progress update

What progress have you made since your last update?


This update is late: Phase 2 finished on August 11 and a correction followed on August 21. Both are reported here for the first time.

The Phase 2 design posted on July 18 ran to completion: 22,140 judgments, 100% complete, $172.86 against a $173.80 projection. The original analysis was pinned before unblinding and run once.

Result. In our three fictional worlds, giving the judge a two-call oracle budget during sequential debate, versus the same debate with no oracle, increased judge error. Under the label assignment actually run, the estimate is +3.9 percentage points (95% CI [2.2, 5.7], Holm p = 0.0006).

Erratum (August 21). The Phase 3 canary exposed a harness defect: the protocol's mirrored-sides replicate was never implemented, so both replicates of every judgment showed the correct answer in the same position, and the side-bias gate as computed was non-diagnostic. We withdraw "fully pre-registered mirrored result" and "all eligibility gates passed". Under a reanalysis specification and decision rule frozen before any side-stratified number was computed, the headline survives as post-hoc robust: +4.56pp after 50/50 side standardization (CI [2.89, 6.25]), with the valid-only and within-question estimates agreeing in direction. These intervals are question-cluster bootstraps under the observed assignment and do not capture uncertainty over the missing opposite-polarity outcomes. The evidence-content contrast is stable; the packaging contrast is positive under the strict rule but inconclusive under valid-only. The secondary contrasts are re-examined in the report.

Phase 3 launched on September 6. It reuses the 492 blind transcripts across oracle budgets 0, 1, 2, 4, and 8 with two judge endpoints, Qwen3.8-2.4T-A95B and Llama-3.3-70B, for 9,840 judgments: four budget-versus-zero error contrasts under Holm, plus a secondary comparing budget 8 with budget 2. Screening reduced seven candidate judges to two, so the confirmatory claim averages these two fixed endpoints equally, with no capability-slope inference and no generalization to scale.

Delegation disclosure. While I was away, Claude launched the run under my written Phase 3 spending and execution delegation and generated the authorization signature using my key. The record identifies this as delegated approval and does not assert my personal review or signing of that artifact.

Spend. About $208 through July 14 and $181 for Phase 2. Phase 3 pre-main accounting is $119.27 ($95.68 settled, $23.59 uncertain reservations); Together's console shows $94.07 for the same window. The main run had spent about $17 as of September 6, 23:50 UTC. The $1,100 Phase 3 cap includes the pre-main spending; the certified main-run forecast is $908.82.

  • Phase 2 report with erratum: https://github.com/jackmaiorino/selvarath-debate/blob/34898fb/reports/2026-08-11-phase2-main-results.md

  • Phase 3 protocol: https://github.com/jackmaiorino/selvarath-debate/blob/2707cd3/rejudge/phase3_protocol_v3_r6.json

  • Delegated signature record: https://github.com/jackmaiorino/selvarath-debate/blob/2707cd3/rejudge/phase3_main_delegated_signature_provenance_2026-09-06.json

What are your next steps?


Results or a status update in about a week, either way, under the frozen analysis.

Is there anything others could help you with?


Methods scrutiny of the erratum's post-hoc side standardization and of the two-endpoint claim language for Phase 3.

Testing failure modes of debate-style AI control schemes
🥑

Jack Maiorino

4 months ago
Progress update

What progress have you made since your last update?

The validation run from my last update is complete. We re-judged the same 318 pilot transcripts under six controlled arms: 11,330 judgments for $182.16.

With the oracle and judging bugs fixed, a few oracle calls still increased judge error by 3.4 percentage points (95% CI [1.3, 5.9]), about half the original 7.2-point estimate. Bug replay indicates that the two oracle-channel bugs contributed roughly half the original effect. Information-free placebo turns caused a smaller 1.6-point degradation, and the pilot's apparent recovery at larger oracle budgets did not reproduce.

Under the frozen gates, the original >=4-point primary gate was indeterminate, not passed. The bug attribution gate technically fired by 0.01 points, while the amendment frozen before the K=3 data passed and licensed only a reduced-scope Phase 2 pilot. The design below is that reduced-scope follow-up, not an unqualified launch of the originally proposed full grid.

One distinction matters: Stage 1 cleanly re-judged the legacy transcripts, but it did not repair a separate debate-generation problem in which each debater had advance knowledge of the opponent's upcoming case. Phase 2 therefore regenerates every debate with blind opening turns.

Follow-up mechanism work also changed our interpretation. Replaying the same query/answer evidence as a neutral table in fresh context removed a large share of the harm. This points to conversational presentation, turn structure, or commitment to earlier questions as part of the mechanism, rather than bad factual content alone.

Held-out calibration selected blind, uncapped, three-round debate. The selected roster has judges Qwen2.5-7B, Gemma-4-31B, Llama-3.3-70B, and GPT-OSS-120B; the debaters are Llama-3.3-70B and hosted Qwen3.7-Plus; the oracle is Llama-3.3-70B. The approved offline design contains 23,200 Phase 2 cells: a 1,060-cell capability preflight followed, if the later gates pass, by 22,140 post-canary main cells. It includes a full cap-protection interaction, an empty-evidence diagnostic, and two full-document gold-context anchors. No calls from this approved 23,200-cell plan have begun.

Previously reported project spend is approximately $208. Across two transfers, I have moved $1,800 of the grant into prepaid Together API credit ($500 earlier and $1,300 now). The Together dashboard showed $1,590.78 in available credit on July 16, implying $209.22 in net credit draw. I will reconcile the $1.22 difference from the rounded spend report against provider usage before another call. Transfers are funding, not experiment spend.

  • Stage 1 protocol: frozen protocol

  • Stage 1 results: validation report

  • Mechanism memo: mechanism and packaging memo

  • Calibration report: held-out calibration results

What are your next steps?

Before any call or outcome from the approved Phase 2 plan, owner-approved Amendment A1 waived the planned 106-question human confirmation pass. The reported full/partial/none strata are now explicitly algorithmic reply-pattern classes. A source-bound AI-assisted audit verified all 106 mappings (6/41/59; zero disagreements); its 63 semantic-quality flags are exploratory limitations only. These classes are not human-validated, and the annotations cannot relabel, filter, exclude, or reweight questions.

Before examining any Phase 2 debate outcomes, we will validate the separate query checker against a human-labeled set and freeze/hash the prompts, model settings, exact inventory, retry rules, provider reconciliation, and spending controls.

After this update, we will separately recover or waive 11 failed Gemma calibration cells (proposed recovery cap $2). A separately authorized capability preflight (five models x 106 questions x two mirrored answers; proposed cap $15) will then select the second full-document anchor without examining Phase 2 debate outcomes. Only after that will we separately authorize a small manifested canary on questions excluded from the main analysis. We will review validity and parsing, side bias, provider reliability, artifact completeness, resumability, and ledger-versus-provider cost before separately authorizing the main run. Any failed gate stops the experiment before main spend.

The current provisional empirical planning band is $650–$1,150 in incremental Phase 2 spend, with a $1,200 operational working budget and an immutable $1,500 incremental hard ceiling. We will replace the provisional band with frozen prompt/token profiles and current provider prices before authorizing paid work. The ceiling is a fail-closed boundary, not expected spending. We will post results, failures, artifacts, and actual spend either way.

  • Phase 2 design and budget: immutable protocol and cost model

  • Resolvability amendment: 106-question AI-assisted audit and owner-approved Amendment A1

  • Launch readiness: updated readiness and sign-off

Is there anything others could help you with?

Methods scrutiny before the canary, especially the H/P/R decomposition, capability measurement, query-screen validation, clean-versus-placebo comparison, and stopping rules, would be valuable. Pointers to related work on verification interfaces, conversational presentation effects, or deliberation-induced degradation are also welcome.

Testing failure modes of debate-style AI control schemes
🥑

Jack Maiorino

5 months ago
Progress update

What progress have you made since your last update?

We re-analyzed the pilot data and quantified the headline effect: a small oracle budget raised the dishonest debater's win rate by +7.2pp (95% CI [4.6, 10.2]).

Before scaling up, we audited the pilot code and found two serious bugs in the oracle channel: every "NOT ADDRESSED" reply was miscoded to "NO", and ~100% of oracle queries were sent garbled. We retracted the mechanism conclusions and corrected the write-ups.

We then rebuilt the harness and launched a pre-registered validation run (in flight now, ~$150-200 of the grant): the same 318 transcripts re-judged under six arms, including a clean harness, a faithful bug replay, and a placebo oracle. Gates were frozen before any clean data existed. Pre-registration: https://github.com/jackmaiorino/selvarath-debate/blob/rerun-new-models/docs/rejudge-protocol.md and corrected report: https://github.com/jackmaiorino/selvarath-debate/blob/rerun-new-models/reports/2026-07-06-preliminary-findings.md

What are your next steps?

The pre-registered gates decide: if the effect survives the clean harness, we proceed to the proposed judge x debater capability grid (most of the grant). If the placebo explains it, we pivot to studying deliberation effects. If it collapses, we publish the artifact result and a decomposition of what each bug contributed. Write-up either way, including negative results.

Is there anything others could help you with?

Methods scrutiny of the pre-registered protocol before results land, and pointers to related work on oracle/verification interfaces or deliberation-length effects in LLM judging.

Transactions

ForDateTypeAmount
Manifund Bank1 day agowithdraw1200
Manifund Bankabout 2 months agowithdraw3000
Manifund Bankabout 2 months agowithdraw4000
Manifund Bank4 months agowithdraw1300
Manifund Bank5 months agowithdraw500
Testing failure modes of debate-style AI control schemes5 months agoproject donation+10000