Final update and close-out (August 2026)

This update closes the project. Apologies that it comes just a bit late! The funded work stopped more than a year ago.

Funding. We asked for $670k. We received $80,680. Thank you, Austin, Adam, and all donors. Because we did not receive full funding and because of inevitable changes in priorities when doing the research, we did not ultimately try to complete all four projects. Other sources supplied the remaining runway, which included grants from the UK AI Security Institute (AISI), Coefficient Giving, and SFF.

Results for the four proposed projects:

1. Understanding-based evals. This work fed into our "singular psychometrics" project and then the SMDL paper. We did not end up publishing singular psychometrics. Other researchers took up some of the key ideas independently (such as the importance of item response theory), so the value of the project decreased. Some of the remaining ideas went into Compressibility Measures Complexity: Minimum Description Length Meets Singular Learning Theory (https://arxiv.org/abs/2510.12077), funded by UK AISI.

2. Backdoor detection with data-restricted LLCs. We did not build a backdoor detector. But someone did independently propose and complete a similar project, with successful results from using the Bayesian Influence Function (BIF) for backdoor detection. The BIF is closed related to data-restricted LLCs.

3. Simplicity biases and deceptive alignment. The conceptual work became our position paper, You Are What You Eat (https://arxiv.org/abs/2502.05475), on the "Occam gap". The applied work continues in Patterning: The Dual of Interpretability (ICML 2026) (https://arxiv.org/abs/2601.13548), which targets reward-model biases. We are continuing research inspired by this thread.

4. Many-shot jailbreaking. We did not do this project. We studied related questions in the project, Elicitation without Backpropagation (https://arxiv.org/abs/2607.18804).

Timaeus has grown a lot in the meantime, and has now merged into Resolution.

Thank you again for your support!

Copying my comment over from the previous listing here.

Note: I didn't fund this but I recommended this project to JueYan Zhang, who funded it.

Points in favor

1. We need more research on interpretability-guided training! There's been a growing interest in "interpretability-guided training": see Goodfire's "intentional design", Neel Nanda, gradient routing and SGTM, and our own work at Timaeus. I think this is an area that is both extremely important and extremely neglected. I'm concerned that a lot of harmful structure in models is hard to remove once it's been trained in. In such cases, prevention is a much better approach than post-hoc treatment.

2. Well-scoped. The work on gradient routing especially demonstrates that, in this line of work, it is tractable for small, independent teams to make meaningful progress. The technique being proposed seems straightforward and practical. You'd think people have already tried regularizing against internal objectives throughout training, but actually, there really hasn't been much work on this. So someone should study it! In addition, the work plan sketched out here seems mostly reasonable (with the only exception being step 4, which is probably still a significant underestimate).

3. Sandy can handle it. From personal experience (COI: I previously employed Sandy as a research engineer at Timaeus), I know that Sandy is conscientious and great at visualizing and presenting information. He can deliver, as his initial milestone demonstrates.

Main reservations

1. Research outreach. My main worry is that this research will fall on deaf ears. The first milestone got very little attention. This is understandable given that Australia's AI safety community, though quite large in relative terms, is also still far from the central AI safety nodes in the Bay Area and London. Sandy is aware of this risk and has proposed some mitigations under "Potential Impact", but it remains a risk.

2. Transfer to real-world settings. I'm typically a fan of working on small toy systems before scaling it up to larger settings. In this case, I think there's a risk of jumping to (small) language models too late. And subsequently a risk of jumping to large language models prematurely. I'd encourage Sandy to be willing to spend relatively more time on milestone (3), even at the expense of dropping milestone (4). I'm a bit wary of jumping straight to concepts as high-level and abstract as "deception." What simpler interventions can you demonstrate in small language models?

Note: I didn't fund this but I recommended this project to JueYan Zhang.

Points in favor

1. We need more research on interpretability-guided training! There's been a growing interest in "interpretability-guided training": see Goodfire's "intentional design", Neel Nanda, gradient routing and SGTM, and our own work at Timaeus. I think this is an area that is both extremely important and extremely neglected. I'm concerned that a lot of harmful structure in models is hard to remove once it's been trained in. In such cases, prevention is a much better approach than post-hoc treatment.

2. Well-scoped. The work on gradient routing especially demonstrates that, in this line of work, it is tractable for small, independent teams to make meaningful progress. The technique being proposed seems straightforward and practical. You'd think people have already tried regularizing against internal objectives throughout training, but actually, there really hasn't been much work on this. So someone should study it! In addition, the work plan sketched out here seems mostly reasonable (with the only exception being step 4, which is probably still a significant underestimate).

3. Sandy can handle it. From personal experience (COI: I previously employed Sandy as a research engineer at Timaeus), I know that Sandy is conscientious and great at visualizing and presenting information. He can deliver, as his initial milestone demonstrates.

Main reservations

1. Research outreach. My main worry is that this research will fall on deaf ears. The first milestone got very little attention. This is understandable given that Australia's AI safety community, though quite large in relative terms, is also still far from the central AI safety nodes in the Bay Area and London. Sandy is aware of this risk and has proposed some mitigations under "Potential Impact", but it remains a risk.

2. Transfer to real-world settings. I'm typically a fan of working on small toy systems before scaling it up to larger settings. In this case, I think there's a risk of jumping to (small) language models too late. And subsequently a risk of jumping to large language models prematurely. I'd encourage Sandy to be willing to spend relatively more time on milestone (3), even at the expense of dropping milestone (4). I'm a bit wary of jumping straight to concepts as high-level and abstract as "deception." What simpler interventions can you demonstrate in small language models?

What progress have you made since your last update?

• See our recent update, "Timaeus in 2024," for a high-level overview of our research progress in 2024.

• Because this Manifund proposal was not fully funded and because progress in separate research projects opened up new research possibilities, we decided to direct our attention immediately to the last of the projects we describe in this proposal: understanding-based evals, under the heading of "Singular Psychometrics."

• We've been working on this project in partnership with the UK AISI and are on track to finish this project by the end of March 2025. As described in the update linked above, we have successfully overcome the engineering obstacles required to scale LLC estimation to models with billions of parameters. This unblocks the primary hurdle to seeing this project to completion.

What are your next steps?

• We're currently working on the final stage of the singular psychometrics project. Our hope for this project is to use SLT-derived metrics to differentiate how different models achieve the same level of performance. Can we distinguish a model that has memorized a given benchmark from one that truly generalizes on that benchmark using the local learning coefficient?

Is there anything others could help you with?

• Not currently. We're looking forward to sharing the final update in April.

Let me copy the earlier progress update we shared (which was meant to close the project):

We've posted a detailed update on LessWrong.

In short:

• We consider this project a major success: SLT & DevInterp's main predictions have been validated in a number of different settings. We are now confident that these research directions are useful for understanding deep learning systems.

• Our priority is now to make direct contact with alignment: It's not enough for this research to help with understanding NNs, we need to move the needle on alignment. In our update, we sketch three major directions of research that we expect to make a difference.

In more detail, with respect to the concrete points above.

• Completing the analysis of phase transitions and associated structure formation in the Toy Models of Superposition (preliminary work reported in the SLT & Alignment summit’s SLT High 4 lecture). See Chen et al. (2023).

• Performing a similar analysis for the Induction Heads paper. See Hoogland et al. (2024).

• For diverse models that are known to contain structure/circuits, we will attempt to:

  • detect phase transitions (using a range of metrics, including train and test losses, RLCT and singular fluctuation),

  • classify weights at each transition into state & control variables,

  • perform mechanistic interpretability analyses at these transitions,

  • compare these analysis to MechInterp structures found at the end of training.

Classifying transitions into state & control variables remains to be done in the next few months. We have performed some mechanistic/structural analysis, and more of this kind of analysis is currently underway.

In more detail, with respect to the concrete points above.

• Completing the analysis of phase transitions and associated structure formation in the Toy Models of Superposition (preliminary work reported in the SLT & Alignment summit’s SLT High 4 lecture). See Chen et al. (2023).

• Performing a similar analysis for the Induction Heads paper. See Hoogland et al. (2024).

• For diverse models that are known to contain structure/circuits, we will attempt to:

  • detect phase transitions (using a range of metrics, including train and test losses, RLCT and singular fluctuation),

  • classify weights at each transition into state & control variables,

  • perform mechanistic interpretability analyses at these transitions,

  • compare these analysis to MechInterp structures found at the end of training.

Classifying transitions into state & control variables remains to be done in the next few months. We have performed some mechanistic/structural analysis, and more of this kind of analysis is currently underway.

We've posted a detailed update on LessWrong.

In short:

• We consider this project a major success: SLT & DevInterp's main predictions have been validated in a number of different settings. We are now confident that these research directions are useful for understanding deep learning systems.

• Our priority is now to make direct contact with alignment: It's not enough for this research to help with understanding NNs, we need to move the needle on alignment. In our update, we sketch three major directions of research that we expect to make a difference.

Hey Rachel, thanks for the suggestion! We decided to wait a little longer to think about this, and it seems no longer necessary.