grantmaking.ai link: https://app.grantmaking.ai/projects/f6f8c4ea-ad05-48a4-9d06-a800d176b12e

Context

Activation Oracles (AOs) aim to uncover harmful behaviours from AI models, and they are currently tested on model organisms, i.e., otherwise-safe models fine-tuned to exhibit unwanted behaviours. Recently, I showed that AOs don’t reliably uncover unwanted behaviors if we remove the assumption of having a safe model to train the AO on. This assumption is unrealistic for two reasons: (i) it is not obvious when, during an actual training, an unwanted behavior arises, (ii) undesired behaviors do not arise in isolation.  For example, some behaviors emerge during pretraining, making it especially hard to extract a model without unwanted behaviour.
However fragile, the assumption of having a safe model at disposal is not unique to AOs. For instance, activation difference steering (ADL), patchscopes, SAEs all rely on the availability of a pre-malign-initialized model A.

Henceforth, let A be a pre-malign-initialized safe model, and B be a malign model trained from A that exhibits an unwanted behaviour X.

Project Summary

The project investigates whether it is possible to create a model C to use as a safe alternative to model A: starting from B, apply broad concept unlearning using a safe and controlled dataset to remove X. With broad concept unlearning I mean a training phase that aims to remove X by broadly compromising the model’s capabilities in a controlled way. Then use C in place of A for auditing purposes. For example, train an activation oracle from C, or compute the activation differences between B and C for steering or ADL. I will refer to C also as the surrogate base model henceforth.

Challenges and possible failures
The main challenge is calibrating the concept unlearning: removing B’s behaviour X while avoiding catastrophic forgetting. The latter is particularly important: erasing all B’s capabilities could make X’s activations buried among the activations of other erased features and behaviours.

The setting seems easier than normal training: without the objective of improving capabilities, we can tune the training methodology (such as datasets, techniques and parameters) to reduce behaviour X while still avoiding catastrophic forgetting.

The most likely failure mode is X being too hard to isolate. Beyond catastrophic forgetting (failure 1), the unwanted behaviour X may be particularly hidden (e.g. it only activates in a very niche context), making it hard to unlearn specifically (failure 2).

Experimental plan
The experimental setting is split into phases and aims to detect failure modes early.

In the first phase, we make two simplifications to ground the proposed technique in an easier setting:

1. The experiments will be run on ad-hoc MOs, with narrow behaviours (such as the ones studied here). These models are built in a way which makes them arguably easier to audit than real unwanted behaviour in LLMs. 
If the surrogate base model setting fails with these MOs, I will likely stop the project and publish the failures found.

2. Use targeted concept unlearning interventions, that is, concept unlearning on specific target contexts to directly address the behaviour X of the model. While in a real scenario the unwanted behaviour will be unknown (otherwise there is no need for auditing), here the aim is to understand how much information about the unwanted behaviour is needed to isolate and remove it.

If it is possible to create a surrogate base model with these simplifications, the project can move to the second phase where we try to relax them. I have less confidence in the following experiments, but I expect a better understanding and better scoping abilities after working through the first phase.

Simplification 1 makes the setting significantly far from being real: current MOs haveunrealistically salient activations  that are easy to isolate. This phase can test against the censorship of Chinese models (e.g., DeepSeek), or explore known undesired behaviours such as sycophancy or refusal/over-refusal.

Simplification 2 is more realistic. Even if in a real setting we are unaware of behaviour X, I believe iterative refinement can narrow down the context and uncover behaviour X. That being said, I aim to progressively relax simplification 2 to characterise how much narrowing is needed to actually uncover the behaviour.

Theory of impact

Using unsafe reference would make many mechanistic interpretability (MI) techniques blind by construction. More concerningly, these techniques would produce plausible results that do not show the potential unsafe audited models’ behaviours.

This silent failure mode would expose us to the risk of deploying a model that is not really safe.

Finding a surrogate reference would instead allow techniques to compare it with the audited models, addressing all those settings where it is not certain to have access to a safe ancestor of the model audited. This would significantly increase the number of auditing settings where MI would be effective in uncovering unsafe behaviours.

This project is a preliminary study to understand whether it is possible to create a more realistic setting for these techniques, either suggesting a new area of research or informing about a real weakness in current MI settings.

How the money will be spent

The requested amount covers research, living and workplace expenses, and I provide an estimate in the following spreadsheet. Many of the estimates are based on a budget estimation of a previous grant funding request that was accepted by Coefficient Giving.

Surrogate Base Model Budget

I would like to highlight the importance of working at the LISA workspace, which is where I am currently working. This office offers natural feedback loops, since residents often engage with each other’s work. It also allows me to easily meet potential mentors and collaborators. This has been happening for the last 7 months with talented fellows from AI Safety programs and other researchers who work here.

The minimum amount ($50k) would cover roughly half of the overall project expenses. If funded at the minimum, I will probably need to reduce living and research expenses, by developing part of the project outside of London and without working at LISA, reducing the amount of feedback and support received.

Team
Full-time on the project:
Raffaello Fornasiere: he is working in technical AI safety research since January 2026. He completed the ARENA 7.0, where he won the first prize at the interpretability hackathon. Then he participated in LASR Labs (mentored by Stefan Heimersheim) where, with his team, he published a paper on model organisms at ICML Interpretability Workshop 2026. Together with his LASR teammates (listed below), he has secured ~£200k in additional funding to continue this research independently.
In October he will participate in MATS at the GDM stream.

Previously he has worked for 3 years as principal software engineer at an Italian Startup and in 2025 collaborated with William Saunders (Anthropic alignment researcher) on an AI Agent framework.
He has a Master’s in Computer Science, with a thesis on using lightweight LLMs to extract medical information from Italian clinical documents, published at  ICNLSP-2024 in the ACL Anthology.

Part-time on the project:
Gabriel Konar-Steenberg holds a BA in computer science from Pomona College, as well as the college’s top CS award for his graduating class. Before LASR, he spent two years as a full-time computational researcher at the US Department of Energy’s National Laboratory of the Rockies, formerly known as the National Renewable Energy Laboratory, performing research software engineering on a multidisciplinary team, exploring ML surrogates for power grid optimization, and studying the application of neurosymbolic AI methods such as AlphaZero to program synthesis in an infrastructure planning context. He has served as a maintainer for several open-source scientific computing packages, presented at conferences on computer vision methods for autonomous vehicle navigation and the design of user-friendly data analysis libraries, and looks forward to the opportunity to further his pivot into AI safety. After this project, he will be conducting a fellowship with Redwood Research.

Nikita Menon is a recent MSc graduate in AI from the University of Edinburgh. She has participated in a range of AI Alignment research fellowships, including at MATS 4.0 (Agent Foundations), SPAR with Cadenza Labs, and AI Safety Camp. She has also conducted NLP research at the Indian Institute of Science, and has a background in Software Development and Engineering. Her AI Safety research projects span deception, mechanistic interpretability, evaluations and bias in LLMs.

Andrzej Szablewski is a recent graduate from the University of Cambridge. He has demonstrated his technical and research skills through several large-scale research projects, including the pre-training of the first open-source LLM for UK languages at University College London and, more recently, the robustness of multilingual tokenisation with researchers from Cambridge, ETH and EPFL. His work on SAE feature transport has been awarded spotlight at the NeurIPS 2025 Mechanistic Interpretability workshop. He serves as the team leader and holds an offer from the University of Cambridge to start a PhD in LLM reasoning and interpretability.