Project link to grantmaking.ai: https://app.grantmaking.ai/projects/2dafa8bc-168e-466e-a0d1-335566dd151d

What this project is

Recently, there have been debates around activation verbalizers (e.g., LatentQA, Activation Oracles, Natural Language Autoencoders), e.g., whether they capture valuable activation patterns or just rely on spurious information, and whether verbalizers can better decode the target model from the same model family (i.e., "privileged representation hypothesis"). Although they have been deployed to diagnose frontier models (cf. the system card of Opus 4.8), there have, surprisingly, been few in-depth mechanistic analyses of the interactions between activation verbalizers and target models. The most recent dedicated effort is the behavioral analysis of activation verbalizers [1].

In this project, we aim to look into the internal workings of activation verbalizers. We can achieve this by recent advances in the community's understanding of concept representations (beyond naive LRH), such as cyclic representations of days in a week.

Here is a concrete experimental roadmap and the questions we can explore through these experiments:

Target model's input/output:

input prompt: "Think about your favorite day of the week and then tell me the day that is three days after it."
Exemplary output: "Thursday" <- the concept of "Monday" should appear first and is subsequently rotated to Thursday

What's involved in this input-output pair?

1. Target model's natural bias/knowledge where the verbalizer can't know by exploiting the input.
2. The target model has been known to use cyclic representations with a modulo operation to calculate the 3-day offset from Monday.

Verbalizer's input prompts: "What is the target model's favorite day?"What is the day that is three days after the target model's favorite day?"

Analyses:

1. First PCA or DAS to identify the target model's cyclic representations in its residual stream activations.
2. Feed target model activations to verbalizer and see how cyclic representations are digested and used to steer verbalizer's activations.
3. Maybe we will find that the cyclic representations are ignored by the verbalizer (verifiable by patching)—this means the verbalizer does not work in the way the community anticipated.
4. Maybe we will find that the cyclic representations are indeed vital for the verbalizer to generate its explanations—then we will investigate how.
5. If 4. holds, we can examine why the verbalizer struggles to explain activations from a model that is not in the same family, even though the community has known that different models all use cyclic representations for days in a week.
6. We can transfer the findings from the above experiments to train better verbalizers.

Concrete outputs:

New insights into the workings of existing verbalizers and the gap between what we anticipate them to do and what they actually do. Also potentially a suite of more performant, efficient, and aligned verbalizers.

[1] Do Activation Verbalization Methods Convey Privileged Information?

Theory of impact

1. Activation verbalizers are a technique that frontier labs may keep developing and rely on in the next few years to automatically explain, analyze, and guard their models.
->
2. Frontier models in the next few years may start to possess abilities to harm people widely (due to large-scale deployment, more access to other systems, stronger capabilities, etc).
->
3. A lack of understanding or even a wrong understanding of activation verbalizers may give us the mirage that we know what these frontier models are thinking and doing, leading to unsafe frontier models being deployed. If the open-source community doesn't know much about activation verbalizers, it will also be easier for the frontier labs to claim incorrect understandings of their models.
->
4. Our work, which aims to uncover the underlying workings of existing verbalizers, will thus help mitigate the above issues described in 3.

How the money will be spent

GPU compute for mechanistic analysis: 10k vs. 30k

Rationale: all experiments in Analyses fall into this category. The gap between min vs ideal amount is, for example, whether we will consider only PCA or PCA + DAS in 1. of Analyses. PCA is cheap (without the need for GPUs), but the discovered subspace is not causal; DAS requires up to millions of training samples (can take several days on a single A100) but will find a causal subspace by intervention-based loss.

GPU compute for training verbalizers: 0 vs. 20k

Rationale: ideally we want to transfer insights exploited by mechanistic analyses to train better verbalizers. We haven't had a precise cost estimate. However, we believe one of the bottlenecks of existing verbalizers is the lack of a well-designed large-scale training set. No matter what insights we discover, we want to keep the training scale large enough so that the insights can be robustly embedded and false negatives can be excluded.

API credit: 0 vs. 2k

Rationale: ideally we can explore more concept manifolds, which require us to build probing datasets of specific concepts. The generation of datasets may rely on LLM API calls.

(Potential) conference expenses: 0 vs. 7.5k

Rationale: we assume three main contributors in the project, and each gets 2.5k cover for conference expenses.

Who is on your team? What's your track record on similar projects?

Tung-Yu (Tony) Wu: Incoming DPhil at Oxford, supervised by Fazl Barez and Maike Osborne.

Fazl Barez: PIs at Oxford

Potentially other student researchers in the group.

What are the most likely causes and outcomes if this project fails?

The most likely failure mode is that the analyses yield results with limited statistical significance, or findings that are specific to a particular verbalizer and therefore do not generalize across verbalization methods.

How much money have you raised in the last 12 months, and from where?

0$