1
Prediction of Inoculation Prompt Side Effects
ProposalGrant
Closes September 12th, 2026$0raised
$5,000minimum funding
$13,610funding goal
Offer to donate
31 daysleft to contribute
You're pledging to donate if the project hits its minimum goal and gets approved. If not, your funds will be returned.
Project summary
A common analogy when describing LLMs is that models are grown, not built, and for this reason, models often learn traits we’d rather not have, such as malice or sycophancy. Recently, new train time techniques, during finetuning, like Inoculation Prompting (Tan et al., 2025; Wichers et al., 2025) have emerged to inoculate against the effects of undesired traits like malice. However, despite progress in the space, no one knows what off-target traits an IP might inoculate; we are shooting in the dark (Tan et al., 2025; Riché & nielsrolf, 2026). We can check whether an IP inoculates a specific trait once we have a clear trait definition, but to check all off-target alternatives is computationally infeasible, nor is it possible to know in advance what might be accidentally inoculated against. Broadly, the planned method first constructs an “Inoculation Vector” by reading the activation shift a candidate prompt induces on the model’s activations, decomposes the vector into interpretable features using SAEs, and outputs a prediction of which traits the IP suppresses or amplifies, including off-target traits that weren’t the intended target. Notably, this method relies only on activations from inference and publicly available or one-time costs (SAEs) that model developers commonly train anyway, rather than numerous costly finetuning runs.
Importantly, to validate this hypothesis, I ran proof-of-concept experiments on various toy and real environments (e.g., GCD sycophancy) where predictions before training correctly identified key traits with high correlation (r≈0.8, based on ground truth fine-tuning results) and maintained effectiveness on out-of-distribution prompts (r≈0.6). Moreover, I evaluated the method against numerous naive baselines (e.g., lexical baselines) to justify the added complexity, and iteratively designed the experiments to avoid confounds (e.g., conditionalization). I plan on running experiments to investigate the mechanism and validate the method across various model families, sizes, and environments (Gemma2-2B-it, Qwen3.5-4B, Qwen3.6-27B, etc.).
I'll be working independently with the help of my project lead at my current research internship/fellowship (Eleuther SOAR) and plan to informally seek the advice of other researchers in the Eleuther community. As an output, I plan on creating an open-source tool and a peer-reviewed paper (goal venue: TMLR). Additionally, even in the case where the method is unfruitful, a negative result will still elucidate useful information about IP's mechanism and effects, which can inform the use and/or future method development.
I'm currently an AI Safety Research Intern at Eleuther AI as a part of their Summer of Open AI Research (SOAR) program (One of 70-80 selected worldwide out of 3500+). I have also been a part of Bluedot's Technical AI Safety Course (a selective, cohort-based class), have completed ARENA's course, and am actively engaged in the AI safety community by taking part in reading groups, for instance. Before my work in AI safety, I did work in BioML, collaborating with researchers at UCSD (IEM), UCSC (Marinescu Lab), and Cambridge (Department of Genetics), with much of my work in the publication pipeline at the moment (e.g., MLCB 2026).
What are this project's goals? How will you achieve them?
Inoculation Prompting (IP) is one of our best defenses for preventing models from learning undesired traits, and due to its effectiveness and ease of use, it is being increasingly used at frontier labs (e.g., Anthropic) during production of their frontier models (e.g., Mythos; see Mythos System Card Section 5.2). In the future, the use of IP will only continue to grow beyond the small-scale use in the status quo.
However, IP, as shown by recent work (Tan et al., 2025; Riché & nielsrolf, 2026), has the potential to inoculate off-target positive traits to a significant extent. Ultimately, this means that as IP becomes more prevalent, the potential negative effects of a bad inoculation prompt have the potential to compound undetected, for the effects of a bad IP are difficult to detect in standard benchmarks and often only manifest in very specific but harmful manners. Specifically, the consequences of unintended off-trait inoculation are a shift in the trait/persona, which could cause a model to be more malicious, more prone to hallucination, and a plethora of other negative effects from the downstream consequences of a shift in traits/persona (e.g, human-AI interactions, misinformation).
As such, funding my work will therefore directly reduce x-risk in two ways. Firstly, it will enable the development of a robust method to detect off-target inoculation (critical as IP scales). Secondly, it will increase understanding of IP's mechanism to help model developers make safer and more robust models by being aware of the effects various IPs have on the model’s traits/persona.
How will this funding be used?
Who is on your team? What's your track record on similar projects?
This is an individual project, and my track record on similar projects has been effective. My past research regarding Inoculation Prompting resulted in the team I was a part of creating a novel, cheap heuristic capable of identifying high-quality inoculation prompts without expensive finetunes. Critically, it enabled model developers and any individuals using inoculation prompting to mitigate negative trait acquisition much better than current methods.
What are the most likely causes and outcomes if this project fails?
The main technical challenges and causes of failure are confounds regarding conditionalization. Conditionalization is a known confound where adding any instruction to the training data can inoculate traits generically, rather than a specific prompt inoculating a specific trait. If not managed carefully, conditionalization can undermine the results and implications of the work. We fix this by comparing each prompt to the average prompt rather than to no prompt, so shared effects cancel out, and by requiring the method to predict beforehand which specific trait each prompt hits and which it leaves alone.
On the non-technical side, the main risks are that I am working solo and that this is a fast-moving area. I mitigate the first by having already built and tested the full pipeline end-to-end, so the remaining tasks are primarily execution rather than extensive engineering. Additionally, my current mentor at SOAR will be informally advising the project, so I will be able to leverage his expertise to maximize the use of resources. Lastly, since I have already worked on a problem in this specific research niche for a while, I am well-versed in the related literature and will be able to make impactful contributions.