Independent technical AI safety research investigating whether activation-based deception probes detect deceptive behaviour or rely on superficial correlations.
I am an independent researcher working on a technical AI safety project through the TARA Fellowship, where I receive research guidance from an assigned teacher.
The question I want to investigate is whether we can rely on a language model's internal activations to detect deceptive behaviour, or whether the methods used to detect deception sometimes pick up on unrelated patterns instead.
As language models become more capable and are given more autonomy, it is becoming increasingly important to understand how we can monitor their behaviour. Looking only at a model's final answer may not always be enough. This has led researchers to explore methods that examine a model's internal activations.
One approach is to train linear probes on these activations to classify whether a model is behaving deceptively. However, I am interested in a problem with this approach: does the probe actually learn something related to deception, or does it learn superficial patterns that happen to appear in the training data?
For example, a probe might classify an honest response as deceptive because it has a particular writing style or appears in a role-play scenario. It might also fail to recognise deceptive behaviour when it appears in a different context.
I want to understand when and why these errors happen, what internal features might be contributing to them, and whether there are ways to reduce them.
My starting point is Apollo Research's work, Detecting Strategic Deception Using Linear Probes. ( https://www.apolloresearch.ai/science/detecting-strategic-deception-using-linear-probes ) I plan to build on this and related research by running controlled experiments on an open-weight language model.
My goal is to produce a small, reproducible research project with experimental code, quantitative results and a report explaining what I find, including results that do not support my initial expectations.
My main goal is to investigate whether linear probes trained to detect deception generalise to unfamiliar examples or rely on superficial correlations in their training data.
I plan to approach this through three main experiments, followed by documenting the results.
1. Establish a baseline
First, I will reproduce or adapt an existing linear-probe approach using an open-weight language model.
I will extract activations from selected transformer layers, train a baseline probe and evaluate it on held-out examples. This will give me a starting point for understanding how well the probe performs and how its performance changes across layers.
I will begin with one model rather than trying to cover several models at once. This should help me keep the experiments manageable and focus on understanding the results.
2. Investigate when and why the probe makes mistakes
Next, I will examine false positives and false negatives to understand the situations in which the probe gives incorrect predictions.
I will construct controlled evaluation examples that vary factors such as wording, response style and context while keeping the underlying behaviour being tested consistent.
For example, I want to investigate whether changing the writing style or surrounding context can change a probe's prediction even when the underlying behaviour has not changed.
I will also compare selected transformer layers and activation configurations. Where possible, I will investigate the internal features associated with particular failure cases.
I am interested in resources such as Neuronary and the Neuron Persona Library, but I will only use them if they are compatible with the model I select and useful for answering the research question.
3. Test whether the failures can be reduced
Based on the results, I will select at least one failure mode and investigate a possible mitigation.
This could involve diversifying the training examples or testing targeted interventions on features associated with incorrect predictions.
I will compare the results against the original baseline to see whether the intervention improves performance on unfamiliar examples without introducing other problems.
I do not expect every intervention to work. If a mitigation does not improve generalisation, I will report that result rather than presenting it as a success.
Research outputs
I will make the experimental code, evaluation scripts and relevant documentation publicly available. I will also write a report explaining the methodology, results and limitations.
If time permits, I may explore an alternative activation-classification approach, including OpenAI's work on training activation classifiers. This will be an additional investigation rather than a requirement for completing the core project.
I will consider the project successful if I can:
Establish a working baseline linear probe using an open-weight language model.
Compare its performance across selected transformer layers and activation configurations.
Identify at least two concrete categories of probe failure.
Measure how changes in wording, response style or context affect its predictions.
Investigate the internal features associated with selected failure cases, where feasible.
Test at least one mitigation strategy and compare it against the baseline.
Publish the code, evaluation scripts, results and a written research report.
I will use metrics such as AUROC, precision, recall and false-positive rates to evaluate the probe. I also want to test it on held-out examples that differ from the training data, rather than relying only on a random train-test split.
An important part of this project is being careful about what the results actually mean. A false statement is not necessarily an intentional deception, and a probe's prediction does not prove that a model intended to deceive.
I therefore want to document how the examples are labelled and what conclusions the experiments can reasonably support.
For me, success does not simply mean achieving higher classification accuracy. Understanding why a probe makes mistakes and how well it generalises is more important.
Even if the mitigation experiment does not work, the findings could still be useful if they show where current approaches are unreliable.
I am requesting $1,000 to cover the computational resources and research expenses needed to carry out this project over approximately 6–8 weeks.
The proposed budget is:
Cloud GPU compute – $550: Running open-weight models, extracting transformer activations, training probes, comparing layers and testing interventions.
OpenRouter API credits – $200: Generating controlled experimental examples and accessing smaller language models as auxiliary reviewers where useful. These models will not be treated as ground truth.
Storage and experiment backups – $50: Storing datasets, activation outputs, experiment results and other research files.
Additional compute and contingency – $200: Covering additional evaluation runs or unexpected computational requirements during the experiments.
I plan to use open-weight models and free research resources wherever possible. I will also monitor compute and API usage so that the budget is spent on experiments directly related to this project.
The funding will be used for research expenses rather than general-purpose software development.
If I receive less than the full amount, I will reduce the scope accordingly. With $500, I will prioritise the baseline and controlled evaluation of probe failures. With more funding, I can run additional experiments and investigate a mitigation strategy more thoroughly.
I am working on this project independently as part of the TARA Fellowship, where I have been assigned a teacher who provides research guidance. I will be responsible for the implementation, experiments, analysis and final research report.
I have approximately two years of software engineering experience. Most of my previous work has involved building software applications and technical systems. This experience has helped me develop practical skills in Python, debugging, working with APIs and building technical workflows.
Over the past several months, I have been focusing more seriously on technical AI safety research.
My preparation includes completing the ARENA curriculum end-to-end, with hands-on coding and implementation exercises in transformer interpretability and related technical topics.
I have also completed training through Lens Academy in AI safety, AI control and computer verification.
I have also studied If Anyone Builds It, Everyone Dies by Eliezer Yudkowsky and Nate Soares.
Alongside this learning, I have worked on practical AI safety projects.
1. Linguistic Reasoning Drift in Low-Resource Languages
In this project, I investigated whether language models exhibit differences in reasoning behaviour across English and Hindi. I designed an evaluation metric and implemented an evaluation pipeline to compare model outputs. I also worked on analysing the results and presenting them through a dashboard.
The project helped me gain practical experience in designing evaluations and working with model outputs.
GitHub: https://github.com/priyankashah3107/local-lang-vulnerability
2. Secret Loyalty and Chain-of-Thought Monitoring
I have also worked on an exploratory project related to secret loyalty and chain-of-thought monitoring. It reflects my interest in understanding how model behaviour and internal reasoning can be evaluated for potential safety concerns.
GitHub: https://github.com/priyankashah3107/secret-loyalty-cot
My AI safety portfolio contains more information about my technical interests and projects:
Portfolio: https://priyanka-shah.vercel.app/ai-safety
LinkedIn: https://www.linkedin.com/in/priyankashah317
I am still early in my research career, so I do not want to overstate my experience. My background is primarily in software engineering, and I am now applying those skills to more focused technical AI safety questions. I intend to make the implementation and findings from this project publicly available so that other researchers can inspect and build on the work.
I see a few possible reasons why the project might not produce the results I expect.
1. Computational limitations
Extracting activations and running experiments across transformer layers can require substantial GPU resources. If compute becomes a bottleneck, I may not be able to run as many experiments as planned.
To manage this, I will start with one open-weight model and prioritise the experiments that are most useful for answering the research question.
2. Difficulty interpreting internal features
Understanding which neurons or features contribute to a probe's predictions is difficult. Available neuron dictionaries may also be incompatible with the model I use.
If this becomes a problem, I will focus on the layer-wise results, probe behaviour and controlled experiments rather than making unsupported claims about individual neurons.
3. Ambiguous deception labels
One challenge is deciding what counts as deception. A false answer does not necessarily mean that a model intentionally deceived someone.
Poorly labelled examples could make the evaluation misleading. I will therefore use clearly defined experimental scenarios and document the labelling criteria. If I use other language models to review examples, I will treat their judgements as imperfect assistance rather than ground truth.
4. Mitigation experiments may not work
An intervention or a change in training data may not improve the probe's ability to generalise. It could even make performance worse.
I will report these results honestly and compare them with the baseline rather than presenting every experiment as an improvement.
Overall, the project might produce fewer experiments or less conclusive results than I initially expect. However, I believe that a careful analysis of where deception probes fail can still be useful, even if I do not find a successful mitigation.
I will make the code, methodology and findings available publicly so that other researchers can examine the results and build on them.
I have not raised external funding for this research project in the past 12 months.
There are no bids on this project.
Comments
No comments yet. Sign in to create one!