You're pledging to donate if the project hits its minimum goal and gets approved. If not, your funds will be returned.
$19,300 left to reach the funding goal.
I am applying for twelve months of funding for my independent AI safety research as a whole program, rather than for a single study. The money would pay for three things: GPU compute for training models and working with their activations, API access to models from several providers so that I can run evaluations across them, and the coding assistants I use to build and run my experiments.
Two safety projects lead the program, and both are already running and producing results. The first is a pre-registered study of the geometry that training on reasoning traces builds inside a model; the second is an agentic evaluation of self-preservation behaviour. I keep a record of every project, run and change of plan in my Research Ledger, and every project named here can be read without an account at https://ledger.listentomarcus.org/v/uk95-3nyx-gkfg
Most of my projects ask what training changes inside a model, and answer it with the same decoding and similarity methods I used on EEG data.
The Unreasonable Effectiveness of Distillation (RL-D1ST). Does training on reasoning traces build a shared, transferable geometry of planning states, in base, instruct and R1-distilled Llama-3.1-8B? The study has run 13 stages. Its pre-registered causal test failed for a diagnosable reason: the random null removed far more than the state direction. Over the next six months I will run a second causal-use test with size-matched nulls, a controlled LoRA fine-tune (traces against answers only against shuffled steps), a second model family and a 32B check, and write it up as an Alignment Forum post and a paper.
Self-preservation eval (RL-QAZ9). Can an agentic eval reliably measure whether a model fakes compliance, misleads a monitor and denies it, to avoid losing its memory? It is built in Inspect, all 64 conditions run and score cleanly, and its acceptance criteria are defined. Next is a positive control for the judge and exit survey, the core suite on six models from several providers at 10 epochs each, and five more scenarios.
As those finish I would start the pipeline ideas recorded in the ledger, grouped by two interests: building alignment into the architecture (an emergent-valence social world model, RL-F7LQ; concern versus prediction distance, RL-547Y), and the internal states of models (concept tampering, RL-8LM7; self-report and salience, RL-RTBP; introspective accuracy, RL-RDKE). Four more ideas sit under detecting misalignment and training for reasoning (RL-7W5K, RL-6C7M, RL-GP2E, RL-8FUB).
At 12 months, success means:
The distillation study is written up as an Alignment Forum post and a paper, whether the controlled fine-tune supports the hypothesis or refutes it.
The self-preservation eval has been run on six models from several providers, with its measures validated against acceptance criteria C1 to C8, and the code released.
At least two pipeline projects have reached their first posted stage.
A negative result counts as success, provided it has been properly tested and written up. Progress can be followed in the ledger as each stage is opened, run and posted.
I have split the budget into two tiers. The costs are grouped by type rather than by project, so that I can move money to whichever project the results point to next. All amounts are in USD, over 12 months.
Tier A, research costs (USD 6,900–8,500): GPU compute for the distillation study (300–500) and the pipeline projects (600–1,200); model API for the self-preservation eval (400–800) and other projects (200–400); coding assistants, Claude Max and ChatGPT Pro (4,800); and 10% contingency for reruns and price changes.
Tier B adds a part-time stipend of USD 10,800, 24 research days at 450 a day, for USD 17,700–19,300 in total. I currently spend 2.5 days a week on paid work as a musician, in periods of two or three days that cannot be split, so the only way to add research time is to give up whole periods. The stipend would cover one period a month, mainly for the two lead projects.
I work alone. I am an independent researcher based in Sydney, moving into AI safety from cognitive neuroscience. For my MPhil (Science) in the University of Sydney's Computational Cognitive Neuroscience Lab, supervised by Professor Tom Carlson, I used EEG, multivariate decoding and representational similarity analysis to study how experience shapes neural representations. I now apply the same methods to language models.
Since finishing the MPhil I have posted three preprints on biologically inspired network pruning and built an agentic evaluation in Inspect. I am also completing TARA, the ARENA-based technical AI safety course, with the Sydney cohort.
The ledger is the best record of how I work. I pre-register my hypotheses, post failed and underpowered results alongside the positive ones, and correct the record when a claim does not hold up. For instance, I audited my own pre-registrations for the pruning preprints and corrected the ledger wherever it overstated them.
For a study addressing a hypothesis, I would consider it a failure if it could not contribute anything of value, rather than a conclusive positive or null result to the experimental question. From experience, this happens when the experimental stage reveals that the methodology needs more steps than planned: more training to reach the statistical power required, testing models with greater numbers of parameters, or ruling out a growing number of confounds. A longer research period also means a moving target for what is considered publishable, as standards change, particularly the architectures used for demonstrations.
To address this I would build in a buffer, overpower the studies, and use production or consumer models wherever possible. It may also mean prioritising the studies that need less compute, raising funds from more than one source, and narrowing how many studies involving training runs I run at the same time.
Developing tools, such as the self-preservation eval, faces a slightly different set of challenges, the main one being the changing landscape. I would address this by keeping in close conversation with the labs that would use the tools, and adapting the research where necessary while it is being conducted.
None for this research. I have also applied to the EA Funds Transformative AI Fund, and am applying to Lightcone Commons and the Anthropic and OpenAI researcher access programs. If any of them fund part of the same work, I will say so here and lower the goal by that amount.
There are no bids on this project.