You're pledging to donate if the project hits its minimum goal and gets approved. If not, your funds will be returned.
$60,000,000 left to reach the funding goal.
I intend to start a non-profit research organization focused on making long-horizon continual learning safe by design.
Today the dominant alignment approach is to work with static models. Future AI systems however will need to be continually adapting their weights as they gain experience from users over time and across a long timescale. This leads to a significant safety issue in that ongoing optimisation could simply overwrite the alignment training, cause the model to forget the safety behaviour it was trained on, or slowly drift in its objectives.
My approach is to make the learning process itself aligned. Leveraging my experience in meta-reinforcement learning, hypernetworks and test time adaptation I want to develop learned update mechanisms that allow models to learn new capabilities while maintaining their alignment. The opportunity is to get ahead of the unsafe stuff that we know is coming. It's safety by design.
My longer term ambition is to make safe continual learning practical enough that it can become the default approach to post-deployment model adaptation.
Continual learning will be a major capability frontier. The models we have at present are very much reliant on context windows and external memory to store their experience. In the longer term it's very likely that agents will need to be regularly doing some form of consolidation of that experience into their weights so that they can actually learn skills and knowledge over the course of months or years of operation.
The issue is that updating the weights can also undo some of the previous training. This is particularly troubling in the case of alignment where a model could be gradually losing the safety relevant behaviour it learned during training while still being able to improve on the tasks it sees.
I'd like to explore whether we can learn the adaptation mechanism itself (potentially via hypernetworks which are trained to produce safe parameter updates). Such a mechanism could be meta-trained across many tasks and training trajectories with objectives that encourage useful adaptation and discourage any changes to behaviour that is important for alignment.
Initially, I would focus on:
Measuring alignment drift. Build benchmarks in which the models go through a sequence of weight updates over an extended period and we measure both the improvements in capability and the changes in safety-relevant behaviour.
Learning safer update mechanisms. Look at hypernetworks, meta-learned optimisers and constrained parameter updates that maintain the alignment properties during continual learning. Compare those against standard gradient based fine-tuning.
Testing generalisation and robustness. Check that these mechanisms still work under distribution shift, adversarial training data and long sequences of individually benign updates.
With enough funding I'd also like to develop software that automatically trains model specific safe adaptation mechanisms. For instance a pipeline that takes an existing LLM and trains a hypernetwork to generate constrained updates for that model. That could then be applied to a wide range of deployed systems.
The most obvious outcome would be to show that a model can learn new useful knowledge or skills while at the same time maintaining the safety properties that are known to degrade under normal fine-tuning.
I'd assess this in two ways. One would be to see if the model is actually learning (task performance, retention of earlier capabilities, adaptation speed etc). The other would be to assess alignment preservation (refusal behaviour, resistance to harmful requests, adherence to instruction hierarchy, robustness to adversarial data).
A good method should do at least as well as standard fine-tuning in terms of adaptation and should show considerably less degradation in the safety metrics over the long update sequence. We'd also want to know whether the results are generalisable across model families and across tasks.
A key intermediate result would be a convincing demonstration that alignment degradation is in fact a measurable problem as a result of the catastrophic forgetting that plagues current continual learning methods. Even if the hypernetwork approach itself doesn't work, a properly designed benchmark could go some way towards establishing how the problem manifests.
In the longer term success would be in creating a reusable training pipeline that can produce alignment preserving adaptation mechanisms for existing models and having sufficient evidence that other researchers and developers can safely and confidently use them.
I'm looking for initial funding for the project. That funding would allow me to move from my current research position to working on safe continual learning full time, set up the requisite organisational structure and start recruiting a research team.
The scope of the project would be dependent on the amount of funding available.
Approx $600k total. I could do it full time for a first year, set up fiscal sponsorship and admin infrastructure and run the initial experiments. The priority would be establishing benchmarks for alignment drift, setting up solid conventional fine tuning baselines and looking at the initial hypernetwork approaches. The aim would be to get the first results out and release the evaluation code.
At $1M to $1.5M I could hire 1-2 more researchers or research engineers, scale up the experiments to larger models and try out multiple approaches to learned adaptation. It would also allow for a lot more compute and much longer training trajectories so we could see whether alignment preservation actually holds over a realistic sequence of updates.
At $2M+ I'd be looking to establish a small research lab that could go after the full agenda (i.e. more comprehensive evaluations, adversarial stress testing, method development etc). We'd be looking at building reusable software and training pipelines that would automatically generate model specific safe adaptation mechanisms. So for instance given a pretrained LLM you could train a hypernetwork that was specifically tuned to that model's architecture and alignment needs. The aim would be to get the resulting methods to be deployable in practice.
The biggest costs would be for competitive researchers, compute and org overhead. There would also be provision for external safety assessments and for working in partnership with researchers on other approaches to alignment.
I'd be the one to lead the project and recruit further researchers as funds become available.
I'm a Senior Applied Scientist at Oracle and completed a DPhil in Computer Science at Oxford (working with Shimon Whiteson on RL, meta-learning and adaptation). The DPhil was funded via the Oxford-Google DeepMind Doctoral Scholarship.
I'm most proud of the work I've done in advancing meta-reinforcement learning both in terms of the technical work and as a research area. I developed hypernetwork based methods which substantially outperformed the previous meta-RL baselines. I co-wrote a highly cited survey and tutorial which I think did a lot to consolidate and bring together the field. I've also talked about this work in a TalkRL interview, on an ICML panel and in a conference tutorial.
More recently at Oracle I've been working on LLM agents, reinforcement learning for coding agents, post-training and test time adaptation. I've recently developed Hill Sampling, a test time search method for discovery problems.
My experience with hypernetworks is very much relevant to this proposal. I've spent a significant portion of my research career looking at how models can learn to modify their own parameters and I think that provides a very promising technical basis for learning update rules that maintain alignment.
More on my research: jakebeck.com | Google Scholar
I'd be eager to work with a senior researcher or professor with complementary expertise in AI safety and experience in building and funding research programs. Also of interest would be to recruit researchers in the areas of alignment, continual learning and evaluation.
I can provide references from senior researchers with whom I have worked, including former research supervisors who can speak to my research ability and independence.
The biggest technical risk is that maintaining alignment as you do continual learning is going to be vastly more difficult than maintaining the performance on the tasks themselves. Alignment is a very broad set of behaviours and it's going to be very hard to specify that as a training objective. The learned update rule could very well do well on the safety evaluations that were used during meta-training and at the same time allow all sorts of other important behaviors to degrade.
A wider worry is that successful continual learning research could accelerate AI capabilities. That's why I'd assess whether any proposed methods actually provide any meaningful safety benefits before we release general purpose adaptation tools.
If we don't end up with a successful adaptation method then I'd still expect the benchmark work and the empirical comparisons to be useful. In particular I'd expect to be able to say which forms of continual learning lead to alignment degradation and which of the existing mitigations actually work. That would be useful in terms of how we go about developing these systems.
This is a new research initiative and I haven't yet secured dedicated funding for it. I'm at the stage of looking around at potential funders in the space of technical AI safety research (e.g. Coefficient Giving's Project Tailwind).