You're pledging to donate if the project hits its minimum goal and gets approved. If not, your funds will be returned.
$12,000 left to reach the funding goal.
We’re two Cambridge undergraduates working with mentorship from Geodesic Research on a project to improve synthetic document fine-tuning (SDF) reliability. We’ll write up our results and share them as a LessWrong post.
Current SDF methods tend to degrade model capabilities, making model organisms trained from them a less effective proxy for models that actually see deployment (see “Your Model Organisms might be fried” by Arcadia Impact)
We aim to improve the reliability of synthetic document fine-tuning through better training recipes (especially for smaller models, as the cost associated with using larger models is one of the bottlenecks to model organism research), as well as understanding what environments it can be effectively applied in. We worry that SDF is being indiscriminately applied for experiments where it’s not the right tool.
Additionally, less-fried models should lead to more applicable results for model organism experiments, at both small and large sizes.
This will be accomplished by running sweeps on SDF variables such as learning rate, token count, LoRA rank, model size and fitting how performance scales with each parameter.
The goal would be to develop heuristics for optimal training recipes given model sizes and desired outcomes (e.g. higher coherency/saliency). We hope to push the Pareto frontier on cost / capability for SDF experiments with these.
For example, one of our hypotheses is that a higher LoRA rank should be adopted for more surprising SDF distributions (high perplexity) - as intuitively it seems we’d require higher dimensional movement in the weight space.
An auxiliary benefit of this is that it will help us upskill a lot: feedback from a mentor while doing an actual project will help us develop better research taste, and the legible output will help a lot in further research fellowship application, aside from contributing to AI safety. It is hard in applications, to signal value alignment and research taste, and a successful version of this project would provide both.
If we discover that there are significantly better training recipes over the current literature e.g. Replicating or improving on existing paper results, at the same cost, but with significantly less-fried models. Additionally, reproducing certain results from the papers we will draw our SDF recipes from on smaller models would be a good trial case for their viability in research.
Funding will be used for two main components:
Generating a corpora of SDF documents using the Anthropic batch API. Each corpus will be ~40 million tokens and aimed at instilling a single trait. We intend to experiment with multiple traits and generating models (e.g. Haiku, Sonnet, Opus) to find out how to make high-quality SDF data.
Next, we train our models on the generated data using next-token-prediction (NTP) on GPU neoclouds (e.g. Modal, Runpod), and perform evaluations on how deeply traits are instilled while maintaining model capability. We estimate that renting GPU-hours for training will be the majority of our expenses.
Rough budget estimate
Traits: 10 (drawn from AuditBench and the original Slocum paper)
Parameters to vary for each trait: 7 (e.g. Learning rate, LorA rank, model size)
Number of parameter values in each sweep: 5
Total training runs: 350
Cost per training run: 14 USD. 1-12 H100 GPU-hours per training run (for sizes 1.7B -> 32B). H100 spot ~$4.00 USD/hr on Modal. Weighted average of 3.5 GPU hrs per run = 3.5 x 4 = $14
Training: 4,900 USD. ~3.5 * 350 = 1200 GPU-hours total
Document generation, API batch generation: 1,200 USD. 400 mil tok Haiku ($175), 40 mil tok Sonnet ($350), 40 mil tok Opus ($700)
Evaluation: 1,000 USD. 10 GPU minutes per test (~0.80 USD), 3 tests per training run, 350 runs, 1050 tests
Contingency for mistakes (e.g. interrupted training runs): 1,800 USD. ~25%
Total: 8,900 USD
We’re a team of two Cambridge second-year natural sciences undergraduates (Liu Hao and Justas Miliauskas), with mentorship from Nathaniel Mitrani (MTS @ Geodesic).
Justas Miliauskas:
Loosely engaged with AI safety for around four years (many hours of LessWrong, Bostrom’s ‘Superintelligence’, etc.) as well as the general EA scene (most recently with EA Cambridge).
Recent events and speed of AI development have driven me to go full steam on contributing to alignment, since my previously planned routes to impact fall beyond the ASI horizon.
Won International Chemistry Olympiad Gold 2024 (23rd in the world) and two International Biology Olympiad silvers
Have done BlueDot’s AI technical safety course, finished all ARENA parts up to reinforcement learning
Over the summer, worked as a machine learning engineer at a drug discovery company in Switzerland
Experience with multiple GPU training of deep learning models (at the aforementioned internship)
Quant research internship focused on building scalable data pipelines for large-scale transaction data
Self-directed mech interp project with a $500 BlueDot Rapid Grant
Liu Hao:
I’ve been engaged with EA for around 4 years. My original intention was to go into quant trading and earn to give. However, AI capabilities in verifiable fields (maths, coding etc.) have accelerated at such an alarming rate from 2024 onwards that I think AI safety and x-risk mitigation might be the highest impact thing I can work on. I’m actively considering a pivot into AI safety and this project would provide research valuable experience.
Ranked in top 2% of my Cambridge cohort and topped my college in first-year exams.
Studying on a highly selective full scholarship and silver medallist at International Astrophysics Olympiad
Received quant trading internship offers from IMC and Optiver
Research experience: two ~3 month internships. 1. Extragalactic astrophysics at Cambridge, 2. HPC compute cluster for remote sensing at NUS
Completed BlueDot technical AI safety course, self-studying ARENA. Key exercises done so far: Implementing attention and transformer blocks, finding induction heads in GPT-2.
Causes:
As with all model organism research, it’s possible that SDF doesn’t scale well to larger models and we see emergent capabilities that we can’t smoothly fit, making our recipe heuristics inapplicable. On the other hand, SDF might not really work well for smaller models due to their weaker ability to coherently integrate new facts, which would make them non-viable for model organism research.
For example, capable models may be better ‘out-of-context’ learners and have a more sophisticated latent space that can model implausible facts/recognize their saliency (already shown in literature) or distinguish their syntheticness (also shown for some less stringent setups).
Our experiment list could miss out some variables which later turn out to be key to improving SDF reliability.
It’s also possible that current SDF setups used in papers are already relatively well tuned and we don’t see further significant improvement (although we think this is pretty unlikely given the Arcadia diagnostic paper)
Outcomes:
We still publish any negative or partial results in hopes that they are useful. We report and return unspent funds.
We still derive mentorship and AI safety research experience from the project, which we hope will be helpful in pivoting into AI safety fellowships or future full-time roles.
We’ll apply for the Bluedot Rapid Impact grant, but haven’t raised any other sources of funding yet.