You're pledging to donate if the project hits its minimum goal and gets approved. If not, your funds will be returned.
Over the past few months I have been testing whether the hidden recurrent state of a frozen RWKV7 language model can be used as the memory and registers of a simple computer.
The base model weights are not changed. I use small trained adapters and direct state injection to put information into the model and get it to operate on that information internally.
So far, on the frozen 7.2b RWKV7 model, I have shown:
• Exact arithmetic including addition, subtraction, comparison, multiplication and division.
• A working stored program computer with instruction memory, data memory, a program counter, branching, multiple instructions and its own halt condition. It completed the held out programs without errors.
• An expanded 13 instruction system with multiple addressing modes.
• A closed execution loop where the model reads its own calculated state, uses that as the input to the next step, and decides itself when execution is finished.
• Planning using memory the model addresses itself. Controls show the result breaks when the memory or reader is removed.
• A rule learning system which can propose an answer, test it, backtrack and settle on the correct rule across a space of 126 possible rules.
• Most recently, a small adapter which takes normal English instructions and compiles them into machine code for the internal computer. It passed the preset tests on unseen combinations, including branching and loops.
The original computer was developed on a .4b model and then moved to the 7.2b model without changing the code. It still worked correctly.
I have tried to be fairly strict with the testing. Before an experiment runs I define what I am testing and what counts as a pass or fail. I also use held out examples and negative controls to make sure the model is not getting the right answer through some shortcut I have missed.
Failures are kept as part of the project rather than quietly retried. The project graph now has more than 700 records, including 129 failed experiments. I have also rejected and repeated some earlier results when later review found problems with the test.
The part I think could be most useful is that I know what computation I have put inside the model and I can check the result against an independent reference.
Normally interpretability research is trying to work out what an unknown process inside a model is doing. This may give researchers the opposite problem: a large pretrained model where part of the internal process is known and can be checked.
If that holds up under more testing and outside review, it could make a useful test case for interpretability work.
Further down the track it may also be useful for systems where some important decisions need to be checked as actual computation rather than relying only on generated text. I have not proven that. It is one possible direction if the current results continue to hold.
The public project summary and methodology are here:
https://github.com/fleeb83/rwkv7-state-machine
The next job is to deal with the two biggest gaps. The system needs to read information from a real external environment, and its internal result needs to cause a real external action.
Once those work, I want to put the pieces together on one complete task and then see whether the results still hold beyond the current 7.2b model.
The main goal is simple. I want to find out whether the parts I have already demonstrated separately can actually work together as one complete system on a real task.
Phase 1 is the two missing interfaces. The model needs to read information from an external environment and turn its internal result into an action. I will test both on the existing 7.2b model using the same controls and preset pass or fail tests I have used for the rest of the project.
Phase 2 is to chain the whole thing together on one real task. The model will read the environment, run the compiled program, make a decision, take an action and decide when it is finished.
I would rather do one task properly than spread the work across a large benchmark and learn less from it.
Phase 3 is scaling to the 13b RWKV7 model.
The move from .4b to 7.2b worked without changing the system. I want to see whether that continues at 13b, whether it improves, or whether something breaks.
The results and methods are already being documented publicly here:
https://github.com/fleeb83/rwkv7-state-machine
The funded work will also produce a more detailed technical write up, repeatable test material, scoring code and model checkpoints where they can be released.
Any funding will go towards compute, storage and related infrastructure.
I am not asking for funding for my own time. I will continue to volunteer my time.
At $2000 I should be able to complete the main 7.2b interface work and make the first attempt at running the complete chained system.
At the full $7500 I would also be able to scale the work to the 13b model and repeat the important tests at that size.
The project is being run by me, Russell Thomas, an independent researcher in Australia.
My track record is basically this project.
Over the past few months I have funded the work myself using local GPUs and rented compute from Hugging Face and RunPod.
The results are outlined above. I use preregistered tests, negative controls, held out data and an append only graph database that I built specifically to keep track of the project.
I have not published a peer reviewed paper or received external research funding before.
A public summary of the work is here:
https://github.com/fleeb83/rwkv7-state-machine
The biggest risk is connecting the model to the outside world.
So far the strongest results happen inside the model's internal state. Reading from the outside world and acting on it reliably is still being worked on.
One approach has already failed. It generated a description of an action instead of producing something the system could actually execute.
I am testing a different approach now which reads the model's own state directly. The execution path is working and I have measured a real speed improvement, but the full correctness testing is not finished.
If it fails, I will record the failure the same way I record a pass.
Scale is another risk.
What worked from .4b to 7.2b may not keep working as the model gets larger. If it breaks at 13b, I want to know where it breaks and why.
If the overall project fails, there should still be a useful record of what worked, what did not, and where the limits appear to be.
I will keep changing direction while the experiments are still telling me something useful.
$0