You're pledging to donate if the project hits its minimum goal and gets approved. If not, your funds will be returned.
For months I've been testing the whether the recurrent state of frozen RWKV models can be used as a simple computer.
I used the base model from hugging face and added small adapters and state injection to put information into that model and to get the model to operate on that information.
On the frozen 7.2b RWKV7 model I have shown:
• Arithmetic including addition, subtraction, comparison, multiplication and division.
• A stored program computer with instruction memory, data memory, a program counter, branching, multiple instructions and its own halt condition. Held out programs were completed without errors.
• A 13 instruction system with multiple addressing modes.
• A closed execution loop where the model reads its own calculated state, uses that as the input to the next step, and decides itself when execution is finished.
• Planning using memory which the model addresses itself. Controls show that the result breaks when the memory or reader is removed.
• A rule learning system which proposes an answer, test the answer, backtrack and settle on the correct rule across a space of 126 possible rules.
• An adapter which takes English instructions and compiles them into machine code for the internal computer. It passed the preset tests on unseen combinations, including branching and loops.
The computer was developed on a .4b model and then moved to the 7.2b model without changing any code. It still worked correctly.
I have intentionally been strict with the testing. Before an experiment runs I define what I am testing and what counts as a pass or fail. I also use held out examples and negative controls to ensure the model is not getting the right answer through some shortcut I might have missed.
Failures are kept as part of the project, not quietly retried. The project graph now has more than 700 records, including 129 failed experiments. I have also rejected and repeated some earlier results when later review found problems with the test.
Normally interpretability research is trying to work out what is happening inside models. This may give researchers an advantage where large pretrained models can have their internal workings checked.
If that holds up under more testing and external review, it could make a useful test case for interpretability work.
Further down the track it may also be useful for systems where some important decisions need to be checked using actual computation rather than relying only on generated text. I have not proven that. It is one possible direction if the current results continue to stand.
The public project summary and methodology are here:
https://github.com/fleeb83/rwkv7-state-machine
The next job is to deal with the two biggest gaps. The system needs to read information from a real external environment, and its internal result needs to cause a real external action.
Once those work, I want to put the pieces together on one complete task and then see whether the results still hold beyond the current 7.2b model.
The main goal is simple. I want to find out whether the parts I have already demonstrated separately can actually work together as one complete looped system on a real world task.
Phase 1 is the two missing interfaces. The model needs to read information from an external environment and turn its internal result into an action. I will test both on the existing 7.2b model using the same controls and preset pass or fail tests I have used for the rest of the project.
Phase 2 is to chain the whole thing together on one real task. The model will read the environment, run the compiled program, make a decision, take an action and decide when it is finished.
I would rather do one task properly than spread the work across a large benchmark and learn less from it.
Phase 3 is scaling to the 13b RWKV7 model.
The move from .4b to 7.2b worked without changing the system. I want to see whether that continues at 13b or whether something breaks.
The results and methods are already being documented publicly here:
https://github.com/fleeb83/rwkv7-state-machine
The funded work will also produce a technical write up, repeatable test material, scoring code and model checkpoints where they can be released.
Any funding will go towards compute and storage.
I am not asking for funding for my own time. I will continue to volunteer my time.
At $2000 I should be able to complete the main 7.2b interface work and make the first attempt at running the complete looped system.
At the full $7500 I would also be able to scale the work to the 13b model and repeat the important tests at that size.
The project is being run by me, Russell Thomas, an independent researcher in Australia.
My track record is this project itself.
Over the past few months I have funded the work myself using local GPUs and rented compute from Hugging Face and RunPod.
The results are outlined on my github repo. I use preregistered tests, negative controls, held out data and an append only graph database that I built specifically to keep track of the project.
I have not published a peer reviewed paper or received external research funding before.
A public summary is here:
https://github.com/fleeb83/rwkv7-state-machine
The biggest risk is connecting the model to the outside world.
So far the strongest results happen inside the model's internal state. Reading from the outside world and acting on it reliably is still being worked on.
One approach has already failed. It generated a description of an action instead of producing something the system could actually execute.
I am testing a different approach now which reads the model's own state directly. The execution path is working and I have measured a real speed improvement, but the testing is not finished yet.
Failed experiments are recorded using the same method as a pass.
Scale is another risk - What worked from .4b to 7.2b may not work on larger variants. If it breaks on larger models, I's want to know where it breaks and why.
If the overall project fails, there should still be a useful record of what worked, what did not, and where the limits appear to be.
I will keep adapting while the experiments are still producing valuable information.
$0
There are no bids on this project.