Project summary

Five different open source language models from five different labs, all with different methods of construction. One of them is not a transformer! I wanted to know if you could get them to work together without the weights being fused into one model, which is how people typically combine the models of AI. But you can only do that when the models are close together.

I tested a long list of tests to demonstrate the internal organization of the same text across the five models. These models have measurable structure (even the non-transformer one) and cope with a lot of stress testing; nobody built them to agree with each other. I used this discovery to create a little prototype tool which transforms the inner model of any of the five models into a common space. Now I'm wondering if that common space is helpful, particularly for a system that might be able to determine which model is most suitable for a particular question rather than sticking with any one.

What are this project's goals? How will you achieve them?

The first was to determine whether or not it was possible to relate to models that do not share anything in common, without training them, or even imposing a common form on them. That part is done. I used the same set of 200 sentences in all 5 models, saw if they were reacting as much as a random sample would, and would try to break the result before I would believe it. It held up.

The challenge now is to determine whether this common structure is of any actual use, namely a small router that chooses from the five models for any particular question. The models I've tried so far don't exhibit a clear specialization. One of them is simply better in all that I have tried. Testing harder and varied tasks to determine if that changes and then training a real router on top of this shared space that I already build.

How will this funding be used?

So far, everything is installed on my own laptop that has no GPU which is running free. That's because what they have done so far is to compare the interiors of the models, which are small (1-1.7 billion parameters), and there is no training involved. What I want are real compute in three phases, corresponding to what it actually does.

First, it must be determined if these five models possess different merits where they would be beneficial to create a router around. The 5 of them: Mamba runs without GPU acceleration on my laptop and it takes about 90 seconds per question; the other 4 take about 1000 times longer. That would be a lot faster with a real GPU. This phase will require between 20 and 40 GPU hours at typical rental rates of $0.50 to $1.50 per hour to create a much larger variety of test questions for all 5 models as well as train a small router if an actual crossover case comes along. That's $300 to $500.

If it has some real result in that phase, one which actually outperforms the other models at some task, I wouldn't trust it until it was tested properly. This implies bigger sets of benchmarks, more elaborate evaluation, testing the router design for robustness in the face of ablation, and testing if the common representational space that I've constructed would extend to models beyond the original 5. That's roughly 150 to 250 GPU hours over 2 to 3 months, or $1,500 to $2,500.

The last part is actually nicely producing the result upon, since I have a publishing history in related work, and executing whatever final assessment passes that is required. That's another $300 to $500.

Total across all three phases is $2,000 to $3,500. Each part is tied to a specific number of GPU hours at a stated rate for a stated purpose, not a lump sum. If the first phase doesn't find a crossover case, phases two and three don't happen and the honest outcome is a negative result, which is still worth having and costs far less than the full amount.

Who is on your team? What's your track record on similar projects?

It's just me on this one. I'm the founder and sole engineer behind Ìyàwó, a clinical decision support system now live across more than 50 Primary Health Centres in Nigeria, supporting over 131,000 patient helped, and I built and own the full stack, from the LLM pipeline to the offline-first product itself. I've also served as AI and technical lead on international health AI programmes, including a €6.5 million EU Horizon project across 12 institutions and a $3 million Wellcome Trust and Google Research programme adapting mental health screening tools for Hausa, Yoruba, and Pidgin. I was principal investigator on a $20,000 research project for the World bank and Government of Japan through GDN, which is the closest match to this project in terms of being the direct grant holder rather than a technical lead inside someone else's funding.

This specific project isn't in the same domain as my clinical work, it's a personal research project on whether different AI model architectures share internal structure, and whether that can be used to combine them without merging their weights. But the skills carrying over are real: I've published on adversarial robustness and cross-lingual failure modes in clinical AI, I currently serve as Country Lead for AI Safety times Nigeria, and I mentor emerging researchers in AI safety and governance through Black in AI Safety and Ethics. The methodology in this project, testing a hypothesis carefully, trying to break my own results before trusting them, replicating findings on independent data before believing them, is the same discipline I've applied in my published clinical AI work.

What are the most likely causes and outcomes if this project fails?

The most likely failure is a boring one. I don't find a task where the five models actually have different strengths, so there's nothing for a router to usefully choose between. If that happens, the honest outcome is a negative result: evidence that this particular set of five models doesn't benefit from being combined. Still worth writing up, since most people assume combining models always helps.

The other failure mode is the shared space turning out too narrow to generalize. It works on the kinds of text I tested but not on real inputs. I've already seen a version of this. It struggles with code and math notation. So this risk is real and partly already visible, not just hypothetical.

How much money have you raised in the last 12 months, and from where?

None