**Disclaimer: This application is AI assisted. I have personally written most, but it has been edited and polished by Claude, using Sonnet""

A live open question in interpretability is whether language models represent

categorical/relational knowledge as reusable, compositional structure (which would be

auditable and would generalize predictably) or as memorized, item-specific lookup

(which wouldn't). Prior work (Engels et al., 2024) showed some cyclic concepts trace

literal circles in Mistral-7B and Llama-3-8B activations, confirmed causally. My project

asked a narrower, harder question: is that a special property of cyclic concepts, or

of small closed enumerable sets generally - and is it really 2D, or does the 2D

picture just look clean while hiding a higher-dimensional mechanism?

Track record - what's already done:

This sits inside a longer-running independent research program (several years,

predating GitHub publication - the public repos only went up in July 2026, when I

moved the work from local files onto git, not when the underlying research started).

The result below is the most directly AI-relevant and most rigorously

causally-tested piece of it.

Repo (public, MIT-licensed, code+data+figures+paper all included):

[https://github.com/ZaPpi3/emergent-geometry-llm-representations](https://github.com/ZaPpi3/emergent-geometry-llm-representations)

Run entirely on hardware-constrained local models (Qwen2.5-0.5B, Mistral-7B-v0.1, one

consumer GPU) - deliberately not requiring lab-scale compute to get a real result.

Six reported findings, arrived at through a genuine correction cycle rather than a

straight line to a clean answer (methodology detail worth having in the application,

since it's the actual evidence of research judgment, not just the headline numbers):

1. A first, naive causal-patching attempt (patching only the final token position) gave

a null result - diagnosed as a specific methodological flaw (the model could

route around that position), not a failure of the underlying hypothesis.

2. Fixing it (patching the subject-token position instead, standard causal-tracing

practice) gave a clean positive result: swapping in a donor day/month's

representation flips the model's answer to match the donor with 83-100% accuracy

through early-to-mid layers, against an exactly-0%-at-every-layer random-direction

control.

3. Testing the stronger claim - that this is literally a 2D circle, as the naive PCA

plot suggests - failed: synthetic rotation within the discovered 2D plane got

the correct answer 0/60 times, statistically identical to a deliberately wrong

rotation angle (also 0/60).

4. Refitting in a full-rank subspace (n-1 principal components) instead of forcing a 2D

answer recovered the full causal effect: a single best-fit rotation generator,

applied with zero real donor data, matches or slightly exceeds the real donor

patch's accuracy (100% vs. 100% for days, 85% vs. 82% for months), while a

deliberately-wrong fractional-power rotation collapses to 8-14%, near chance. The

code is genuinely rotational, just not 2D.

5. Scaling the probe set from 77 to 500 tokens across 25 categories showed the pattern

generalizes to "small closed enumerable named sets" broadly (continents, zodiac

signs, playing-card ranks, planets), not specifically cyclic/temporal concepts.

6. Testing the mechanism on those new categories partially replicated: playing

cards reached 82%/73% accuracy (p < 1e-6 vs. chance), zodiac signs 42%/33%

(p < 0.02), but planets' 33%/17% was not statistically distinguishable from

chance (p = 0.17, only 6 testable items) - reported as a genuine limitation, not

smoothed over. The three testable categories' accuracy tracks average subject-name

token length in the direction a diagnosed leakage mechanism predicts, but with wide

CIs across only 3-5 points, stated in the paper as consistent with, not

confirming, that explanation.

I'm including the negative/partial results deliberately - the null at step 1, the

failed 2D test at step 3, and the non-significant planets result at step 6 are, I

think, the actual evidence this is careful work rather than a result chased into

existing.

What the funding is for:

Two concrete extensions, both direct continuations of the open items already stated in

the paper's own Limitations section (not new scope invented for this application):

1. Scale to a larger open model (candidates: Llama-3-70B, Qwen2.5-72B, or similar,

4-bit quantized) to test whether the rotational-code finding and the token-length

pattern found in items 5-6 above hold at a different model scale, or are an

artifact of the sub-7B regime tested so far.

2. A direct causal test of the token-length mechanism, which the paper currently

only supports correlationally. The real test (already specified, not yet run):

patch every sub-token position of a multi-token subject (zodiac signs, planets),

not just the last one, and check whether that recovers the accuracy gap against

playing cards/days/months. This is the single most concrete open question the

existing work leaves on the table.

Both are inference-only (no training), so the compute need is real but modest -

the limiting factor is VRAM to load a larger model, not GPU-hours.

Budget:

Est. cost:

Cloud GPU rental (A100 80GB or equivalent, ~250-400 hrs across both experiments, spot/on-demand pricing on RunPod/Lambda/Vast.ai, ~$1.50-2/hr) - $500 - $800

Buffer for reruns, debugging, and a second model if the first result is ambiguous - $500

Storage/bandwidth (activation caches for a 70B-class model across 500+ probe tokens are non-trivial) $150

Total, compute-only ask | ~$1,200 - $1,500

This is a deliberately small, scoped ask: enough to answer one specific open question

with the existing methodology, not a request to fund an open-ended research program.

Timeline:

- Weeks 1-2: set up larger-model inference (quantization, activation hooks), rerun the

existing causal-patching pipeline as a sanity check against the 7B results.

- Weeks 3-5: run the sub-token-position patching experiment on zodiac/planets; run the

full generalization sweep on the larger model.

- Weeks 6-7: analysis, statistics, writeup as a revision/extension of the existing

paper (or a short follow-up note if the result is a clean yes/no).

- Week 8: publish updated code/data/paper to the existing public repo.

Risks and how I'd handle them:

- The larger model shows a different mechanism entirely. That's a genuine result,

not a failure - would be reported as such, same as the planets null result already

was in the completed work.

- Sub-token patching doesn't cleanly resolve the token-length question (multi-token

binding might not be separable this way). Fallback: report the attempt and what it

rules in/out, rather than forcing a clean story.

- Quantization artifacts confound results at the larger scale. Mitigated by running

the existing 7B pipeline unchanged as a same-session control before trusting any new

finding.

Why this is worth funding:

Not because it will resolve interpretability in general, but because it's a small,

well-scoped, already-partially-answered question with a track record showing the

applicant reports negative results honestly rather than only positive ones - which is

the main thing that makes a small independent grant worth the risk relative to a

larger, unproven ask.