What a special community!

Lately, I have been thinking again about how we decide the value of an idea.

Before we know whether something is actually true, we usually have only a few things to rely on: who is presenting it, how convincing the explanation sounds, their experience, and whatever early evidence is already available.

But isn't it precisely when something has not yet been proven that the real question begins?

I do not know whether someone's experience, reputation, network, or credentials are the best way to estimate the value of an idea. I also do not know whether a small amount of early evidence is enough to decide that a research project deserves a chance.

Perhaps there is no simple answer.

We all have to make decisions with incomplete information. We choose what we want to read, what we want to believe, who we want to support, and which research we think is worth pursuing. In the end, every one of those choices also carries the possibility that we are wrong.

This has made me think that perhaps the most important thing is not how confident we are in an idea when we first encounter it.

Perhaps what matters more is what happens when that idea begins to face reality.

Are we willing to measure it?

Are we willing to show the results when they do not meet our expectations?

Are we willing to question assumptions that we previously believed were correct?

And perhaps most importantly, are we willing to let the results of an experiment change our own minds?

I do not know whether this way of thinking is right.

But I have started to believe that an idea should not be judged only by how beautiful or convincing it sounds while it still exists in our minds. It also needs the opportunity to face reality.

Because that is where we eventually find something that belief, reputation, and words alone cannot provide:

results.

And those results do not always have to be successful.

Sometimes an honest failure gives us a much better question than a success we never truly understood.

@tzukitchan *SPARC's supplementary summer chapter.

[betting on the founder] im familiar with David's work, highly competent executor and very thoughtful+articulate+intentional on vibes-curation conducive with the theory of change for any given camp; really excited for SPARC's next chapter.

[betting on the founder] i'm familiar with Cecil's work and I trust that whatever project he outputs is more often than not going to be net-good.

[betting on the founder] the EA / AIS community is lucky to have someone like Constance to begin with; I've seen her various works in public and trust whatever she has to output.

@beno Makes sense, thanks!

matching my claude code sub so that i'll feel less guilty when the Day comes

Happy to see this endorsed by Gavin, Marcus, and Anton through grantmaking.ai (original proposal here).

Thanks a lot for approving this and for your encouragement, @carol !

Approved! This sounds great; excited for more geographically diverse fieldbuilding and especially in Hong Kong.

I have developed a complete and detailed research proposal for this project. The topic has also attracted interest from multiple potential academic supervisors; however, the available funding from these supervisors is currently limited. I am therefore seeking independent funding to carry out the proposed research at a meaningful scale, including the required compute, experimentation, evaluation, and open dissemination of the results.

SAIGE continues to perform / output at a high-quality level and Jessica continues to consistently execute competently.

@tzukitchan Thanks a lot. ❤️None of this would’ve been possible without your handholding.

What progress have you made since your last update?

Finished deliverable 2.1. It was scoped as:

"Does SCA work in a transformer at all? Anchor a concept such as red across the residual stream in the color-mixing task; probe each layer for the anchored concept, and confirm that completion accuracy (predicting the correct result color) matches an un-anchored baseline."

tl;dr: Yes, it works.

I tested the method piece by piece over six preregistered experiments (2.1.6-2.1.11; see Appendix A below).

Anchoring. Residual stream probes show that the target concept (red) aligns with the anchor as expected. A repulsive term reserves the subspace, keeping the alignment graded (less-red colors are less aligned; non-red colors are nearly orthogonal).

Labelling. Sparse, noisy labels suffice, as they did in M1. Labels indicate "there is something red in this equation" (but not which tokens). Smooth minimum pooling allows the regularizer to target the "right" token, attracting it to the anchor position.

Accuracy. Guiding the concept during training left the model's accuracy on its actual task essentially unchanged (it mixes colors accurately). I searched for the point where that stops being true, and the main experiments are operating well below it.

I have not yet tested whether the anchored concept can be intervened on (e.g. to prevent the model from completing red equations in particular). That is scheduled for D2.2.

Delivered as source code and experiment reports (on GitHub).

What are your next steps?

I will begin work on D2.2: anchoring operations, and intervention (selective suppression of capabilities).

I also plan to post on LessWrong about the work so far.

Is there anything others could help you with?

Same as last time: feedback from other researchers. I'm happy to discuss if that's easier than reading the reports.

I plan to seek feedback directly from some people. I said that last time too, but I didn't feel like I had enough results to talk about. I think I do now.

Appendix A. Anchored experiments

Every anchored experiment was preregistered in Git, with the method, hypotheses and gates frozen before running. Several have hypotheses that failed, which informed the next design.

2.1.6: Our first anchored transformer. We anchor red as we did in autoencoders, but this time the noisy labels are applied to whole equations (according to the first operand), and the model/optimizer has to figure out which tokens that applies to on its own. It worked: red was decodable from the residual stream where we expected it to be. But without the repulsive terms (omitted on purpose), the whole color space is dragged toward the anchor.

2.1.7: Reserving the subspace, and narrowing the anchor. 2x2 factorial: 1. We restore the anti-subspace term from M1, which pushes all samples away from the anchor to reserve the space for the labelled concept. 2. We also try limiting the anchor term to only affect the first operand (even though this would be unrealistic for language model application). Both work, but 2. worked better, and their effects stack somewhat. Figure: red is the most aligned with the anchor, but the other colors come along for the ride when the pull is not limited to the first operand.

2.1.8: Repulsion tuning. We tested anti-subspace schedules (trailing timing and strength) to find an operating point that contains the cube-wide drift without reducing selectivity. Holding the repulsion high for longer contains the drift and keeps the margin, which nothing before this did. It's unclear if the response is well-graded.

2.1.9: Softmin sequence pooling. Anchoring on op1 alone worked best so far, but in natural language we won't know which tokens hold the concept. So we pool over sequences with a soft minimum (mellowmax) and let the pull choose its own position. At the embedding it picks op1 unaided, the operating point stays healthy, and grading improves. Only the softest pooling stays graded in every run.

2.1.10: A label that doesn't point. Either operand can trigger the label, and the pooled pull finds the red operand line by line: the weight profiles track the label groups, and selectivity matches the slot oracle. Figure: mellowmax allocates the pull to the red operand automatically, whichever that is.

2.1.11: A hyperparameter survey to close out D2.1. The schedules and weights in the recipe were inherited piece by piece, some from M1, and it was unclear which were needed and what the best values were. We ablate first by replacing schedules with constants, and dropping training epochs. Then we run a search over what is left (anchor weight, pooling temperature, repulsion dose). The anchor schedule and half the epochs could go. The anti-subspace schedule must be kept: a constant delivering the same total dose loses grading.

Appendix B. One more unanchored experiment

2.1.5: Do disjoint vocabularies learn the same representations? A model trained on named and hex colors, but not both in any one equation, learns both tasks well but uses different latent representations. Curiously, a variant that included both types could mix colors between the vocabularies, but the representations remain separate. There's more to discover down this path, if we want to learn more about the latent space. Figure: residual stream probes at all depths show that the named colors are represented as a unified concept, while hex colors are represented (and computed) one channel at a time.

I spent 20hours+ handholding this bunch at the very initial stages till they have a reliable weekly-meeting cadence + structure + tacit knowledge on handling various unfamiliar decision-making situations that arise in running a group;

they are a team i literally woke up at 6am PT for to coach in the very initial stages (not an exaggeration).

i'm strongly rooting for this core team; namely under Kenneth Chan's and Abeer's leadership.

I just found out that one Manifund grantee met their serious longtime girlfriend of 2y through this dating show! Huge shoutout to @42irrationalist for organizing, and the various impact investors and retro funders for helping make this happen.

Final update and close-out (August 2026)

This update closes the project. Apologies that it comes just a bit late! The funded work stopped more than a year ago.

Funding. We asked for $670k. We received $80,680. Thank you, Austin, Adam, and all donors. Because we did not receive full funding and because of inevitable changes in priorities when doing the research, we did not ultimately try to complete all four projects. Other sources supplied the remaining runway, which included grants from the UK AI Security Institute (AISI), Coefficient Giving, and SFF.

Results for the four proposed projects:

1. Understanding-based evals. This work fed into our "singular psychometrics" project and then the SMDL paper. We did not end up publishing singular psychometrics. Other researchers took up some of the key ideas independently (such as the importance of item response theory), so the value of the project decreased. Some of the remaining ideas went into Compressibility Measures Complexity: Minimum Description Length Meets Singular Learning Theory (https://arxiv.org/abs/2510.12077), funded by UK AISI.

2. Backdoor detection with data-restricted LLCs. We did not build a backdoor detector. But someone did independently propose and complete a similar project, with successful results from using the Bayesian Influence Function (BIF) for backdoor detection. The BIF is closed related to data-restricted LLCs.

3. Simplicity biases and deceptive alignment. The conceptual work became our position paper, You Are What You Eat (https://arxiv.org/abs/2502.05475), on the "Occam gap". The applied work continues in Patterning: The Dual of Interpretability (ICML 2026) (https://arxiv.org/abs/2601.13548), which targets reward-model biases. We are continuing research inspired by this thread.

4. Many-shot jailbreaking. We did not do this project. We studied related questions in the project, Elicitation without Backpropagation (https://arxiv.org/abs/2607.18804).

Timaeus has grown a lot in the meantime, and has now merged into Resolution.

Thank you again for your support!

What progress have you made since your last update?

Spent the last two days fixing the skill/memory feature — turns out it was completely broken after it went live. Users could ask the agent to create or manage skills in chat, but nothing actually happened. The agent just wrote text back saying "done!" when it hadn't done anything.

Dug into it and found 6 separate bugs stacked on top of each other. Each one alone would've broken the feature, but together it was totally dead:

• The agent literally didn't know the tools existed (no prompt guidance)

• Even if it did, the tools weren't registered in the DB so they'd get filtered out anyway

• The consent gate that's supposed to pop up a confirmation card? Its trigger logic was unreliable

• Asking "what skills do I have?" would accidentally trigger an investment clarification popup (super confusing UX)

• And a bunch of other small stuff

Also built memory deletion, which was supposed to be there from the start but got missed. So now users can both save and remove things the agent remembers about them.

Fixed everything and tested on production with real conversations — create, list, delete skills, save and forget memories, all going through the consent flow properly. Tested with both deepseek and nemotron models to make sure it's not model-specific.

What are your next steps?

• Keep an eye on whether people actually use the skill/memory stuff now that it works

• The prompt is pretty strict but weak models (like deepseek free tier) still occasionally pretend they did something without actually calling the tool. Might add a UI hint suggesting stronger models for this feature

• There's a missing DB table for audit logging that I keep running into — not urgent but should fix eventually

Is there anything others could help you with?

• The deploy platform (Zeabur) sometimes can't reach GitHub during builds — nothing to do with our code, just network hiccups. Usually fixes itself on retry

• I've got an approximation for user retention stats but doing it properly would need a new table. Not blocking anything, just a nice-to-have for later

Who can say no to unlocking the limitless potential of this community? Happy to support BASE in all the ways I can. We need your voices in this space.

I first found out about the New Critic when on their article "p(doom)", which I found to be a good dive into the prediction markets scene, and extremely well written. I invited one of their journalists to come to Manifest, resulting in their article "Manifest Man".

Rufus then reached out asking about a larger grant to continue sponsoring longform pieces similar to those. I've learned much from longform; find quality longform writing on topics I care about to be rare; and believe that longform is extremely underprovided as a public good.

(fwiw, some other initiatives that Manifund has supported in this vein include both Inkhaven cohorts, the Asterisk blogging fellowship, and the Blog Revival Project.)

After some back and forth, and a call with the whole New Critic editorial team, I've decided to recommend a $20k grant, split into two chunks: $10k unrestricted for operations, and $10k to fund 3 longform pieces on topics of interest to Manifund (as Rufus outlines above). I'm excited to both support this team in their ambitions, and see whether this commission model can produce essays that we find to be exemplary.

What a special community!

Lately, I have been thinking again about how we decide the value of an idea.

Before we know whether something is actually true, we usually have only a few things to rely on: who is presenting it, how convincing the explanation sounds, their experience, and whatever early evidence is already available.

But isn't it precisely when something has not yet been proven that the real question begins?

I do not know whether someone's experience, reputation, network, or credentials are the best way to estimate the value of an idea. I also do not know whether a small amount of early evidence is enough to decide that a research project deserves a chance.

Perhaps there is no simple answer.

We all have to make decisions with incomplete information. We choose what we want to read, what we want to believe, who we want to support, and which research we think is worth pursuing. In the end, every one of those choices also carries the possibility that we are wrong.

This has made me think that perhaps the most important thing is not how confident we are in an idea when we first encounter it.

Perhaps what matters more is what happens when that idea begins to face reality.

Are we willing to measure it?

Are we willing to show the results when they do not meet our expectations?

Are we willing to question assumptions that we previously believed were correct?

And perhaps most importantly, are we willing to let the results of an experiment change our own minds?

I do not know whether this way of thinking is right.

But I have started to believe that an idea should not be judged only by how beautiful or convincing it sounds while it still exists in our minds. It also needs the opportunity to face reality.

Because that is where we eventually find something that belief, reputation, and words alone cannot provide:

results.

And those results do not always have to be successful.

Sometimes an honest failure gives us a much better question than a success we never truly understood.

@tzukitchan *SPARC's supplementary summer chapter.

[betting on the founder] im familiar with David's work, highly competent executor and very thoughtful+articulate+intentional on vibes-curation conducive with the theory of change for any given camp; really excited for SPARC's next chapter.

[betting on the founder] i'm familiar with Cecil's work and I trust that whatever project he outputs is more often than not going to be net-good.

[betting on the founder] the EA / AIS community is lucky to have someone like Constance to begin with; I've seen her various works in public and trust whatever she has to output.

@beno Makes sense, thanks!

matching my claude code sub so that i'll feel less guilty when the Day comes

Happy to see this endorsed by Gavin, Marcus, and Anton through grantmaking.ai (original proposal here).

Thanks a lot for approving this and for your encouragement, @carol !

Approved! This sounds great; excited for more geographically diverse fieldbuilding and especially in Hong Kong.

I have developed a complete and detailed research proposal for this project. The topic has also attracted interest from multiple potential academic supervisors; however, the available funding from these supervisors is currently limited. I am therefore seeking independent funding to carry out the proposed research at a meaningful scale, including the required compute, experimentation, evaluation, and open dissemination of the results.

SAIGE continues to perform / output at a high-quality level and Jessica continues to consistently execute competently.

@tzukitchan Thanks a lot. ❤️None of this would’ve been possible without your handholding.

What progress have you made since your last update?

Finished deliverable 2.1. It was scoped as:

"Does SCA work in a transformer at all? Anchor a concept such as red across the residual stream in the color-mixing task; probe each layer for the anchored concept, and confirm that completion accuracy (predicting the correct result color) matches an un-anchored baseline."

tl;dr: Yes, it works.

I tested the method piece by piece over six preregistered experiments (2.1.6-2.1.11; see Appendix A below).

Anchoring. Residual stream probes show that the target concept (red) aligns with the anchor as expected. A repulsive term reserves the subspace, keeping the alignment graded (less-red colors are less aligned; non-red colors are nearly orthogonal).

Labelling. Sparse, noisy labels suffice, as they did in M1. Labels indicate "there is something red in this equation" (but not which tokens). Smooth minimum pooling allows the regularizer to target the "right" token, attracting it to the anchor position.

Accuracy. Guiding the concept during training left the model's accuracy on its actual task essentially unchanged (it mixes colors accurately). I searched for the point where that stops being true, and the main experiments are operating well below it.

I have not yet tested whether the anchored concept can be intervened on (e.g. to prevent the model from completing red equations in particular). That is scheduled for D2.2.

Delivered as source code and experiment reports (on GitHub).

What are your next steps?

I will begin work on D2.2: anchoring operations, and intervention (selective suppression of capabilities).

I also plan to post on LessWrong about the work so far.

Is there anything others could help you with?

Same as last time: feedback from other researchers. I'm happy to discuss if that's easier than reading the reports.

I plan to seek feedback directly from some people. I said that last time too, but I didn't feel like I had enough results to talk about. I think I do now.

Appendix A. Anchored experiments

Every anchored experiment was preregistered in Git, with the method, hypotheses and gates frozen before running. Several have hypotheses that failed, which informed the next design.

2.1.6: Our first anchored transformer. We anchor red as we did in autoencoders, but this time the noisy labels are applied to whole equations (according to the first operand), and the model/optimizer has to figure out which tokens that applies to on its own. It worked: red was decodable from the residual stream where we expected it to be. But without the repulsive terms (omitted on purpose), the whole color space is dragged toward the anchor.

2.1.7: Reserving the subspace, and narrowing the anchor. 2x2 factorial: 1. We restore the anti-subspace term from M1, which pushes all samples away from the anchor to reserve the space for the labelled concept. 2. We also try limiting the anchor term to only affect the first operand (even though this would be unrealistic for language model application). Both work, but 2. worked better, and their effects stack somewhat. Figure: red is the most aligned with the anchor, but the other colors come along for the ride when the pull is not limited to the first operand.

2.1.8: Repulsion tuning. We tested anti-subspace schedules (trailing timing and strength) to find an operating point that contains the cube-wide drift without reducing selectivity. Holding the repulsion high for longer contains the drift and keeps the margin, which nothing before this did. It's unclear if the response is well-graded.

2.1.9: Softmin sequence pooling. Anchoring on op1 alone worked best so far, but in natural language we won't know which tokens hold the concept. So we pool over sequences with a soft minimum (mellowmax) and let the pull choose its own position. At the embedding it picks op1 unaided, the operating point stays healthy, and grading improves. Only the softest pooling stays graded in every run.

2.1.10: A label that doesn't point. Either operand can trigger the label, and the pooled pull finds the red operand line by line: the weight profiles track the label groups, and selectivity matches the slot oracle. Figure: mellowmax allocates the pull to the red operand automatically, whichever that is.

2.1.11: A hyperparameter survey to close out D2.1. The schedules and weights in the recipe were inherited piece by piece, some from M1, and it was unclear which were needed and what the best values were. We ablate first by replacing schedules with constants, and dropping training epochs. Then we run a search over what is left (anchor weight, pooling temperature, repulsion dose). The anchor schedule and half the epochs could go. The anti-subspace schedule must be kept: a constant delivering the same total dose loses grading.

Appendix B. One more unanchored experiment

2.1.5: Do disjoint vocabularies learn the same representations? A model trained on named and hex colors, but not both in any one equation, learns both tasks well but uses different latent representations. Curiously, a variant that included both types could mix colors between the vocabularies, but the representations remain separate. There's more to discover down this path, if we want to learn more about the latent space. Figure: residual stream probes at all depths show that the named colors are represented as a unified concept, while hex colors are represented (and computed) one channel at a time.

I spent 20hours+ handholding this bunch at the very initial stages till they have a reliable weekly-meeting cadence + structure + tacit knowledge on handling various unfamiliar decision-making situations that arise in running a group;

they are a team i literally woke up at 6am PT for to coach in the very initial stages (not an exaggeration).

i'm strongly rooting for this core team; namely under Kenneth Chan's and Abeer's leadership.

I just found out that one Manifund grantee met their serious longtime girlfriend of 2y through this dating show! Huge shoutout to @42irrationalist for organizing, and the various impact investors and retro funders for helping make this happen.

Final update and close-out (August 2026)

This update closes the project. Apologies that it comes just a bit late! The funded work stopped more than a year ago.

Funding. We asked for $670k. We received $80,680. Thank you, Austin, Adam, and all donors. Because we did not receive full funding and because of inevitable changes in priorities when doing the research, we did not ultimately try to complete all four projects. Other sources supplied the remaining runway, which included grants from the UK AI Security Institute (AISI), Coefficient Giving, and SFF.

Results for the four proposed projects:

1. Understanding-based evals. This work fed into our "singular psychometrics" project and then the SMDL paper. We did not end up publishing singular psychometrics. Other researchers took up some of the key ideas independently (such as the importance of item response theory), so the value of the project decreased. Some of the remaining ideas went into Compressibility Measures Complexity: Minimum Description Length Meets Singular Learning Theory (https://arxiv.org/abs/2510.12077), funded by UK AISI.

2. Backdoor detection with data-restricted LLCs. We did not build a backdoor detector. But someone did independently propose and complete a similar project, with successful results from using the Bayesian Influence Function (BIF) for backdoor detection. The BIF is closed related to data-restricted LLCs.

3. Simplicity biases and deceptive alignment. The conceptual work became our position paper, You Are What You Eat (https://arxiv.org/abs/2502.05475), on the "Occam gap". The applied work continues in Patterning: The Dual of Interpretability (ICML 2026) (https://arxiv.org/abs/2601.13548), which targets reward-model biases. We are continuing research inspired by this thread.

4. Many-shot jailbreaking. We did not do this project. We studied related questions in the project, Elicitation without Backpropagation (https://arxiv.org/abs/2607.18804).

Timaeus has grown a lot in the meantime, and has now merged into Resolution.

Thank you again for your support!

What progress have you made since your last update?

Spent the last two days fixing the skill/memory feature — turns out it was completely broken after it went live. Users could ask the agent to create or manage skills in chat, but nothing actually happened. The agent just wrote text back saying "done!" when it hadn't done anything.

Dug into it and found 6 separate bugs stacked on top of each other. Each one alone would've broken the feature, but together it was totally dead:

• The agent literally didn't know the tools existed (no prompt guidance)

• Even if it did, the tools weren't registered in the DB so they'd get filtered out anyway

• The consent gate that's supposed to pop up a confirmation card? Its trigger logic was unreliable

• Asking "what skills do I have?" would accidentally trigger an investment clarification popup (super confusing UX)

• And a bunch of other small stuff

Also built memory deletion, which was supposed to be there from the start but got missed. So now users can both save and remove things the agent remembers about them.

Fixed everything and tested on production with real conversations — create, list, delete skills, save and forget memories, all going through the consent flow properly. Tested with both deepseek and nemotron models to make sure it's not model-specific.

What are your next steps?

• Keep an eye on whether people actually use the skill/memory stuff now that it works

• The prompt is pretty strict but weak models (like deepseek free tier) still occasionally pretend they did something without actually calling the tool. Might add a UI hint suggesting stronger models for this feature

• There's a missing DB table for audit logging that I keep running into — not urgent but should fix eventually

Is there anything others could help you with?

• The deploy platform (Zeabur) sometimes can't reach GitHub during builds — nothing to do with our code, just network hiccups. Usually fixes itself on retry

• I've got an approximation for user retention stats but doing it properly would need a new table. Not blocking anything, just a nice-to-have for later

Who can say no to unlocking the limitless potential of this community? Happy to support BASE in all the ways I can. We need your voices in this space.

I first found out about the New Critic when on their article "p(doom)", which I found to be a good dive into the prediction markets scene, and extremely well written. I invited one of their journalists to come to Manifest, resulting in their article "Manifest Man".

Rufus then reached out asking about a larger grant to continue sponsoring longform pieces similar to those. I've learned much from longform; find quality longform writing on topics I care about to be rare; and believe that longform is extremely underprovided as a public good.

(fwiw, some other initiatives that Manifund has supported in this vein include both Inkhaven cohorts, the Asterisk blogging fellowship, and the Blog Revival Project.)

After some back and forth, and a call with the whole New Critic editorial team, I've decided to recommend a $20k grant, split into two chunks: $10k unrestricted for operations, and $10k to fund 3 longform pieces on topics of interest to Manifund (as Rufus outlines above). I'm excited to both support this team in their ambitions, and see whether this commission model can produce essays that we find to be exemplary.