Project goals

I want to write this website: thermontology.com

Basically, the intention is to help AIs become more philosophically competent. I think philosophy/world-model-improving proceeds most efficiently by proceeding down empirical "flywheels", e.g. the development of probability theory from gambling, "going with the grain of reality" so to speak, so I'm gathering a bunch of empirical observations and potential research paths which I believe are promising for doing that. Near-term AIs being more philosophically competent would be good because they're at least kinda aligned right now. Also could make humans more philosophically competent which could be good for making good choices regarding AGI.

I also want to do research into "wide width limit" theory(stuff along the lines of NTK, https://www.lesswrong.com/posts/rduzFkTKx5pGKWKcL/mean-field-sequence-an-introduction, https://arxiv.org/abs/2605.05179).

There's people working here, but not nearly enough relative to how promising it is IMO. I also think I have good research taste.

Successful theory could then be used to develop better auto-interp tools, principled training regimes, etc.

As an intermediate goal I'm going to try to win ARC's white-box estimation challenge(https://www.lesswrong.com/posts/Kben8CzS4awCwNw5c/announcing-the-arc-white-box-estimation-challenge).

How are you planning to spend the money?

I'm in a PhD program. I want to use the money to convince my supervisor to give me a few months to focus on the above areas.

What have you done in the past that proves you will be good at doing this? focus on substance, not credentials

Regarding general research aptitude, I wrote this paper about the kolakoski sequence which is IMO very good math research: https://arxiv.org/abs/2605.14234 I think it could lead to a proof of a somewhat famous open problem (described e.g. here https://www.youtube.com/watch?v=co5sOgZ3XcM&t=3s). I wrote about the strategy whereby it could do that here: https://noahmacaulay.com/how-to-prove.pdf

Now about AI/theory in particular, I wrote these LW posts which were the first on LW writing about the "wide width limit" stuff:

https://www.lesswrong.com/posts/KrQvZM8uFjSTJ7hq3/recent-progress-in-the-theory-of-neural-networks-

https://www.lesswrong.com/posts/76cReK4Mix3zKCWNT/ntk-gp-models-of-neural-nets-can-t-learn-features

I think this was somewhat prescient relative to other safety-interested people, as people have since devoted an increasing amount of attention to these limits, e.g. dimitry vaintrob, ARC, or the recent paper "we will have a theory of deep learning". The muP and muP- depth limits are I believe the only theory things that have had an impact on ML practice.

I also think this post was pretty good at describing an underappreciated way the future could go:

https://www.lesswrong.com/posts/2XLFyhuKP7m4xpgea/will-values-and-competition-decouple-1

TLDR: https://www.lesswrong.com/posts/BuaFZud9BwkiSCGpd/alignment-might-never-be-solved-by-humans-or-ai

What is something that you think everyone else is wrong about? Convince me that you are right.

We should expect that quantum mechanics is relevant to the structure of the mind/abstractions/reality. This can be seen from an algorithmic statistics perspective.

Quantum mechanics is weird relative to a generic universe we would imagine, in other words it is in a small region of our prior, so it's surprising, but if we can make an anthropic update that life/mind is more likely there, that reduces our surprise. And making such an update would be a great thing for our epistemics by compressing very deep parts of our world model.

(The framework presented here is a bit sketchy, but IMO basically makes sense. But the generating intuition is really just "quantum mechanics is so darn weird, there must be some reason for it...")

As for what the update might be: not direct relevance(decoherence mostly prevents that) but as a sort of inductive limit/base case grounding a hierarchy of abstractions. e.g. humans are made of organs made of cells made of proteins, in each case there is an "interface layer" which is only occasionally "breached", propagating details from the lower levels upwards, quantum measurement is the base case, higher level abstractions inherit some properties from it(contextuality)

What's something you recently changed your mind on? Ideally something that you very strongly believed in.

I recently started worrying a bit more about centralization of power risks, previously I thought that any human attaining a god-like intent-aligned AI would be fine because they would likely eventually become reasonable if only after millennia. I still somewhat believe this but it now seems different people may have different extrapolated notions of "reasonable" in many respects, although perhaps the same in others. Also it started to seem that AIs might undergo similar convergence in values for similar reason(but probably still less than with other humans)

I suppose on a gut level my perception of intrinsic competitive dynamics between humans has gone up recently and this may be the root of the change.