Intuitively AlphaGo does not "give a damn" about winning go games---though clearly it is a game-winning optimizer. If this is true, what would it take for AI systems genuinely to "give a damn" about their goals? Building on Terrence Deacon's groundbreaking work on the physical basis of teleology, I'm creating a framework to make this distinction. This work directly addresses which systems might pose agentic dangers through convergent instrumental goals, and at the same time addresses which systems might themselves be moral patients deserving ethical consideration. Through weekly collaborations with John Wentworth and Sahil (formerly MIRI), plus participation in Nick Bostrom's Digital Minds working group, I'm working to operationalize these philosophical insights into concrete assessment tools for AI developers.
My project aims to develop a more tractable approach to both digital sentience and dangerous AI agency. This approach centers teleology as fundamental, rather than starting with consciousness or "optimization". The central question is: which physical systems genuinely "give a damn" about their goals, versus those that "merely optimize"? Systems that give a damn plausibly have interests that could ground subjective well-being---and they're also more likely to pursue convergent instrumental goals like self-preservation that are especially likely to pose agency risks.
Building on Terrence Deacon's groundbreaking work on the physical basis of teleology, I'm developing a framework for distinguishing AI systems that exhibit genuine agency from those that perform sophisticated optimization. If Deacon's research is correct---that genuine "caring" must be built from the bottom up, agency built upon agency---then different AI architectures may have fundamentally different capacities for the kind of agency that could ground sentience. For example, traditional von Neumann architectures might be incapable of genuine caring, regardless of computational sophistication.
I achieve these goals through intensive collaboration with technical AI researchers: weekly meetings with John Wentworth (independent researcher) and Sahil (formerly MIRI) to analyze AI architectures against teleological criteria, plus biweekly participation in the Digital Minds working group led by Eleos AI and Nick Bostrom. This methodology involves the standard analytic philosopher approach of seeking conceptual coherence and formal models where there's little clear understanding beforehand.
The funding ($12,000 per semester) would support a one-course teaching reduction, freeing approximately six hours per week for focused research. This time is essential for the intensive preparation required for technical collaborations and sustained engagement with Deacon's complex theoretical framework.
Each collaboration meeting requires significant "homework" time---I need to deeply understand technical AI architectures to apply teleological analysis meaningfully. During a typical teaching semester, finding this focused time is nearly impossible. The course reduction enables me to maintain regular collaborative momentum rather than limiting this work to school breaks, which would significantly slow progress and reduce practical relevance.
I serve as the lead researcher, though I take guidance from my collaborators. My core team includes John Wentworth (independent researcher, familiar to the alignment community) and Sahil (formerly MIRI's Agent Foundations team). I also participate in the Digital Minds reading group with Nick Bostrom, Patrick Butlin, and Rob Long. I also find frequent occasions to collaborate with Abram Demski, Ramana Kumar, and other technical alignment researchers.
I've been working and publishing in philosophy of AI since 2005---back when AI work was considered more a stain than a laurel on a philosophical CV. An early publication from 2015 attempted to refute Bostrom's arguments, but found them harder to dismiss than I liked, and by about 2018 I became dedicated to alignment work. I've published academic papers and given many talks on alignment, and I'm learning to engage in non-traditional academic forums like co-posting on LessWrong.
My work has been supported by the Survival and Flourishing Fund, Long-Term Future Fund, Future of Life Institute, Nonlinear, Center for AI Safety, and Ramana Kumar. Most recently, SFF funded an extra sabbatical semester in 2021-2022, LTFF funded a course buyout for 2022-2023, and a combination of CAIS/Nonlinear/Ramana Kumar funded 2023-2024.
The primary risk is that philosophical frameworks may not translate to practical assessment tools that AI developers can use. I'm mitigating this by working closely with technical collaborators to ensure teleological criteria can be operationalized as concrete features.
A secondary risk is that physical substrate requirements for genuine agency may be so restrictive that no current AI systems qualify, making the framework less immediately relevant. However, even negative results would be valuable---helping identify which technological developments might be necessary for genuinely sentient AI.
The strongest objection, as OpenPhil notes in their "flag" to agent foundations work, is that philosophical approaches may not yield tractable insights on AI safety timelines. But as MIRI discovered, fundamentally difficult philosophical problems lurk beneath the computer science. Agent foundations is "philosophy with a deadline," in Bostrom's frightening phrase.
But as John Wentworth argues, we can't all keep looking for the keys under the lamp post just because "that's where the light is." If Deacon's framework is correct about the physical basis of genuine agency, it could fundamentally reshape how we think about which AI systems pose risks.
In the past 12 months, I received a $12,500 SFF speculation grant for a fall 2024 course reduction, though I did not receive the full grant to fund spring semester 2025 as well. My evaluators suggested this was only due only to budget constraints. The difference in available research time is certainly noticeable.
I was also turned down by LTFF for 2024-2025 funding. I recently applied to OpenPhil for the same funding purpose, but they've indicated they're not emphasizing theory-heavy, non-empirical research, so I'm not optimistic about that application.
FLI has also provided several smaller grants to fund alignment conference trips, including to IASEAI and both ILIAD conferences.
Steve Petersen
7 months ago
This project turned out to be separately funded for the time being. Thank you to everyone who considered it.
(no pledged money will be spent from this application)
Neel Nanda
7 months ago
I've funded this up to 1 semesters worth. On my inside view, I'm not particularly excited about this kind of theory heavy non-empirical alignment work. on the outside view it sure seems like enough people are into this kind of thing that it should be taken seriously and given at least some resources. And my sense is that AI safety is a lot more constrained by plausibly reasonable places to spend money, than money. So this seems like a pretty reasonable thing to fund. I don't feel compelled enough to fully fund this, but would be happy to see someone else who feels excited about this do so.
Good luck! I hope it goes well.
Neel Nanda
7 months ago
Could you share a few examples of your work (in the AI alignment space) that you feel proudest of? (Blog posts are fine, no need to try to stick to published papers - just whatever you consider your best work to be).Or, if I wanted to ask some alignment researchers who've engaged a bunch with your research and found it valuable, who would you recommend?
Steve Petersen
7 months ago
@NeelNanda Thanks for asking, Neel! As I say I mostly engage with John Wentworth and Sahil (I meet each weekly), and with the "Digital Minds" reading group run by Patrick Butlin, Rob Long, and Nick Bostrom (biweekly).
Here's a forthcoming paper on my AIT account of abstraction. Here are recordings of three talks I gave at ILIAD last summer. And here's a slide summary and draft link, from TAIS, of my most recent work, discussed above; there's also a talk version I gave at Hong Kong University. Let me know if you have questions!
Abram Demski
7 months ago
Steve Petersen provides an important bridge between mainstream academic philosophy and AI safety work. I have had many valuable discussions with him. He has work in algorithmic information theory, establishing an important theory of abstraction, which is a central concern of several AI safety research agendas (John Wentworth and Sam Eisenstat come to mind). This demonstrates his technical chops & ability to craft deep theories.