AI safety for builders hackathon / fellowship

AI safety for builder hackathon / fellowship - to build tools, products, etc.

A Mechanistic Analysis of Activation Verbalizers: Promise and Risks

Analyze how activation verbalizers use target-model activation concepts via PCA/DAS/patching, explain cross-family failures, and improve verbalizers

US Policy Communication Playbook for an AI Crisis

A playbook to help AI safety policy advocates communicate with the U.S. government during the window of opportunity during an AI-related crisis.

FOIs about AI Existential and Catastrophic Risks

Submitting Freedom of Information requests across EU Member States to reveal how governments understand and address advanced AI risks.

Faithful Preference Learning : Cognitively-Aligned Post-Training

This project builds cognitively-aligned preference learning that interprets feedback the way human actually decide rather than as a reward to maximize.

Formalizing mathematical AI safety research in Lean

A hand-verified library of AI-safety theorem statements in Lean 4 with AI-generated proofs, building the skills to trust AI formalization.

Getting Split-Personality Training ready for deployment

Clearing barrieers to adoption for an existing ICML-published interpretability technique that can elicit latent knowledge from red teamed model organisms

Evaluating the Tacit Coordination Capacity and Decision Theory of LMs

A benchmark to empirically investigate: (i) the ability of models to tacitly coordinate with copies of themselves and (ii) which decision theory best explains t

Scaling adoption of synthesis screening outside the US and EU

An outreach campaign from IBBIS to rapidly increase adoption of effective synthesis screening tools during a critical regulatory window.

Launching Humans in Control's Student Program

Humans in Control (HIC), a nonpartisan grassroots advocacy organization supporting AI safeguards, is raising funds to start a student program

BioRAG-Guard: Poisoning Evals for Biological AI Agents

An open-source benchmark and defense toolkit for testing whether corrupted biological databases can hijack retrieval-augmented AI agents used in genomics, prote

Help the Argentinian AI Safety Community Grow

The Argentinian AI Safety community (BAISH, baish.com.ar) is the largest in Latin-America. Support BAISH's growth, by providing funding for paying salaries.

Formally verified autoresearch for theoretical mech interp

LLM agents collaborate to discover and formally verify theorems about the internal computations of transformers, beginning with a simple pilot question: how man

Belief state geometry of language model personas

A formal, testable account of LLM persona selection as Bayesian inference, validated with model internals, so labs can monitor and steer personas.

Interpreting Latent Reasoning Models with Natural Language Autoencoders

Using deals to find misalignment in current AI models

Run experiments on what incentives for deals with AI increase performance for using agents to discover misalignment

Token taxes as a mechanism for reducing AI-driven power concentration.

A policy memo, co-authored with the Institute for Public Policy Research, resolving the open technical, economic, and legal questions blocking real-world implem

Modelling Takeoff

Flourishing Futures Fellowship

A scalable fellowship training researchers to develop interventions for achieving high-value long-term futures

Creating a better way to compile to transformer architectures