Project summary

We are planning an online AI Safety for Builders pilot fellowship to build tools and products.

If we take AI progress and its implications seriously, ensuring AI goes well means automating a lot of AI safety work. Which means building more AI safety tools, infrastructure, pipelines, etc.

This is currently under-served. While a bunch of fellowships and research orgs exist to steer AI safety research, relatively less is done to build AI safety tools.

What is being done now?

The past few months have seen some focus, albeit indirect, on AI safety tools and AI safety for builders:

• Generalist- and fieldbuilder-focused streams and programmes, such as Generator Residency, the MATS and ERA fieldbuilding streams, BlueDot’s incubator week, and Gen Stream for generalists.

• VCs like Halcyon, Juniper, SAIF.vc, Babuschkin Ventures, Fifty Years, Mythos’s Substack and Lionheart.

• Incubators like Seldon Lab, Surplus and Catalyze Impact.

• A few dedicated orgs like Neolithic and Meridian launched.

How will this help counter risks?

Broadly, in two ways:

1. Bring AI safety work from research orgs and toy implementations into real-world use faster.

2. Engage countries like India, where the AI safety ecosystem is small despite the country's significant share of the world's population and technical talent. Indian tech has a strong builder focus, and this fellowship can help build momentum for that talent to contribute to AI safety.

What are this project's goals? How will you achieve them?

Our goal is to help promising AI safety ideas cross the gap between research and real world use. Many useful ideas stop at a paper, proposal, or proof of concept. At the same time, technically strong people want to contribute to AI safety but lack a tractable project, domain guidance or a team.

We want to address both bottlenecks through a 4 week, online, part time AI Safety for Builders Fellowship. Small teams of builders will work with AI safety experts to produce tools, evals and prototypes that can be tested and continued after the programme.

Outcome for the fellows

1. Turn important safety problems into buildable projects: Each project will begin with a clear use case, intended user and scoped deliverable with a mentor.

2. Produce useful artifacts: Outputs can include open-source tools, eval suites, benchmarks, datasets, dashboards, standards, implementation guides or tested product specifications.

3. Develop a pipeline of AI safety builders: We will recruit globally while making a deliberate effort to reach strong technical talent in India and other regions underrepresented in AI safety.

4. Create paths for promising work to continue: either adoption by an org, open-source maintenance, further research, further grants, etc.

Areas where AI safety needs builders

See the areas and fields we have identified so far: main areas and fields

How will this funding be used?

[below is for the approved funds of $10,000, which fund the 4-week pilot described in the next section; if we get more funds, we'll be able to run a longer or additional cohort of the AI safety for builders fellowship]

• Compute, model APIs and software: $5,000

• Organizer time: $2,000

• Programme operations, platform and communications: $1,500

• Accounting, payment and compliance reserve: $1,000

• General contingency: $500

What will the fellowship look like?

We plan to run the pilot as a free, online, part-time 4-week micro fellowship, with 20-25 fellows in 7-10 teams of 2-5, contributing around 8-12 hours per week. Fellows join a small team on a project scoped by a mentor and ship something concrete and open-source. The four weeks cover Scope, Build, Test, and Demo Day with awards. Mentor matching will be based on project needs, complementary skills, preferences, time zones, and working style.

More details, including the call for mentors, are on the fellowship site: https://aisafety.builders/

Who is on your team? What's your track record on similar projects?

• Ankur Pandey is a 3 time AI/SaaS founder, builder, and AI safety generalist with 15 years of applied AI experience and over a decade of involvement in EA, rationality, and AI safety, beginning with a MIRI workshop in 2013.

  He started one of India’s earliest large freethinker and rationalist communities in 2010 and has since helped grow the country’s AI safety ecosystem through events, ACX meetups, and an AI safety mixer at the India AI Impact Summit 2026. His recent training includes programmes with BlueDot Impact, CAIS, Cambridge Digital Minds, SPAR, AI Safety Camp, and PIBBSS workshop, where he has worked on AI reasoning failures and integrity disorders in AI systems.

  He also created a comprehensive global directory of AI safety organisations, consulted with Alice on AI security, and currently advises Groundless.ai on epistemic and AI-codependency risks.

  As a founder, he has built products used by hundreds of thousands of people, spanning fake-news detection and general-purpose AI agents, and has received an award from OpenAI. His background also includes public policy and the humanities, complementing his technical and entrepreneurial experience.

• Soumya Jain is an AI governance researcher, research manager, and product leader with experience spanning technical safety, governance, enterprise AI, and programme delivery. She currently serves as Research Manager for the Cambridge AI Safety Hub’s MARS 5 programme, where she helps technical-safety and governance teams do good research.

  Previously, as a MARS 4 research fellow, she led a project on compute policy and AI chip export controls with Erich Grunewald (IAPS). She also supports the Jinesis Lab at the University of Toronto and EuroSafe AI with grant development and research management across AI governance research streams.

  Before moving into AI safety, Soumya spent more than five years in product management and data science across enterprise AI, banking, and regulated fintech. At Terrabase, she led product for an agentic product that transforms siloed business data and documents into governed context, insights, and decisions.

  She has also judged Apart Research’s Global South AI Safety Hackathon, mentored through the Paragon Policy Fellowship, and been selected to mentor an agent-governance project through the Sentient Futures Project Incubator. She founded and facilitated Decoding the Mind, a discussion series on morality, phenomenology, self-reference, and suffering.
  

What are the most likely causes and outcomes if this project fails?

The first two risks are more urgent.

• Teams don't finish. Some teams may not complete their projects or end up with rough, untested prototypes.

  • Scope is too ambitious.

  • Data or infrastructure isn't available.

  • Teams aren't matched well.

  • Participants drop out.

  • Mentorship is inconsistent.

• Projects aren't useful. Teams build something, but it doesn't solve an important problem or isn't practical to use.

  • Projects are chosen for novelty instead of real need.

  • No clear end user.

  • Practical constraints are discovered too late.

• Nothing gets adopted. A good prototype is built, demoed, and then forgotten.

  • No organisation takes ownership.

  • No one maintains or deploys it.

  • There's no clear path to continue after the fellowship.

How much money have you raised in the last 12 months, and from where?

The project has been funded with 10,000 USD by grantmaking.ai, see link to the project page