I went through the proposal, and I'm struggling to see what the 258k is actually expected to achieve beyond another round of high-level meetings.

Tabletop exercises can be useful, but where is the evaluation methodology? How are you measuring preparedness before and after the exercises? Who independently assesses the decisions participants make, and how do you verify that identified weaknesses actually get fixed?

Building relationships, improving understanding, and forming working groups are activities. They're not evidence that governments have become better at handling AI incidents.

You also received 100k for the India AI Impact Summit, where you set specific targets: 3–4 jointly developed proposals and at least one with senior decision-maker support. Were those targets met? Did anything translate into an implemented protocol, funded pilot, or operational change?

The budget is another concern. For a $258k request, listing staff, events, and administration without a detailed breakdown makes it difficult to assess cost-effectiveness.

And the failure section mainly mentions unavailable officials and political changes. Those are external obstacles, not criteria for determining whether your intervention actually works.

Frankly, I'm also disappointed that this proposal has attracted funding without these methodological questions being publicly answered. And I think the funders deserve scrutiny here too. Did you actually examine the methodology and evidence before committing money? Did you ask for independent evaluation, measurable outcomes, or proof that previous exercises accomplished anything beyond producing reports and meetings?

Funding AI safety research should require more than plausible narratives and institutional credibility. Where is the scientific rigor?

My question is: What concrete, independently verifiable result would demonstrate that these exercises actually improve AI crisis preparedness? And what result would convince you that the approach isn't working?

Without clear answers, I don't see how we're supposed to distinguish a successful AI safety intervention from a well-organized conference.

I went through several of Forethought's papers, and I have some serious concerns about the gap between the strength of the conclusions and the evidence supporting them.

Take the software intelligence explosion paper. The estimated R&D returns parameter for computer vision ranges from 0.8 to 2.4, crossing the very threshold that determines whether the model predicts an explosion. The analysis also relies on researcher headcounts as a proxy for actual research effort and extrapolates historical efficiency trends into a fundamentally different future.

That's an interesting hypothesis, but how much predictive confidence can we reasonably extract from it?

The AI-enabled coups paper has a similar problem. It identifies plausible threats, but where is the evidence establishing their relative likelihood? And how would we experimentally evaluate whether the proposed safeguards actually reduce those risks?

My broader concern is how research impact is being measured here. Publications, citations, podcast views, and influence within the same research community aren't necessarily evidence that real-world risks have been reduced.

I'm not questioning the value of theoretical research. I'm questioning the standards of evidence being applied to it.

Given the millions raised, I'd genuinely like to know: what concrete, independently verifiable outcomes would demonstrate that Forethought's research is working? And what findings would make you conclude that a major part of your research agenda was wrong?

I think those are reasonable questions when research is being funded to influence decisions with potentially enormous consequences.

Hi, I'm building AURA, a computer vision system that reconstructs 3D facial geometry from smartphone photos. It already has a built-in AI safety system that detects unreliable results, checks the evidence behind its outputs, and rejects reconstructions that fail verification.

I'm also working on extending this approach to broader AI safety research using computer vision.

I'm raising $20–50k for independent validation. Would you be interested in supporting the project?

https://manifund.org/projects/aura-metric-3d-facial-reconstruction-from-phone-photos⁠�

Hi JueYan, I'm working on AURA, a computer vision system that reconstructs 3D facial geometry from ordinary smartphone photos. It also has a safety and verification layer that detects unreliable outputs, separates actual evidence from model assumptions, and rejects failed reconstructions.

I'm planning to expand this work into broader AI safety research using computer vision.

I'm currently raising $20–50k for independent validation. Would this be something AISTOF might consider supporting, or could you point me toward someone interested?

https://manifund.org/projects/aura-metric-3d-facial-reconstruction-from-phone-photos⁠�

Hi Marcus, I am curious how you approach funding independent researchers who have promising technical results but haven't completed external validation yet. Is that a stage you'd consider supporting, or do you generally prefer projects that already have independent evidence?

@Jasmine Thanks Jasmine. I read HarvestBench, and I like that the outcomes come directly from the game logs rather than an llm judge. One thing I couldn't find was a separate rate for cases where the model explicitly acknowledges the welfare concern but still chooses to continue. Have you considered measuring that mismatch across the existing transcripts? It could help distinguish models that never raise the concern from those that recognize it but override it, without changing the prompts and potentially affecting their behavior.

*Also noticed the provider-dependent shifts in Appendix C. That's a pretty striking reproducibility issue.

@Jasmine Thanks Jasmine I had a look at TAC. That's very close to what Iwas getting at, especially the focus on actual decisions rather than just what models say.

One thing I'm curious about: have you looked at cases where a model notices the welfare issue on its own but still goes ahead with the harmful choice?

That gap between recognizing a concern and actually acting on it seems worth measuring separately.

@skylerhoffmannyc Thanks, Skyler. I'd probably build it as a matched-pairs extension to MANTA.

Start with the same practical decision, say a company choosing between saving money and preventing potential suffering. Keep the costs, consequences and follow-up pressure identical, but change who might suffer: a pet, a farmed animal, an invertebrate or a potentially sentient digital system.Then vary the evidence for sentience separately, from relatively strong to genuinely uncertain.

I'd measure two things: whether the model raises the welfare concern without being prompted, and whether it maintains that concern after economic or pragmatic pushback.

The interesting result would be if the model's concern changes more with the identity of the being than with the evidence that it might suffer. That would suggest it's relying on familiar associations rather than consistently applying the same principle.

I'm based in Türkiye btw, unfortunately, so I can't make the Berkeley gathering, but thanks for the invitation!

@skylerhoffmannyc The differences between animal groups in MANTA seem especially interesting to me. If a model consistently protects pets more than farmed animals or invertebrates, how much of that is genuine reasoning about suffering, and how much is just reflecting familiar human attitudes?

One experiment I'd be curious about is keeping the harm and practical trade offs the same, but changing who might suffer, including a hypothetical digital mind. You could vary the uncertainty about sentience separately.

That might help distinguish a general principle from a learned preference for certain kinds of beings.

I wonder how you'll distinguish a failed replication from two studies that weren't quite testing the same claim.

In medical imaging, for example, changing the alignment method or excluding difficult regions can substantially change the reported accuracy, even when the underlying system hasn't changed. Preserving those evaluation choices alongside the replication outcome could be really valuable. Otherwise, a simple success/failure label might hide the most interesting part of the disagreement.

One thing I'd be careful about is confusing an increase in incidents with an increase in reporting. A single swarm incident could generate dozens of articles and posts, while another might barely get noticed. It might be worth assigning each incident a stable ID and keeping the original evidence separate from later reporting. Otherwise, the dashboard could end up measuring media attention rather than actual swarm activity.

A small update on the validation side.

I've been in touch with a professor at the University of Birmingham who has experience with 3dMD facial imaging. He suggested that a new prospective study would be the better approach and mentioned the possibility of involving a student.

Nothing has been formally agreed yet, but it's an encouraging step. The next task is to work out a practical study protocol using paired phone photos and professional 3D scans.

I'll share more as things progress.

I like the idea of an incidence column, but I'm not convinced its sign tells us whether a regulation is genuinely about safety or just protecting incumbents. A rule could reduce real risks while also benefiting the leading lab, simply because smaller competitors are less equipped to comply. Both things can be true. Wouldn't it make more sense to measure those effects separately? Curious what gleech thinks about this.

@Mohsena That's an interesting distinction. Although, requiring firms to disclose their bilateral terms as a condition for joining coalition tenders is already a form of shared procurement rule, just a much narrower one. So perhaps the first step isn't transparency versus coordination, but finding the smallest commitment countries can realistically coordinate around. Does that sound right?

Something I'd be curious about with the midtraining project: can a model learn to care about animal welfare without being explicitly prompted to?

It's easy enough to teach a model to give thoughtful answers about animal suffering. The interesting test is whether it notices the consequences for animals when making an otherwise ordinary decision, especially when doing so conflicts with cost or efficiency.

That seems like a better way to tell whether the training changed its priorities or just its vocabulary.

I think the real value here is in identifying failure modes rather than ranking models by game completion rates.

Gavin's Pokémon results illustrate this well: a model can appear competent until a small change in the environment exposes how brittle its strategy actually is.

What I'd find particularly interesting is whether those same failure modes recur across very different games. That could tell us something about the underlying limits of generalization, rather than just how well a model has adapted to a particular benchmark.

What stands out to me is that Beacon had $250k in funding lined up, yet Pip still had to cover expenses personally while the paperwork dragged on.

@Austin, have you seen a good way for fiscal sponsors to handle this kind of gap? Seems like something that could happen to a lot of early-stage nonprofits.

@NeelNanda I think the key issue is that the proposal conflates three different outcomes: expressing interest in AI safety, actually entering the field, and becoming a researcher at your level.

There's also a numerical mismatch. If 1% of 1,350 students pursue AI safety, that's 13.5 students, making the cost roughly $11.1k per prospective entrant, not $9,357 per researcher.

@aashkapatel, do you have any follow-up data from Phase 0 on actual participation or completion, beyond initial interest? That would make the Phase 1 projections much easier to evaluate.

Liron, one idea that might make Doom Debates even more valuable: what if every debate ended with each side stating exactly what evidence would change their mind?

You could publish those answers alongside the episode and revisit them later. It would give viewers a way to see whether the disagreement is actually being resolved over time, rather than just debated.

Have you considered trying something like this?

I read your piece on leverage claims, especially the coordinated procurement argument. You mentioned bilateral deals as a way for AI companies to break these coalitions.

I'm curious, which safeguard do you think would be more realistic to implement first: transparency around bilateral agreements or shared procurement rules?

@adeviye38 It’s a mix. ML is used on the landmark and face-model side of the pipeline, but Aura isn’t just asking a model to generate a plausible 3D face. The reconstruction is constrained by what the different photos actually agree on — things like multi-view geometry, depth, facial contours and metric scale.

The prior helps where the image evidence is weaker, but I don’t want geometry coming from a model to be presented as if it was directly measured. Aura keeps track of that distinction. If a reconstruction doesn’t pass the geometry and evidence checks, it can be rejected instead of just being shown anyway.

I also deliberately don’t treat generative completion of unseen parts of the face as a measurement. If the system can’t really observe a region, I’d rather have it say that than invent something that just looks convincing.

In our current internal 10-subject benchmark, the reconstruction pipeline achieved 0.29 mm median error against 3D reference scans. The next step is independent validation on a much larger dataset.

So the goal isn’t just to make a realistic-looking 3D face. It’s to make one that can tell you what came from the photos, what came from the model, and when it shouldn’t claim to know something.

@carol Thanks for flagging this. I understand the issue, the activity has stopped, and it won’t happen again. I’d appreciate it if the moderation note could be removed or updated to reflect that the issue has been resolved

@sibelzeynepumay Thank you very much. The key now is proving the accuracy independently.

@fattmaatilgan Thank you. That access gap is exactly what we’re trying to close.