@Austin I decided against doing this in the end! I'm not super short of money right now so maybe it's better to spend it somewhere else - also after doing a deep dive into ToCs and literature I realized that well no one knows what to do when it comes to technical research. AI governanace and field building might be a different thing.

BTW what would be the organizations that you would be most interested in reading a cost effectiveness analysis on?

@Austin Hi! After some thinking and I decided against taking the money - I don't need funds to do it and it seems it can be spent in other places.

@RyanKidd Thanks! I agree, I might not be the best person to do this with my limited experience. I searched EA forum and LW and only found those CE estimates:
- CAIS
- arb's AISC estimate
- Nuno sempre work on long termist orgs
- MATS self assessment

Which is indeed limited. Given this I think it's a good idea to do this, even to just put out something and let other people work on this to improve this. Worst case scenario, this will end up being something along lines of "How good are AIS orgs at communicating their findings (measured by citations)"

To improve quality I will try to get feedback from some organizations and EAs.

My rough plan is to break down the theories of change that are pursued in AIS down into:

• Evals

• Field-building

• Governance

• Research

And measure impact in each category seprately.

Evals can be measured by quality and number of evals, relevance to ex-risks. It seems pretty straightforward to differentiate a bad eval org from a good eval org—engaging with major labs, having a lot of evals, and a relation to existential risks.

Field-building—having a lot of participants who do awesome things after the project.

Research—I argue that the number of citations is also a good proxy for the impact of a paper. It's definitely easy to measure and is related to how much engagement a paper received. In the absence of any work done to bring the paper to the attention of key decision makers, it's very related to the engagement.

I'm not sure how to think about governance.