2

Pilot for new benchmark by Epoch AI

🦑

Epoch AI

ActiveGrant
$200,000raised
$200,000funding goal
Fully funded and not currently accepting donations.

For confidentiality reasons, keeping this brief—more available to Manifund on request.

This is for the pilot of a new benchmark that I think could be very valuable for tracking and predicting future AI progress to AGI. Funding will go towards API costs, labelers, software engineer and part of researcher salaries.

This is a pilot for a potentially larger effort. I will be spending some of my time helping mentor the pilot.

Austin avatar

Austin Chen

28 days ago

Approving this grant! While this writeup is somewhat sparser than we'd prefer, Epoch doesn't want to be scooped on their work, which seems reasonable; they should be able to post more publicly once the benchmark is released (Leopold says "maybe spring for the pilot, more like summer for the full thing").

In any case, Epoch has done some of the best work in the field on benchmarking and visualizing AI trends; we'd be happy to support unrestricted grants to their org anyways. (And as usual, as part of our regranting model, we extend considerable deference to our AI safety regrantors).

donated $200,000
LeopoldAschenbrenner avatar

I think Epoch has done truly outstanding work on core trends in AI progress in the past few years. I'm also excited by their recent foray into benchmarking in the form of FrontierMath. I think highly of core team members involved in the project. I found our initial discussions about this project very promising.

Better benchmarks that help us forecast time to AGI (and especially time to relevant capabilities, such as automated AI research) and do so in a highly credible and scientific way are very valuable for informing policymakers and catalyzing important policy efforts.

Donor's main reservations

It's a pilot, it might not work.

Epoch has other funding—but not for this effort, and benchmarking is especially expensive (API calls, labelers).

Process for deciding amount

I reviewed a proposed budget. (Confidential, more on request from Manifund.)

Conflicts of interest

Please disclose e.g. any romantic, professional, financial, housemate, or familial relationships you have with the grant recipient(s).

No COIs.