Fund compute, events, and open education materials for a volunteer community building better agent evaluation methods and checking if agent tests can be trusted
Agent evaluations and benchmarks guide a market reaching trillions of dollars, but most of the tests are done by for-profit companies or by people who sell the systems themselves. Almost no one has tested the rigor of these evaluations: their reliability, validity, accuracy, efficiency, and safety. A 2025 Oxford study reviewed 445 AI benchmarks and found that only 16% of them applied statistics to assess whether the difference between two models is statistically significant (you can roughly think of this as checking if the difference is real). EvalScience is a 501(c)(3) nonprofit that promotes scientific evaluation of agentic systems. We build the community, host events, and produce research to build open standards and education materials. Since our founding in August, over 60 researchers from Harvard, MIT, Princeton, Columbia, Cornell, NYU, Brown, Northeastern, UMass Amherst and across industry (Microsoft Research, Google, Google DeepMind) have been involved in a volunteer format.
We are calling for funding for our Mentored Projects and community. Mentored Projects currently consists of 12 projects, each led by professors and top PhD students with strong publication records. In this era, everyone does research and writes research-paper-like material with AI, so we believe guidance is very important, especially deep thinking about study design. The first round of this mentorship program has attracted over 10 mentors and 40+ mentees, spanning statistics, psychology, HCI, and engineering. Most of us do not have lab budgets and are doing this as volunteer work to advance the rigor of agent evaluation. We will use the funding to support the mentorship program with compute and potential publication costs.
We believe that in this field today, research and experience sharing with continuous and prompt feedback from experienced researchers is more important than conference papers. We are also trying to change the way people share knowledge and experience in this fast-moving field. We do not require teams to write their results and experience as conference papers. Rather, in week 12, we invite every team to share their results as notes on evalscience.org, advised by their mentors and reviewed by everyone else in the cohort (and potentially by our working groups too, separate groups that meet monthly in each city for research exchange, whose members include leaders in the field). All teams will share their results, successes or failures, at our symposium in a dedicated program. We do this to accumulate knowledge about agent evaluation, not to add more papers to the field.
We consider it successful if more than 2/3 of the funded teams publish notes. After each cohort, we will publish the notes and a summary on our website, and we will have a dedicated program at our biannual symposium to share the results.
EvalScience will review proposals and decide the amount to give each team. This will be a budget specifically for compute and potential publication costs. $15,000 will likely cover compute and API costs for the first cohort, with a budget of $1,500 for each project. We will likely end up with more than 10 projects (capped at 25 per cohort). At $40,000, you could also sponsor our 2027 spring in-person meeting and open material construction, which includes the notes and a course we are building as a team. Up to $50,000 would let us support more projects and cohorts and hire someone to help manage the program.
Yining Hua (Harvard PhD candidate) founded EvalScience and leads the organization as Executive Director. She has been working on evaluation since 2019, and her benchmarks and evaluations have been used in the model releases of 13 AI companies. Our team spans Harvard, MIT, Princeton, Columbia, Cornell, NYU, Brown, Northeastern, and UMass Amherst. Our team, board, and advisors are listed on our people page. More information and events can be found at evalscience.org.
We are on track with building open evaluation tools. For example, our healthcare agent evaluation group, the HEED consortium, has over 30 experts across agentic systems, healthcare deployment, AI governance, and more. Participants and ongoing work are not shared publicly but can be shared with regrantors on request.
Our first symposium is taking place in New York on 11/20/2026. We received over 200 submissions and accepted 77, covering 404 authors from 119 institutions, and we had 1,300+ registrations for 150 seats. Our Boston, Stanford, and New York working groups have roughly 50–60 active researchers.
(1) Volunteers run out of time; (2) lack of progress tracking. We have designed an easy way to share results and promised a dedicated program at our next symposium to feature the projects, so we think it is less likely for the projects to fail. We consider failed projects useful experience that should also be shared. We do not require a large quantity of work, only rigor and sharing. We interview participants and check in with each team monthly.
We are currently sponsored by Schmidt Sciences, Jane Street, and Fleet AI, and are in sponsorship discussions with 5+ more companies, which we prefer not to name at this point. We would be happy to provide more information and details if needed by regrantors. Please contact us at nin@evalscience.org.
There are no bids on this project.
Comments
No comments yet. Sign in to create one!