ActiveGrant
$279,583raised
$1,000,000funding goal

Project summary

The ML Alignment & Theory Scholars (MATS) Program is an educational seminar and independent research program that aims to provide talented scholars with talks, workshops, and research mentorship in the field of AI alignment and connect them with the Berkeley AI safety research community.

MATS helps expand the talent pipeline for AI safety research by empowering scholars to work on AI safety at existing research teams, found new research teams, and pursue independent research. To this end, MATS connects scholars with research mentorship and funding, and provides a seminar program, office space, housing, research coaching, networking opportunities, community support, and logistical support to scholars. MATS supports mentors with logistics, advertising, applicant selection, and complementary scholar support systems, greatly reducing the barriers to research mentorship.

What are this project's goals and how will you achieve them?

  • Find + accelerate high-impact research scholars:

    • Pair scholars with research mentors via specialized mentor-generated selection questions;

    • Provide a thriving academic community for research collaboration, peer feedback, and social networking;

    • Develop scholars according to the “T-model of research” (depth/breadth/taste)

    • Offer opt-in curriculum elements, including seminars, research strategy workshops, 1-1 research coaching, peer study groups, and networking events.

  • Support high-impact research mentors:

    • Scholars are often good research assistants and future hires;

    • Scholars can offer substantive new critiques of alignment proposals;

    • Our operations and community free up valuable mentor time and increase scholar output.

  • Help parallelize high-impact AI alignment research:

    • Find, develop, and refer scholars with strong research ability, value alignment, and epistemics;

    • Use alumni for peer mentoring in later cohorts;

    • Update mentor list and curriculum as the needs of the field change.

How will this funding be used?

We are seeking general support funds for MATS, including future MATS cohorts, scholar stipends, scholar housing and accommodation, office rental, food, computing, events, travel, contractor labor, and payroll.

The most recent MATS cohort hosted 60 scholars and 15 mentors. It costs approximately $35k to fund one scholar through the entirety of the program (not including staff time spent on program elements).

Who is on your team and what's your track record on similar projects?

We are a small team of eight full-time staff members. An organization chart can be found here.

MATS has successfully run five cohorts over the last two years, including two extension programs. We have successfully scaled our twice-yearly program from 30 scholars and 5 mentors to 60 scholars and 15 mentors. Past mentors listed here.

Recent success stories:

What are the most likely causes and outcomes if this project fails? (premortem)

  • Poor selection of scholars would waste mentor time + grant money. How we address this:

    • We defer to best-in-class research mentors for their applicant selection needs rather than attempting to select applicants ourselves and pairing them with mentors after the fact.

    • We use difficult, mentor-specific applicant selection questions so there is sufficiently high granularity distinguishing between top scholars.

    • We choose mentors with a proven track record of high-quality research or strong endorsements from the alignment research community.

  • Some mentors are less engaged or experienced than others, and the mentorship experience might be less helpful for their scholars. How we address this:

    • We soft cap the number of scholars in streams with low mentor engagement unless the applicants seem particularly self-directed.

    • Scholar support staff supplement the mentorship experience by providing 1-1 research strategy and unblocking support for scholars.

    • We hold workshops on research strategy, technical writing, research tools, and more to benefit scholars and offload time mentors might otherwise spend teaching these skills.

  • There might not be enough jobs or funding for all alumni to receive financial support for their research efforts (sometimes known as the “mass movement building” concern). How we address this:

    • Some of our alumni’s projects are attracting funding and hiring further researchers. Our alumni have started alignment teams/organizations that absorb further talent (listed above).

    • With the elevated interest in AI and alignment, we expect more organizations and funders to enter the ecosystem. We believe it is important to install competent, aligned safety researchers at new organizations early, and our program is positioned to help capture and upskill interested talent.

    • Sometimes, it is hard to distinguish truly promising researchers in two months, hence our four-month extension program. We likely provide more benefits through accelerating researchers than can be seen in the immediate hiring of alumni.

    • Alumni who return to academia or industry are still a success for the program if they do more alignment-relevant work or acquire skills for later hiring into alignment roles.

  • Scholars might overly defer to their mentors and fail to critically analyze important assumptions, decreasing the average epistemic integrity of the field. How we address this:

    • Our scholars are encouraged to “own” their research project and not unnecessarily defer to their mentor or other “experts.” Scholars have far more contact with their peers than mentors, which encourages an atmosphere of examining assumptions and absorbing diverse models from divergent streams. Several scholars have switched mentors during the program when their research interests diverged.

    • We require scholars to submit “Scholar Research Plans” one month into the in-person phase of the program, detailing a threat model they are targeting, their research project’s theory of change, and a concrete plan of action (including planned outputs and deadlines).

    • We encourage an atmosphere of friendly disagreement, curiosity, and academic rigor at our office, seminars, workshops, and networking events. Including a diverse portfolio of alignment agendas and researchers from a variety of backgrounds allows for earnest disagreement among the cohort. We tell scholars to “download but don’t defer” in regard to their mentors’ models. Our seminar program includes diverse and opposing alignment viewpoints. While scholars generally share an office room with their research stream, we split streams up for accommodation and discussion groups to encourage intermingling between streams.

What other funding are you or your project getting?

We generally seek funding from multiple sources, including Open Philanthropy, and Survival and Flourishing Fund. We recently received a grant from Open Philanthropy supporting MATS runway costs for 12 months. This does not include the cost of running another cohort.

RyanKidd avatar

Ryan Kidd

2 months ago

Update: mentors for the Winter 2024-25 Program released! Currently fundraising for the Summer 2025 Program; 48 mentors have already expressed interest!

Thank you to Manifund for collectively supporting ~11 additional research scholars with your collective donations so far! It's very important for us at MATS to continue seeking funding from diverse sources, which lets us run larger programs with further research diversity. We will continue to do our best to turn your donations into impact!

donated $10
michaeltrazzi avatar

A lot of people I interact with regularly have done the MATS program and received a lot of value from it.

Similar to Neel below I want to give a token of appreciation, voting for the quadratic funding, given that MATS is a large funding opportunity.

RyanKidd avatar

Ryan Kidd

3 months ago

Update: we have assembled a team of expert advisors to select mentors for the Winter 2024-25 Program, as we received 78 mentor applicants! Our mentor pool has never been more exciting!

donated $50
NeelNanda avatar

Neel Nanda

3 months ago

I've found being a MATS mentor a very valuable experience! I think my scholars have done kick-ass work, several of them have had very high impact roles going forwards, and I've mentored many more people than I would have done on my own in a way that I believe has significantly magnified my and their impact, and I appreciate MATS for facilitating this.

I'm not donating more, as MATS is a large funding opportunity that I don't think EACC is well placed for, but a token donation seems in the spirit.

RyanKidd avatar

Ryan Kidd

6 months ago

Update: we recently published a blog post summarizing our takes on talent needs of technical AI safety teams based on 31 interviews with key figures in AI safety, including senior researchers, organization leaders, social scientists, strategists, funders, and policy experts.

RyanKidd avatar

Ryan Kidd

6 months ago

Update: we recently published our Winter 2024-25 Retrospective: https://www.lesswrong.com/posts/Z87fSrxQb4yLXKcTk/mats-winter-2023-24-retrospective

RyanKidd avatar

Ryan Kidd

6 months ago

@RyanKidd Oops, I meant Winter 2023-24

RyanKidd avatar

Ryan Kidd

7 months ago

Update: MATS is no longer in need of additional funding for our Summer 2024 Program. We are still accepting donations towards our Winter 2024-25 Program, however!

donated $1,040
🥭

Philip Gubbins

7 months ago

AE Studio Survey of EA/Alignment communities, 26 respondents chose to donate $40 to MATS

Austin avatar

Austin Chen

7 months ago

@phgubbins That's awesome -- thanks to all the participants of the survey, and to you for facilitating this!

RyanKidd avatar

Ryan Kidd

7 months ago

Update: thanks to your donations, we were able to support an additional 8.5 scholars in the Winter 2023-24 Program, at an ex post cost of $22.4k/scholar! Thank you so much for your contributions to the field of AI safety :)

We are currently fundraising for our Summer 2024 Program and again expect to receive less funding than our ideal program. We can support marginal scholars at a cost of $24.4k/scholar. We currently have 1220 applicants for Summer 2024 and expect to accept ~3-5% (i.e., MIT's admissions rate). Given the high calibre of applicants and mentors, we would love further funding to support additional scholars!

We have announced the following mentors and hope to announce more as we confirm additional funding: https://docs.google.com/document/d/1sDnD9Igr3gkWX-N_l9W8itVBpqx-pChh-61atxGYkPc/edit

🥦

Craig Falls

8 months ago

Is this a 501c3? Is there a way to donate from a DAF, or do I have to use a credit card?

Austin avatar

Austin Chen

8 months ago

@cfalls Hi Craig! I'm not sure that MATS is a 501c3, but Manifund is, and we're able to accept DAF payments on their behalf and forward the funds to them. (We do ask for DAF payments to clear a $5k minimum donation, however, due to the operational cost of processing these.) If you're interested, see instructions here.

🥦

Craig Falls

8 months ago

@Austin I eventually figured out I could send from my DAF to BERI and mention MATS. Good to know Manifund would also work, but then I think I have to wait for the check to clear and then do another round of sending, like basically Manifund is itself similar to a DAF?

Austin avatar

Austin Chen

8 months ago

@cfalls haha yeah, for cases like this we function as a DAF ourselves (albeit one that primarily supports EA and adjacent projects instead of "all other 501c3s"; provides fiscal sponsorship for individuals/orgs like MATS; and also hosts programs like ACX Grants).

I wasn't aware that BERI would also facilitate this for MATS, good to know!

donated $150,000
trishume avatar

Tristan Hume

about 1 year ago

I've been very impressed with the MATS program. Lots of impressive people have gotten into and connected through their program and when I've visited I've been impressed with the caliber of people I met.

An example is Marius Hobbhahn doing interpretability research during MATS that helped inform the Anthropic interpretability team's strategy, and then Marius going on to co-found Apollo.

RyanKidd avatar

Ryan Kidd

about 1 year ago

@trishume Woohoo! We can support an additional ~7 scholars with this grant, based on the updated marginal cost of $21k/scholar.

RyanKidd avatar

Ryan Kidd

about 1 year ago

Update: we don't appear to be funding constrained for Winter, but will continue accepting donations for our Summer 2024 Program!

RyanKidd avatar

Ryan Kidd

about 1 year ago

Update update: Several more awesome mentors have come forward and we now are funding constrained again for Winter!