AdamGleave avatar
Adam Gleave

@AdamGleave

regrantor

CEO & co-founder at FAR AI, a trustworthy AI non-profit; PhD AI UC Berkeley 2022; LTFF fund manager 2020-2022

https://www.gleave.me/

Donate

This is a donation to this user's regranting budget, which is not withdrawable.

Sign in to donate
$122,898total balance
$122,898charity balance
$0cash balance

$0 in pending offers

About Me

I am the CEO and co-founder of FAR AI, an alignment research non-profit working to incubate and accelerate new alignment research agendas. I received my PhD from UC Berkeley under the supervision of Stuart Russell. I was fortunate to be part of the Center for Human-Compatible AI during my PhD and funded by an Open Philanthropy Fellowship. My grantmaking experience includes being a Fund Manager at the Long-Term Future Fund between Feb 2020 and Jan 2022. Please see my CV for a more comprehensive list of my prior experience.

Outgoing donations

Comments

AdamGleave avatar

Adam Gleave

3 months ago

Alex Lintz has a track record of good work in this space, and took a career risk in order to start a potentially high-impact organization. Although this particular org didn't pan out, this is par for the course in entrepeneurship. I'm excited for Alex to have more shots on goal.

I would not in general offer retroactive funding for activities of this nature -- I'd encourage people to either secure sufficient personal runway to derisk this, or secure funding before starting a new org -- but Alex's case is a bit unusual as he had a good-faith reason to expect to receive funding for this it just fell through for (after having heard the specifics of the case) somewhat arbitrary reasons out of his control.

AdamGleave avatar

Adam Gleave

3 months ago

Donated as this seems like a very leveraged grant. In-person interaction is important when starting work on a new research agenda, and this effectively gets 3 months * 2 people = 6 person-months more of that for a relatively low cost.

Damian and Pietro have relevant experience and I expect to execute well on this project.

My main hesitation is I feel skeptical of the research direction they will be working on (theoretical work to support the AI Scientist agenda). I'm both unconvinced of the tractability of the ambitious versions of it, and more tractable work like the team's previous preprint on Bayesian oracles is theoretically neat but feels like brushing the hard parts of the safety problem under the rug (into e.g. the safety specification). However, enough people are excited by this direction that I feel inclined to support high-leveraged exploratory work in this area to see if the agenda can be refined.

AdamGleave avatar

Adam Gleave

3 months ago

@AdamGleave Full-disclosure, I have two small COIs here:

  1. I'm an informal advisor of Timaeus, which consists of me having some advising meetings with team members on a ~once/month basis, and I'm e.g. listed on their website. However, I do not have any hard power: explicitly I am not a board member, and no one at Timaeus reports to me.

  2. Jesse, the Timaeus founder, works from my organizations co-working space FAR.Labs. If Timaeus expanded and made in-person hires in the Bay it is likely they would also work from FAR.Labs. So FAR.AI might get a small amount of revenue from Timaeus expansion.

AdamGleave avatar

Adam Gleave

3 months ago

I've generally been impressed by how well Timaeus have executed. They've in short order assembled a strong team who are collaborating & working well together, producing substantial research and outreach outputs. They have a distinctive research vision, and I think deserve some credit for popularizing studying the evolution of networks throughout training from an interpretability perspective with e.g. EleutherAI's interpretability team now pursuing their own "development interpretability" flavored research.

I have not had time to conduct an in-depth evaluation of Timaeus's research, so would not want people to defer to me on this point. In particular I don't currently have a good enough understanding of Singular Learning Theory to judge the tractability of this approach. I do have some background skepticism of this category of approaches (having seen a number of theoretically nice approximations like neural tangent kernel not actually have that much explanatory power) that would make me have a higher burden of evidence to buy-into this approach, however if it is a dead-end I would expect Timaeus to eventually recognize that & be able to pivot to ohter valuable directions.

AdamGleave avatar

Adam Gleave

5 months ago

This is a cheap way to both enable a promising early-care researcher to expand their network and gain feedback on their work, and disseminate safety research findings to a relevant audience.

Usually I think costs for things like this should fall on the person’s employer but there were some bureaucratic issues that made this a non-starter in this instance.

AdamGleave avatar

Adam Gleave

7 months ago

Main points in favor of this grant

Good paper that would be exciting to disseminate. This is a very cheap way of supporting that.

Donor's main reservations

The paper is very theoretical in nature and may not end up improving AI safety in practice.

Process for deciding amount

I asked Joar for an estimate of the travel costs, checked they were reasonable, and then added my own estimate for costs for food/sundries in Vienna.

Conflicts of interest

Please disclose e.g. any romantic, professional, financial, housemate, or familial relationships you have with the grant recipient(s).

I previously supervised Joar Skalse on a related project. Joar is currently a contractor working on a separate project for my organization, FAR AI.

AdamGleave avatar

Adam Gleave

11 months ago

Main points in favor of this grant

  • Promising research idea; "obvious next step" but not one that anyone else seems to be working on.

  • Can Rager has relevant research experience.

  • David Bau's lab is a recognized name in the field and a competent collaborator.

Donor's main reservations

  • Limited track record from Can.

  • Research project is high-risk, high-reward.

Process for deciding amount

  • $6000-$9000/month seems to be around the going rate for junior independent research based on previous LTFF grants. I went on the higher end as: (a) stipend may need to pay for office expenses not just living expenses; (b) Can intends to be based in the Bay Area for some of this time, a high cost-of-living location.

Conflicts of interest

Can may spend some of his stipend on a desk & membership in FAR Labs, an AI safety co-working space administered by the non-profit FAR AI that I am the founder and CEO of. This is not a condition of this grant, and I have encouraged Can to explore other office options as well. I do not directly benefit financially from additional members at FAR Labs, nor would one member materially change FAR AI's financial position. No other conflicts of interest.


AdamGleave avatar

Adam Gleave

over 1 year ago

Typo: salary is $91,260 annualized not $92,260.

AdamGleave avatar

Adam Gleave

over 1 year ago

Main points in favor of this grant

There's been an explosion of interest in Singular Learning Theory lately in the alignment community, and good introductory resources could save people a lot of time. A scholarly literature review also has the benefit of making this area more accessible to the ML research community more broadly. Matthew seems well placed to conduct this, having already familiarized himself with the field during his MS thesis and collected a database of papers. He also has extensive teaching experience and experience writing publications aimed at the ML research community.

Donor's main reservations

I'm unsure how useful Singular Learning Theory is going to be for alignment. I'm most unsure whether it'll actually deliver on the promise of better understanding deep networks. The positive case is that traditional statistical learning theory has some serious limitations, making predictions that contradict empirical results on deep networks, so we need some replacement. But grandiose theories pop up now and again (the neural tangent kernel was hot last year, for example) yet rarely pan out. Singular learning theory has been around for several decades, so that it only recently gained popularity in ML should also give some pause for thought. It seems plausible enough and enough people are excited by it what I'm willing to give it a shot for a relatively small grant like this, but this grant is definitely not me endorsing singular learning theory -- I'd need to understand it a lot better to really give an inside-view evaluation.

Conditional on singular learning theory actually enabling deeper understanding of neural networks, there's still a question of it that's actually useful for alignment. I feel reasonably confident that it would be a positive development: generally having theoretical frameworks to engage with (even if approximate) seems a key component of engineering systems with strong guarantees. Whereas just making something that works well most of the time is much more tractable via a trial-and-error approach. So, understanding seems to differentially help building reliable systems than just systems that mostly work. But, understanding does accelerate both -- so there is a non-trivial backfire risk.

Process for deciding amount

Fully funded Matthew's ask, which amounts to $92,260/year annualized. The salary seems reasonable given his experience level. It's higher than US PhD stipends (~50k/year), but below that of most alignment research non-profits in the SF Bay Area (LCA filings from Redwood show at least $140k/year for an ML Researcher; FAR AI's pay scale is $80k-$175k/year for Research Engineers) and significantly below for-profit tech jobs. Matthew will be working from Australia where tech salaries are lower; Levels.fyi gives a median of $54k/year USD total comp, but short-term contractor positions are often up to 2x that of salaried employees, so I still consider the ask to be in a reasonable range.

Not directly relevant in this grant, but I generally would advocate for independently conducted research to receive lower compensation than at alignment organizations, as I usually expect people to be significantly more productive in an organization where they can receive mentorship (and many of these organizations are at least partially funding constrained).

Conflicts of interest

Please disclose e.g. any romantic, professional, financial, housemate, or familial relationships you have with the grant recipient(s).

I supervised Matthew for an internship in 2021 at CHAI; I have continued collaborating with him (although relatively light-touch) to see that project through to publication.


Transactions

ForDateTypeAmount
Funding for AI safety comms strategy & career transition support3 months agoproject donation38952
Relocating to Montreal to work full time on AI safety3 months agoproject donation10000
Next Steps in Developmental Interpretability3 months agoproject donation50000
Travel grant to present AI safety paper at ACM FAccT5 months agoproject donation1650
Travel funding for International Conference on Learning Representations6 months agoproject donation1500
Manifund Bank7 months agodeposit+225000
Manifund Bank7 months agoreturn bank funds364470
Automatic circuit discovery on sparse autoencoded features10 months agoproject donation25000
Introductory resources for Singular Learning Theoryover 1 year agoproject donation10530
Manifund Bankover 1 year agodeposit+400000