3

Eliciting Probability Trees

Not fundedGrant
$0raised

Project summary

Continue building tools to elicit probability trees from groups, especially expert groups.

What are this project's goals and how will you achieve them?

As a forecaster, I pin down key factors and analyse complexity. But currently forecasting isn’t very useful for decision makers. Forecasts focus on the wrong outcomes and don’t consider the tractable inputs. Forecasts rarely seem to add usefully to discussion. And where they do, the feedback mechanism is slow. 

I want to make easy-to-use tools for building probability trees, to help people think about inputs, outputs and likely worlds, and to trial them with decision makers and forecasters. I want to be able to:

  • Map the problem space quantitatively

  • Capture tractable decisions and relevant outcomes

  • Understand and display expert models of cause and effect

  • Share simple visual models of these

  • Refine and develop methods

I will test the tools and process by approaching top EA decision makers and forecasters.  I will have succeeded when organisations pay a small fee for them, or users routinely express it was valuable for them. 

The tools are currently open source. 

Currently I work on:

  • Estimaker - a tool for quickly building probability trees. It has had 70 users, which has included at least 1 set of key decision makers. It is similar to Guesstimate but easier to move nodes around to change the framing of the problem. It has taken about 3 months of development so far, on and off. Currently working on a more flexible v2.0

  • Viewpoints - a tool for aggregating opinions from large groups. I have run 3 events with it and have more planned. Several people have noted its value so far. It is like Polis but easier to use and for me to extend. 

  • Odd and Ends podcast - I interview experts on important issues and the likelihood they happen. A few forecaster friends have complimented me on it.

Previously I have worked on:

  • Doubtful - a forecasting question generation platform. Users could collaboratively edit and then post to prediction markets. The editing process was too complex so users didn’t stick. I would like to try again with a simpler process. 

Projects I’m interested in exploring. I would perhaps use this grant on these too, but feedback suggests that giving the main project I'm interested in is more useful to people.

  • Wikis with forecasts - Currently wikis talk well about the past, but less so about the future. What are the key blockers for nerdy wikis to use forecasts as references for what will, won’t and might happen?

  • Better debates - People often want to have public debates, but they rarely create value for me. How can we have debates between experts that better inform listeners?

How will this funding be used?

A year of:

  • My time as product manager

  • 1 FTE Development lead

  • 1 FTE UX designer

  • 1-3 FTE Developer

As I have currently been doing, I will design and interview users for the projects. The above projects took about 3 months of development time and .3 FTE of my own time. I will aim produce 8x the above projects per year.

Who is on your team and what's your track record on similar projects?


Me (Nathan Young) - @NathanpmYoung

I have built estimaker and viewpoints in the last 6 months. I am confident that with more runway I can deliver growth on these or other products.  

Testimonials

Austin Chen - “[thanks to] Nathan, for trying out mechanisms like this market and viewpoints.xyz to facilitate good conversation on a heated topic”

Luke Freeman, ED, Giving What We Can “[referring to a viewpoints poll] I always love how quickly you turn things into polls. Find it pretty interesting to get a more granular and clustering view of what people thing that’s often not reflected by upvotes and comments.”

Marcus Buffet, after a user interview using Estimaker - “Knowing the value I got, I would have paid for this consultation"  

Funding

Viewpoints was initially funded by Vitalik Buterin

Estimaker and Doubtful were funded by FTXFF

Other details

I am a forecaster for the Swift Centre.

I co-founded the Coronavirus Tech Handbook. In the early days of Covid I launched a repository of resources that was viewed 600,000 times, connected teachers, doctors and engineers across the world. It has its own wikipedia page and was mentioned in 100 articles.

What are the most likely causes and outcomes if this project fails? (premortem)

There is some chance that forecasting and polling are a dead end and that there aren’t good tools to build in this space. People often like Viewpoints but I’m pretty uncertain if it is deeply useful

After that, I consider the largest chance is that for some reason just not much progress is made. Perhaps I fall very ill.

There is some chance that the US government tries to claw back the money I got from FTX and bankrupts me and takes this money in the process.

After that I think there is a small chance of very bad relationship breakdown with a dev team that writes off a chunk of work done, but this doesn’t seem very likely. I generally work on a month by month basis so even in the past relationship breakdown hasn’t written off work.

What other funding are you or your project getting?

Viewpoints was funded about a year ago for $110k by Vitalik Buterin. I have about half of this left, but I need more to work stably in the long term.

FTX money. I have $45k left, but it’s liable to be clawed back.  

I intend to apply to basically anywhere I can think of for individual projects and this more fluid funding

I'd appreciate feedback in the comments here or anonymously

NunoSempere avatar

Nuño Sempere

about 1 year ago

I have too many conflicts of interest to fund this myself, but here are some thoughts:

I like thinking of Nathan's work in terms of the running theme of helping communities arrive at better beliefs, collectively. And figuring out how to make that happen.

On the value of that line of work:

- I have a pretty strong aversion to doing that work myself. I think that it's difficult to do and requires a bunch of finesse and patience that I lack.

- I buy that it's potentially very valuable. Otherwise, you end with a Cassandra situation, where those who have the best models can't communicate them to others. Or you get top-down decisions, where a small group arrives an opinion and transmits it from on high. Or you get various more complex problems, where different people in a community have different perspectives on a topic, and they don't get integrated well.

- I think a bottleneck on my previous job, at the Quantified Uncertainty Research Institute, was to not take into account this social dimension and put too much emphasis on technical aspects.

One thing Nathan didn't mention is that estimaker, viewpoints and his podcast can feed on each other: e.g., he has interviewed a bunch of people and got them to make quantified models about AI using estimaker: (Katja Grace: https://www.youtube.com/watch?v=Zum2QTaByeo&list=PLAA8NhPG-VO_PnBm3EkxGYObLIMs4r2wZ&index=8, Rohit Krishnan: https://www.youtube.com/watch?v=cqCYMgEnP7E&list=PLAA8NhPG-VO_PnBm3EkxGYObLIMs4r2wZ&index=10, Garett Jones: https://www.youtube.com/watch?v=FSM94rmJUAU&list=PLAA8NhPG-VO_PnBm3EkxGYObLIMs4r2wZ&index=4, Aditya Prasad: https://www.youtube.com/watch?v=rwTb7VgSZKU&list=PLAA8NhPG-VO_PnBm3EkxGYObLIMs4r2wZ&index=6). This plausibly seems like a better way forward than the MIRI conversations https://www.lesswrong.com/s/n945eovrA3oDueqtq.

Generally, you could imagine an interesting loop: viewpoint elicitation surfaces disagreements => representatives of each faction make quantified models => some process explains the quantified models to a public => you do an adversarial collaboration on the quantified models, parametrizing unresolvable disagreements so that members of the public can input their values but otherwise reuse the model.

I see reason to be excited about epistemic social technology like that, and about having someone like Nathan figure things out in this space.

saulmunn avatar

Saul Munn

about 1 year ago

it'd be great to have a clear theory of change, if you have one — if you don't, that's okay, but if the goal is (e.g.) "get political researchers to use Estimaker in their research," then you might consider writing out a plan on how you intend to (e.g.) get political researchers to use Estimaker in their research.

tldr: these projects look cool; what is your concrete plan to turn it from "oh cool, this little app thingy" to "a bunch of {key decisionmakers, relevant academics, etc} are using this"?

also, smaller comments:

  • i just signed up for estimaker... but i have no idea how to use it? i'm just shown a snowflake, a percentage, and a blanking cursor for code. was there some tutorial, or onboarding process that i missed? i don't even know what language to use :( it'd be really helpful to at least have a documentation! (and if there is one & i couldn't find it in 2-3 minutes of looking, that makes it an easy problem to fix — just make it more prominent!)

  • on viewpoints, it'd be awesome if i could use keypresses instead of clicks (maybe up/down/left/right arrows?)

  • viewpoints sorta feels like i'm filling out a census report. it almost feels like a game, but there could be much more gamified elements that make it more fun to fill out; after 2-3 minutes, it gets to be pretty much exactly the same, over & over again.

good luck nathan! always here if you want or need help :D

NathanYoung avatar

Nathan Young

about 1 year ago

Thanks for the feedback.


What do you think about "What are this project's goals" section now?

On estimaker, thank you for the advice.