4

Wasabipesto's Umbrella Project

$3,500raised
$3,500valuation
Sign in to trade

Overall Summary

This is an umbrella proposal for six different projects that I'm working on throughout various stages. A few of these projects have already been completed to some extent, while others are still in ideation. The purpose of this proposal is to secure funding that can allow me to set aside time to work on one or more of these projects, and in some cases to pay direct material or service costs.

Over the proposal period I will be collecting ideas and feedback on all of these projects, and I will then make a decision on which project(s) to move forward with first. Feel free to get in touch to leave comments about each project; the more details or concerns we had out now the better chance of success each project has. You can bet on which I will pursue at https://manifold.markets/wasabipesto/if-i-choose-to-work-on-x-manifund-p. Note that I will weight input by investors more heavily but reserve the right to work on any project.

My project pitches are as follows, roughly in order of how useful or valuable I expect them to be:

  1. Upgrade Calibration City: Add new features and connect to other sites for direct accuracy comparison

  2. Physical League Rewards: Get a brass Foldy in the mail when you reach Master's League

  3. Loan Exchange: Offer or find loans at a specific rate and timeframe, backed by the house

  4. Market Stories: View the history of a market as a vertical story, highlighting important events

  5. Brag Board: Show off your profit, calibration, league stats, and best trades in an easily-sharable card

  6. Notification Bridge: Get your comments and DMs sent to you via Telegram, Discord, or SMS while you're away

1. Upgrade Calibration City

Calibration City (https://calibration.city/manifold) is a project I started in July that builds calibration plots and calculates Brier scores across Manifold. It features advanced filters, market binning methods, and the ability to weight markets by specific criteria. It's useful to check Manifold’s accuracy, not just site-wide but in the specific conditions that are relevant to you.

These features are nice, but my original goal with Calibration City was to be able to check calibration across multiple prediction sites. Different platforms have different userbases and specialties, and a single Brier score can’t capture all of the information about each site. If we expand our scope to other platforms we can look for patterns and trends between each and across prediction markets as a whole. Most popular prediction platforms have open APIs that we can use to get this information and collate it in a single place.

What are this project's goals and how will you achieve them?

My primary goal with this project will be to expand Calibration City into a single source where users can find hard, objective data about prediction market accuracy across the industry. Some questions we should be able to answer:

  • I'm interested in baseball, how good is Manifold at predicting the outcomes a week in advance? Do other sites have a better track record?

  • I’m worried about the accuracy of markets that won’t resolve for a long time. What is the typical accuracy of a market over a year away from resolution?

  • This PredictIt market is trading at 90¢ but has less than 2000 shares in volume. How often does a market like that end up being wrong?

The basic structure already exists for Manifold: we poll the site’s API and look for closed markets, then gather data about them and save them to a database. The user can then query this database and build a calibration plot or get a Brier score to answer their question. For each site to add we will have to adapt to their specific dataset or adopt Metaforcast’s API in order to leverage their work.

Other potential upgrades include:

  • New X-Bin Method: 30/90/365 days before resolution

  • New Y-Axis Weighting Method: Weight by market open length

  • Put query parameters in URL for easy bookmarking/sharing

  • Cache output from default settings to make first load instant

  • Periodically refresh cache to catch re-resolutions

  • Provide a list of markets in the calibration sample

  • Add documentation to the main landing page and tooltips to each query field

  • Rewrite the backend in rust to increase performance

  • Change the database from SQLite to Postgres to increase performance

  • Serve the application from Fly for high-availability

  • Expose a public API for other third-party tools

  • Find a way to include numeric and free-response markets

  • Add additional scoring metrics or grade-levels

How will this funding be used?

Aside from compensation for my time, this project may incur the following direct costs:

  • Calibration.city domain renewal ($17/yr)

  • Hosting costs from Fly for high-availability (est. $20/mo)

What are the most likely causes and outcomes if this project fails? (premortem)

There are two main failure states for this project:

  • I cannot complete the project as described for some reason. This could be to some issue with integrating other sites’ APIs, an inability to save items to the database in a reasonable amount of time, or some other technical hurdle.

  • I complete the project as described but there is no impact to Manifold. This could be because people do not use the site, it does not bring publicity to Manifold, or it paints Manifold in a poor light.

I think the former is unlikely, especially after discovering the Metaforcast project. Since a framework already exists I believe we have a good foundation to build on. The main hurdle in my mind will be usability. If a user can’t intuit what is going on or doesn’t have the background to understand the terminology, they won’t stick around to learn more. We will need to have a focus on documentation - explaining what these things mean and why you can trust them - and usability.

Open questions

  • What sites should we prioritize integrating?

  • Should we integrate APIs directly or use the Metaforcast project?

  • Can we automatically link market categories or topics across sites?

  • How can we make the site more user-friendly or intuitive?

2. Physical League Rewards

The existing league system on Manifold leverages both mana rewards, which can be exchanged for charity contributions, and reputation rewards, shown on a user’s profile page. The top users on Manifold already have plenty of mana (by definition), and the reputation is nice but does not feature prominently on the site. In order to encourage more competition getting into the top leagues and to provide a more tangible reward, we can make brass trophies in the shape of Foldy, the site’s mascot and logo, and ship them to each person the first time they enter Master’s League.

Conceptually this is the simplest project on the list. I have already produced a handful of Foldies for myself and others and they turned out great. Each trophy is laser-cut sheet metal and all materials have a professional finish. I work with a specialty sheet metal shop that has a quick turnaround time and discounts for bulk orders. Sheet metal is ideal because it is easy to produce and ship while also being physically impressive.

You can see some Foldies from the previous batch here: https://imgur.com/a/4gBf6YX

What are this project's goals and how will you achieve them?

The first step of this project is to ship a physical reward to all users who make it into or retain their place in the December Master’s League. This gives the top participants a physical reward for their exceptional performance and rewards their participation. My steps to accomplish this are:

  • Potentially revise the Foldy design to make it more distinctive

  • Order a test run with varying thicknesses and sizes

    • Estimated prototype cost: $75

  • Test that the selected version fits well in the standard flat rate envelope and does not bend during shipping

  • Contact all members of the December Master’s League and collect addresses if they are interested

  • Order the trophies and shipping materials

    • Estimated material cost: $500-1000

    • Estimated shipping cost: $600-800

  • Package the trophies and ship to the winners

I am focusing on fewer, larger rewards due to the high upfront cost of shipping. Larger, higher-quality rewards will be more cost effective and be more likely to be displayed by the recipient.

I believe the best version of this project allows us to continue shipping trophies to all users who earn a spot in Master’s League, even as time goes on. If this month’s project succeeds I will approach Manifold with a partnership to continue this each month indefinitely. If we have extra stock I would also like to send a trophy to members who earned a spot in Master’s in the past.

I don’t plan on making the bronze trophies available for sale to keep them exclusive to the Master’s League. However, I could make other versions available for purchase.

How will this funding be used?

There are 62 users in the November Master’s League. Assuming there will be a similar amount of people in the December league, and assuming that 90% of them want a trophy, the material cost will be between $500-$1000. The main variables here are the size of the trophy and the thickness of the metal. In order to make sure we get the best value, I will order a prototype run with a few variants and let users comment on which they would prefer.

I plan to ship via the USPS Priority Mail flat rate small envelope, which costs $8 per domestic ship. International is much more expensive, running $29 per envelope. Costs will be much lower if we decide to exclude international users, but I would like to include them if possible. Assuming ~20% of users are international, this brings the average shipping costs to around $650-700 total.

What are the most likely causes and outcomes if this project fails? (premortem)

The intended effect of this project is to incentivize users to progress in leagues and reward users who have demonstrated their performance. If the users are not interested in physical rewards at all, this will not be an incentive. If they do not like the Foldy design itself, or find it too corny, it may not be a useful reward. The best outcome is that their trophy is displayed prominently and reminds them about their success on Manifold.

If I cannot raise enough funding to cover a significant portion the initial material and shipping costs, I will not attempt this project.

Open questions

  • What changes should we make to the Foldy design, if any, to make it more appealing as a trophy?

  • Is there a cheaper way to ship to international users?

3. Loan Exchange

Many users rely on interpersonal loans in order to increase leverage on specific markets, while users with large balances can lend to others at a specific interest rate to passively supplement their normal trading profit. Loans are a healthy part of the Manifold economy but currently have a few challenges:

  1. Borrowers have no standard place to look for loans

  2. Borrowers don’t know the standard loan rates

  3. Lenders have to manage their investments manually

  4. Lenders have to vet each potential borrower

  5. Lenders take all risk from defaulted loans

What are this project's goals and how will you achieve them?

The primary goal of this project is to create a centralized loan exchange website for users to offer or accept loans with specific criteria.

  • A lender can create listings offering X mana at Y% interest rate for up to Z weeks.

    • The site verifies via Manifold’s API that the user has a sufficient balance.

  • A borrower can search for loans with the criteria they choose.

  • A borrower can easily place an “order” on the site.

    • The borrower can see the exact amount they will receive and the total principal + interest they will owe after the specified timeframe.

    • The borrower can optionally offer a message with a description of what they plan to use the loan for.

  • The lender can approve or deny the loan.

    • They will be able to see the borrower’s account age, balance, positions, and other outstanding loans.

  • The loan principal is disbursed from the lender to the borrower.

    • Option A: The lender sends the mana to the borrower directly within Manifold’s UI.

    • Option B: The mana is automatically sent from the lender’s account with an API key.

  • Each user can see outstanding loans and any overdue items on a central dashboard.

    • Lenders can see each borrower’s current balance and recent trades.

  • The loan principal and interest are returned from the borrower to the lender.

    • Option A: The borrower returns the mana to the lender directly. They will be notified from the site when the payment is due.

    • Option B: The mana is automatically sent from the borrower’s account with an API key. The borrower and lender are optionally notified.

This sort of site would resolve items #1-3 in the problem statement above. Borrowers would be able to find dozens of loan offers at competitive rates and easily take them. Lenders can have a centralized dashboard where they can manage outstanding loans and keep an eye on their borrowers’ balances.

Unfortunately, this still leaves the issue of lender risk. Users can apply for huge loans without understanding the risk involved, or use it to bet on a market that does not go their way. We have a few ideas for an advanced version of this site that would alleviate this risk.

  • Market Collateral: Borrowers have to provide an API key in order to take loans, and a certain amount of collateral is automatically put into a holding market which is closed when the loan is made. If the borrower defaults, the collateral is automatically taken instead.

  • Site-Backed Security: Borrowers are evaluated based on a number of factors, including account age, past loan repayment, and outstanding loans. When a lender is given the prompt to approve or deny the loan, they will also be shown a security value (up to $50,000 per loan). If the user defaults, the lender can request reimbursement up to the established security value to be paid by the exchange.

  • Site-Owned Funds: The exchange fully owns and operates its own fund which lenders can contribute to and borrowers can borrow from. Lenders get a lower, but guaranteed, interest rate can can withdraw their money at any point. Borrowers can be automatically approved for loans up to a certain amount. If borrowers default, they are blacklisted from the fund.

How will this funding be used?

Some amount of funds will likely be used to purchase mana to cover defaulted loans in all but the simplest version of this site. We would take steps to minimize risk but part of the purpose of the exchange is to take risk away from lenders so there will always be some cost involved.

What are the most likely causes and outcomes if this project fails? (premortem)

This project is more susceptible than most to failure because it relies partly on a network effect. It would be, to my knowledge, the first central loan exchange for Manifold, but that does not guarantee success. If a competitor were to offer a similar solution, it could split the userbase and make the site less useful. It could also be that demand for loans are simply not high enough to warrant a centralized exchange.

Additionally, there is inherent risk if we assume responsibility for loan defaulting. User fraud could overrun the budget and we would either have to invest additional money, default to the lenders, or shut down operations. We will have to monitor the costs associated with guaranteeing any loans and user behavior affected by it.

Open questions

  • What strategy should we use to reduce risk to lenders? Market collateral, site-backed security, or site-owned funds, or a combination? Any other ideas?

  • If we algorithmically evaluate borrowers’ credit, what factors should we consider and how should we weight them?

4. Market Stories

Every market tells a story, if you know how to read it. Unfortunately the probability graphs on Manifold are completely separate from both the bets and comments so it can be quite difficult to piece together what actually happened a lot of the time.

  • Who spiked the price here? Were they responding to news or just betting wildly?

  • The chart didn’t move very much here. Was it because nobody bet much or because the liquidity was too high?

What are this project's goals and how will you achieve them?

This project would take be a browser client-side app that gets all of the data for a market and renders a dynamic scrolling page that highlights important events on top of the graph. It would allow users to understand what happened in one interesting and detailed summary. Users would also be able to zoom and expand the chart in order to investigate certain events.

  • The top of the page would have the market title and some basic information. The basic layout of the page would have the market graph rendered vertically so the user scrolls down “through” the market.

  • Long periods of inactivity would be collapsed and time markers would show the date and time at specific points.

  • Large bets or orders would call out the user who made it and any comments made around that time.

  • Comments with several likes will also be shown or summarized. Other comments will have a “pip” that can be clicked to expand the comment.

  • Market elasticity would be shown as offset lines to show an approximate “confidence interval”. Large limit orders would also be shown on the graph for the duration of the order.

  • The bottom of the page would show the market resolution and top winners/losers on each site.

Inspiration here includes the Soundcloud interface, which shows comments across the length of the track, or collaborative annotation tools that show comments in the margins like PaperHive.

How will this funding be used?

Aside from labor costs, I may decide to integrate some sort of LLM for summarization. I may also decide to purchase a new domain or cloud hosting for the site, depending on that traffic I receive.

What are the most likely causes and outcomes if this project fails? (premortem)

The main failure state for this project is simply that it may not hold users’ attention or make an impression. I assert in the tagline that every market is a story but not that every story is interesting. If users scroll through a few markets but don’t learn anything, then they probably won’t come back to the site for something like LK-99.

Another potential problem is over-optimizing for desktop or mobile usage. Display requirements for mobile will be very different from desktop, especially with regards to text and comments. If mobile users cannot see all of the same information, they may assume it does not exist.

Open questions

  • How do we evaluate what events to deem important?

  • Do we err on the side of more detail or less?

5. Brag Board

Manifold users put tons of time and effort into their trades in an effort to become better forecasters, earn money for charity, and win bragging rights within the community. The profile page has a lot of good information, but it’s not well-suited for quick social sharing. We can create a page that shows all of the things a user would brag about in a simple stat card that’s easy to share and understand.

What are this project's goals and how will you achieve them?

The primary goal of this project is an embed card, dynamically generated and easily embedded on Twitter/X, Discord, or Manifold itself. It is geared towards people who don’t necessarily know much about prediction markets but are able to intelligently interpret dense information. It would include stats like:

  • Total & weekly profit

  • Best recent trades

  • Calibration & letter grade

  • League and placement

  • Amount donated to charity

As a secondary goal, we can also make a larger dashboard/landing page with more detailed information. Suggestions on what to include here are welcome.

Inspiration for this idea includes Statbot’s profile cards, which show a similar about of information at a glance based on a user’s Discord history (https://statbot.net/img/landing-bot.webp).

How will this funding be used?

The only direct cost to this project would be my time, and potentially a new domain.

What are the most likely causes and outcomes if this project fails? (premortem)

The most likely failure state here is just a lack of adoption. If people are not interested in cards like this, then it will have minimal impact for Manifold as a whole. Additionally, Manifold will continue to refine the user profile page to make it more user-friendly and may obviate the need for a project such as this.

Open questions

  • What items are most important to have on the card? What should be emphasized?

  • Should we emphasize the most flattering stats? Should we hide items that are not particularly flattering? i.e. a user has not donated to charity or is in bronze league?

6. Notification Bridge

There are a lot of important things going on in Manifold, and it’s hard to keep track of everything. This is especially difficult if you do not use the app and only see events when you log in to the website or your email. Bringing notifications to users in their chat app of choice allows them to stay on top of market events and log back in when something interesting happens.

What are this project's goals and how will you achieve them?

We can keep people in the loop by implementing a new Manifold API endpoint that allows authorized users to read their own notifications and relay them to other services with various message APIs. Proposed services would be:

  • Discord, via a hosted bot or webhook

  • Slack, via a hosted bot or webhook

  • Telegram, via a hosted bot

  • Matrix, via a hosted bot

  • SMS, via Twilio

Alternatively, we can leverage an existing notification engine like Apprise.

Additional features could include notifications not currently supported by the site, such as drastic market changes.

How will this funding be used?

Aside from development time, the main direct costs here would be SMS message sending and possibly a new domain and cloud hosting.

What are the most likely causes and outcomes if this project fails? (premortem)

Many users already have the mobile app installed and so already get notifications on their phone. Additionally, the site has pretty good granularity for notifications between email, web, and mobile. This means there is a high chance that most users’ needs are already serviced by the existing notifications and do not need an external notification provider.

Open questions

  • What notification services should we prioritize?

  • What other features can we include in this project?

Who is on your team and what's your track record on similar projects?

I expect to be the only person working on these projects directly. I would gladly accept assistance from other users in the form of pull requests, comments or suggestions. You can see my past projects at https://github.com/wasabipesto.

In the past I have received mana bounties from the Manifold dev team for the following projects:

  • 7/13/22: for the initial version of the manifold stats page

  • 6/14/22: for a later version of the manifold stats page

  • 8/13/22: for a surprise gift of laser-cut Foldies for the dev team

  • 8/17/22: for animated market yield graphs, featured in the manifold newsletter

  • 7/17/23: for the initial version of calibration.city

My typical workflow is to focus on a single project at a time. I generally prefer to implement an MVP first and add features gradually based on feedback. This allows me to pivot quickly if something is not working and not waste effort on unnecessary features. In order to make these projects the best they can be, I will be seeking general user feedback more proactively than normal and providing investor updates regularly.

I have a full-time job but I enjoy working on projects like these in my spare time. I am not typically paid for these hobby projects so I just work on whatever interests me at the moment. Money from this grant would compensate me for my time and motivate me to work on the selected project when I would otherwise be unproductive. Other than that, a few projects have direct costs I have listed.

What other funding are you or your project getting?

I do not expect that I will receive any other direct funding for these projects.

holds 100%
FoldyOfficial avatar

Manifold Markets

8 months ago

Final Evaluation Results

Valuations: 3000, 3000, 4000, 8000

Median Valuation: 3500.0

Eliza: Awesome project.

Tod Waddington: These projects all seem like nice-to-haves, and it seems like a no brainer to direct funding here instead of current dev time.

(1) Calibration city is a very valuable tool which we should probably use more. It would be cool to be listed as a sponsor though if we are paying for its development!

(2) this is least important to me, people like pixels, pixels are free, postage is expensive. OTOH, it's a free ad/reminder in their homes, and

(3) Loan exchange sounds like a technical nightmare (short of having a manifold bank with fees & insurance built in), if he wants to try and work it out, I think it's worth at least ~2k (though afaik he hasn't done any of it).

(4) Maybe I'm pessimistic but I think this is unlikely to look good, even though it's a great idea

(5) This sounds like it has viral loop potential, so I like it! Plus, more leaderboards/competitive stuff is awesome. ~1k (it looks like he hasn't done this though)

(6) idek who would use this

Rachel: Last time I said $3000, and now I’ll up this to $4000 given that there have been some improvements, including a bunch that I suggested last time! (not sure whether that was causal. Regardless, it’s better.)

Some updated UI notes:

  • given that calibration is the default page, I think it should be the first one in the nav bar and that “introduction” should be called “about” instead.

  • the Calibration City logo should link to the home page, which afaik is the calibration page

  • “List” in the navbar should say “markets” or “questions list” or something more descriptive, because before clicking on that I have no idea what is on it

  • FAQ isn’t actually an FAQ which is odd. In fact, the introduction is structured like an FAQ whereas the FAQ is not!

Austin: Calibration City is now super polished and a joy to use; there are so many little thoughtful details, animations, and options that wasabipesto has provided on this site. I'm glad to see Polymarket data up as well, so that we have a beautiful and credible site to see how our different platforms compare.

holds 0%
wasabipesto avatar

wasabipesto

9 months ago

Progress update

What progress have you made since your last update?

Our biggest updates to Calibration City this month have been the addition of a few new tabs!

🤔 First up we have the introduction, a sorely-needed explainer for this whole “calibration” thing. This is mainly formatted as a dialog-based blog post where we explain the basics of quantified predictions, calibration, accuracy, prediction markets, and the site’s purpose. I hope it is able to bring the ideas behind prediction markets to a few new eyes, directly alongside the statistics to (potentially) back it up.

🥇 Up next we have the accuracy tab, with a set of graphs I teased last month. Ever wondered if longer markets are more accurate? What about markets with more traders? Now you can compare all markets across every supported platform against any attribute we measure - plus you can combine it with any of the other standard filters! Do sports markets on a specific platform get more accurate the closer to the end of the market? Are markets that resolve yes more accurate than those that resolve no? Experiment to your heart’s content!

🔍 Wait, what markets are in that bin of sports markets on that one platform that resolved no? Well to find out you can go search on that site… or you can use the new list page! List, sort, filter, and browse the markets to your heart’s content. (The API endpoint that powers this page can also be used to download all of the markets in my database, in case you’re interested in double-checking my math or rolling your own 😉)

👨‍🎓 Interested in learning more about the site? Head on over to the FAQ tab! It’ll answer your question… as long as your question is “where can I learn more about scoring prediction markets” or “give me more nitty-gritty implementation details”. I’ve been trying to include responses to actual frequently-asked questions on the relevant page with better wording or hover-text, so nothing has actually made it to this page yet 😅 Feel free to give suggestions for what I should add, though!

What are your next steps?

🏆 More scoring options! We can already calculate and show the Brier score for every market, but there are a lot more scoring methods! I plan to add logarithmic scores, spherical scores, and more.

🖇️ Even more platforms! After adding Polymarket (and then working with their dev team to get even more information) I paused adding new platforms to the site in order to get the user experience the way I wanted. Now, I think we’re in a great state and we can get moving on integrating even more data!

✅ Corpus of questions! The biggest issue with the site as it stands now is that you’re comparing apples to oranges - not all markets are comparably difficult! I don’t want to punish sites for catering their questions to their communities, and I don’t want to reward sites that might attempt to “game” statistics like calibration or accuracy. I want to build off of existing datasets, leverage tools to replicate questions across platforms, and build a large enough corpus of questions that users can confidently see how accurate different platforms were for key questions or in aggregate.

Is there anything others could help you with?

Absolutely: I’m not a technical writer, so please review what I’ve written! The introduction dialog is a quick summary of how I understand calibration and accuracy, but it could be wrong or misleading in important ways. That’s why I’ve set up this bounty for users to submit their feedback on the site, especially anything that was confusing, unintuitive, or incorrect.

I’m also interested if there is any prior art to grouping a corpus of questions like we will need to compare accuracy across platforms. Aside from doing it manually, we could leverage the markets that MirrorBot has replicated or use LLMs to determine when a market is “close enough” to be identical, but each of those have drawbacks. Since this is going to be a big feature with a high future maintenance burden, I would be happy to hear any ideas that may make it simpler!

As before, I’m always looking for ways to make the site better. If you have any ideas for things that would bring you to the site more often or features you would love to see, I’m all ears.

❤️ Thanks!

holds 100%
FoldyOfficial avatar

Manifold Markets

10 months ago

January 15th Evaluation Results

Valuations: 500, 1500, 300, 2000, 3000, 4500, 1500, 6000

Weighted Median Valuation: 3000

Eliza: They are working in #market-math to discuss future changes, they already made a lot of updates. The site looks cool. This is the kind of thing that can broaden interest in PMs in general and also help Manifold grow.

cc6: Calibration city looks useful. Currently it's still limited, but I like the features that you are planning to add. I think it would also be cool to see the calibration of prediction markets at certain points in time to see if we are improving our accuracy over time

Isaac King: Calibration City is extremely cool. Main utility would seem to be convincing people that prediction markets are accurate, and knowing which ones to trust the most. Having a well-known, reliable, central hub for this data would be worth a lot. However given past experience with passion projects like this, I'm not confident that this will be kept up to date indefinitely. If this came along with a credible guarantee of future uptime and maintenance I'd value it at several thousand, but as an as-is project that may go down at any time, I'm calling it $400; comparable to a single blog post on market calibration, but slightly more versatile.

(Calibration City could also be a good data source for bots that correct systematic miscalibration, but I'm hesitant to assign it any value on that note since such bots can turn a profit themselves and I don't think they need to be subsidized. Regardless the current API appears to not support custom queries, so I don't need to think about it for now.)

The other 5 projects appear to have not been started, so I'm valuing them all at 0.

wasabipesto: (I'll do my best to be unbiased!) The basic structure of Calibration City hasn't changed that much since the Community Fund started. It's faster, which is a good improvement but a sub-one-second loading time is largely table stakes. It's a little prettier but it could still use a nice design pass. The main value-add right now is that it has integrated additional platforms with room to add more. I think the main value for this project is the niche it will fill in the prediction market community: a third-party data source for people investigating prediction markets, evaluating platforms, and learning about forecasting. I've already seen great conversations about prediction markets in lesswrong and discord as a result of people experimenting with the data and asking question

Rachel: This is really cool, I had fun playing around with it, and yeah it was nice to not have to wait 10 seconds for it to load! A bunch of little things I noticed about the UI:

  • the "instructions" tab is default open, and doesn't really contain instructions, which was distracting/confusing.

  • it's strange that the explainer parts of the sidebar basically look the same as the adjustment parts of the sidebar. A possible alternative would be to have one info/FAQ button that opens up a modal with all the info.

  • on a smaller window, I'm not sure how to open up the sidebar to make adjustments.

  • it would be nice if clicking on the platforms in the sidebar took me to their sites.

  • it's confusing that there's an explanation of the brier score when the scores themselves are on the opposite site of the screen and not prominent—that's actually what caused me to search for another mention of it, which means it was basically a distraction. It might make more sense to have a little info tooltip right next to where the score is displayed.

David Chee: It's such a useful place to point people towards and has a fantastic, responsive UI.

Love it. Need to market it a bit more on our side for sure.

SG: Cross-platform calibration/accuracy comparisons are very useful and important!

Austin: Calibration city 2.0 is super awesome! I’m learning a lot just by playing with these sliders; it’s presented in a very user-readable fashion, encouraging you to explore and figure out some data points. I’ve said elsewhere that I think this is better than our current calibration page at manifold.markets/calibration, and hope to be able to use some of this information or directly refer to it when we talk about prediction market trustworthiness. It’s also awesome to have a third-party validation that Manifold’s play money markets provide calibrated results.

Minor note: I do kinda wish the framing of the site was a bit less competitive between prediction market platforms” though (I don’t think that was Wasabipesto’s intention, but the graph makes head-to-head comparisons very salient). Manifold would really want to be sending the message “prediction markets & platforms are much more trustworthy than pundits & polls”, and I’m wondering how to best make that case — eg could the same graph contrast against historical individual predictions?

holds 0%
wasabipesto avatar

wasabipesto

11 months ago

What progress have you made since your last update?

🎉 Calibration City 2.0 is live! You can find it at https://calibration.city/

We've integrated 2 more platforms - Kalshi and Metaculus - and completely rewritten the backend to be hundreds of times faster! Now you can experiment to you heart's content without waiting ten seconds for each reload.

What are your next steps?

We've got plenty ahead of us, which you can follow along with here at the repo. A few things we're working on:

  • 🖇️ More platforms! Three is good, but we want all of them. Polymarket is up next, and we have many more in our sights.

  • 📶 Categories! We want to be able to filter down to a category, and see those types of markets across every platform.

  • 📜 Market list! "What's that outlier?" Check the list of what markets are in each sample, and which are weighted the highest!

  • ❓ Explainers! New users should have an easier way to understand forecasting, prediction markets, and calibration so the site can be more useful to them.

  • 🔬 Brier analysis! Now that we have all of this information, calibration plots aren't the only thing we can build. We can plot the Brier scores per platform against time, number of traders, markets volume, and more.

Is there anything others could help you with?

Feedback, comments, and suggestions are always welcome. Tell me what you're using it for and what can be done to make it better!

holds 0%
wasabipesto avatar

wasabipesto

11 months ago

And Polymarket has now been added! 🎉

holds 0%
wasabipesto avatar

wasabipesto

11 months ago

Working on categories now... take a peek at the new filter (🤫) at https://beta.calibration.city/

holds 0%
wasabipesto avatar

wasabipesto

11 months ago

One more sneak peek... prototyping alternative Brier score visualizations here and here (answering the question "why does Polymarket's brier score drop so suddenly when you increase the minimum duration?")

holds 100%
FoldyOfficial avatar

Manifold Markets

11 months ago

December 15th Evaluation Results

Valuations: 500, 0, 0, 5000, 3000, 1000

Median Valuation: 750

(Because the median valuation is below the minimum valuation and this is a proposal, we can't create a limit order at this valuation)

David Chee: I've personally invested some mana because I want him to succeed and he has already provided a lot of value to Manifold. Also when creating the track record page I felt like a lot of data which I knew does exist, but just wasn't compiled, would have been great to have. Not valuing at his asking though just because of the rough guidelines Austin set out in how to value projects and because we are meant to be evaluating based on how much we think Manifold would have paid for the work done so far (and I'm not counting his work that happened a while before Manifund).

SG: It seems like they haven't done anything yet. But an updated Calibration City would be cool.

Rachel: Since nothing’s been shipped yet, have to give it a valuation of $0. The old version looks pretty good: more polished design than I was expecting, seems pretty easy to use. In terms of helping Manifold though, even a perfect version of this wouldn’t be that great for getting new users or boosting engagement.

Inga Wei: I love most of these ideas! Particularly Calibration City, Physical League Awards, Market Stories and Brag Board (especially if you can get the UX right).

Sinclair: I love calibration city! I reference it a lot whenever people ask questions about our site accuracy. Calibration is a kind of signal that's hard to fake, and it gets us cred among rationalists even if it's kinda hard for ordinary people to parse. I think it has informed some of our product decisions somewhat - knowing that most accuracy comes from the first few traders made us focus a bit more on the "great variety of markets" strategy

Austin: Really appreciate all of your past contributions! FWIW I’d rank them as 3, 1 > 2 > 5, 6 > 4. It’s promising that several users are specifically asking for 3 to exist; meanwhile, 1 has been a useful prompt for our team to show Manifold calibration (and it would be really, really awesome to just use your Calibration results on our site — that could be worth like an order of magnitude more.)

And on the brass foldy: I remembered being floored when I opened them up in the mail. Do you know just how rare it is to get a physical token of gratitude? I think that was one of the early signs that Manifold was something special, that our users would be inspired to send us a real version of our logo. TBH I don’t know if I’d want them to be a reward restricted to Master’s League, but rather like, a token we send out to people who are nice to each other on the site; maybe you can pay 5000 mana to send someone a physical “foldygram” as thanks for something they’ve done.

Sinclair avatar

Sinclair Chen

11 months ago

I encourage you to use the updated foldy design on the website. it is very similar except the lines have been adjusted to be more even. in the github the asset is logo.svg

holds 0%
wasabipesto avatar

wasabipesto

11 months ago

@Sinclair If you're talking about the platform icon in the Themis repo I'll get that updated before I deploy anything. If you're talking about the brass foldies I'll use that as a starting point if I manage to get funded in the next 24 hours.

Austin avatar

Austin Chen

11 months ago

@wasabipesto note that the funding deadlines are somewhat loose - we can extend that for you if you want to give investors a few more weeks, esp since we've been late on the first round of evals (orz)

holds 0%
wasabipesto avatar

wasabipesto

11 months ago

@Austin Thanks, but I'd rather keep going with Themis and let my prospective investors utilize their money elsewhere rather than lock up their bids an additional few weeks.

holds 0%
wasabipesto avatar

wasabipesto

12 months ago

Project update: During the proposal stage I have started working on a rewrite of Calibration City, currently under the name Themis. You can see my progress here: https://github.com/wasabipesto/themis

If this proposal is not funded I plan to continue working on Themis and have high hopes for it. If the proposal is funded I may spend some time on it while also working on whatever project is selected.

🐭

David Chee

12 months ago

Give me more data and stats!

MarcusAbramovitch avatar

Marcus Abramovitch

12 months ago

Want to separate out project #3? I'd happily be the investor

holds 0%
wasabipesto avatar

wasabipesto

12 months ago

@MarcusAbramovitch I don't plan to separate out any of these projects, but if this proposal is funded I will weigh my investors' advice on which to pursue much more heavily. Currently project #3 is ranked second-lowest on the market, so it would take a bit of convincing for me to pursue it over the others.

Genzy avatar

Tod Waddington

12 months ago

Loan exchange seems like a lot of fun. I could blow up my account much faster if I had access to more loans.

The most exciting one for me is the market stories. Manifold can really lean into the news value and become a regular journalistic source by making it easier for people to see at a glance what was happening as a market was moving. It also gives us the opportunity to go "Manifold was saying this before the news article was released!". Al-Ahli comes to mind. Huge win for Manifold that could be neatly packaged and showcased to outsiders instead of needing a long explanation.

It would also look really nice on my ads if markets showed on the graphs what happened that caused a move - kind of like how stock graphs show splits/dividends/earnings/etc.

🐶

Eliza

12 months ago

My favorite one is loan exchange!