@RyanKidd Thank you for the contribution and the kind words! I agree AI forecasting is very important and is therefore one of the primary topic areas I intend to be featured on the site. I also think that the most important questions in that area will be long-horizon and future accuracy may not be reflected by past performance, but I'm sure there is still plenty to learn.
@wasabipesto
$0 in pending offers
Investments
Comments
wasabipesto
3 months ago
This seems like a technically interesting solution but extremely niche application. It will be difficult to get people to trust you with their anonymity for the most beneficial situations and for less privacy-critical situations many groups will default to the simpler solution of âtext X person if youâre interestedâ. There might be some growth potential if the site can become a âplatformâ like Kickstarter or GoFundMe, but Iâm not sure how that would work with your target applications.
What does success look like for you? Is there a path forward where this product exists but without a full-time support team?
wasabipesto
3 months ago
Nice to see this project is still going, I havenât heard much about it recently.
Some questions:
Does https://brierfoxforecast.com/ represent the current state of development? Are there any other currently-active instances?
Whatâs the advantage to running an independent instance? Why would a small group go through the trouble to run an instance instead of using an already-established server?
Do you have any changes to market structure or feature set planned to differentiate you from competitors? Are you implementing the same AMM used by Manifold?
If the application is intended to be run locally, have you tested the resource usage for a typical instance? Could it run from, say, a laptop or home computer? What size is the DigitalOcean server used for the demo instance?
wasabipesto
3 months ago
An interesting set of ideas! Some questions:
I believe Manifold is getting rid of the managram system as part of the prize point pivot, so unfortunately I donât think managram subscriptions will be very useful.
For an automated arbitrage bot I think it would probably be better to submit this separately from the comments section to avoid revealing your linked positions. If someone had that information they could take advantage of it by e.g. forcing you to trade into their limit orders. **If this was in your web app you could overlay the two target markets and keep a list of active arbitrages, or even arbitrage against a different siteâs market.
The secret market idea is interesting, but if I wanted to do that right now Iâd just make one called âSecret Marketâ and communicate the true purpose of the market to whoever I wanted to trade with separately, through direct messages or a private google doc. What advantage would this extension have?
The manifoldmaven account is neat! I would love to see something like that with a little more signal:noise. The latest 3 posts are all about the RFK news, and the same Trump twitter market is on there three times. Thereâs also a few that are only caused by transient spikes which were quickly reverted. I think thereâs also some low-hanging fruit to finding the root cause of a swing, usually a news article posted recently in the market.
Additionally, I wanted to see what the âGet these alerts in your inboxâ link would do, but it just takes me to a place to pay $20/mo. If I were to pay this would I get all of these alerts in my email inbox? SMS inbox? Would I be able to customize the swing percentage or timeframe? More information would be good!
Unfortunately I donât think I would pay any amount of money for this information, since learning about news instantly is nice but has basically no value to me over learning about the news whenever I get around to reading it.
Reading further down I think the Predictor's Digest is much closer to what I would want out of this, a full newsletter linked to prediction markets. Iâm unsure of the AI aspect but would be interested to see.
I think loan infrastructure for Manifold would be very nice and has lots of potential, but with mana exchange going away I think loans and tips will be no more.
For the personal prediction tracker, how would this compare to something like Fatebook? I have used Fatebook in the past and it has a very nice interface for this sort of thing.
wasabipesto
3 months ago
This seems like an extremely beneficial service that is sorely needed by the community!
Could you provide an example of content you have produced for an EA org in the past?
Approximately how many additional projects would you be able to take on if this proposal is completely funded?
wasabipesto
3 months ago
What types of social science research are you expecting to fund with this?
Why should we award this proposal rather than finding those researchers directly?
What service do you use to find participants and ensure they are valid participants?
wasabipesto
3 months ago
I am unfamiliar with both fields involved here (neuroimaging and cryptography) so I had to do some digging to understand this proposal. Hereâs my understanding of the situation and some basic questions, let me know if Iâm misunderstanding:
The problem is that researchers arenât sharing anonymized brain imaging data. This is already legal but rarely done in practice due to privacy concerns.
What types of advancements are lacking good brain scan data? What sorts of research is this data good for?
The solution is to apply a special form of encryption to the brain scan data. When the data is encrypted this way it allows researchers to do some stuff with the data but not other stuff.
What exactly can you do with an encrypted scan vs a non-encrypted scan? If itâs already anonymized, what danger exists in a non-encrypted scan thatâs removed when itâs encrypted?
There are a lot of ways to do this special encryption. The first part of this project is to take some existing brain scans and try each way of encrypting them, hopefully finding one that works but is still easy to use for researchers.
What is your list of HE implementations? What criteria do they need to pass in order to be useful? What sort of performance impact do you expect from this intermediate processing step?
The last part of this project is to take a live brain scan and encrypt it in real-time so itâs never unencrypted.
What benefit does this have? Shouldnât original researcher should still have full access to the original scan?
wasabipesto
3 months ago
Wow, I love what youâve done so far. As someone whoâs gone through the same process, your blog post on serverless ETL was very interesting!
I think the new data platform screenshots are great. Being able to successfully match markets to trending news stories is nothing short of magical. If you can connect equivalent markets across platforms then Iâd likely just start using your API instead of building my own!
I think the RSS reader idea is nice, but I donât see myself moving away from FreshRSS anytime soon. I am quite interested in a newsletter however!
wasabipesto
9 months ago
What progress have you made since your last update?
Our biggest updates to Calibration City this month have been the addition of a few new tabs!
đ¤ First up we have the introduction, a sorely-needed explainer for this whole âcalibrationâ thing. This is mainly formatted as a dialog-based blog post where we explain the basics of quantified predictions, calibration, accuracy, prediction markets, and the siteâs purpose. I hope it is able to bring the ideas behind prediction markets to a few new eyes, directly alongside the statistics to (potentially) back it up.
đĽ Up next we have the accuracy tab, with a set of graphs I teased last month. Ever wondered if longer markets are more accurate? What about markets with more traders? Now you can compare all markets across every supported platform against any attribute we measure - plus you can combine it with any of the other standard filters! Do sports markets on a specific platform get more accurate the closer to the end of the market? Are markets that resolve yes more accurate than those that resolve no? Experiment to your heartâs content!
đ Wait, what markets are in that bin of sports markets on that one platform that resolved no? Well to find out you can go search on that site⌠or you can use the new list page! List, sort, filter, and browse the markets to your heartâs content. (The API endpoint that powers this page can also be used to download all of the markets in my database, in case youâre interested in double-checking my math or rolling your own đ)
đ¨âđ Interested in learning more about the site? Head on over to the FAQ tab! Itâll answer your question⌠as long as your question is âwhere can I learn more about scoring prediction marketsâ or âgive me more nitty-gritty implementation detailsâ. Iâve been trying to include responses to actual frequently-asked questions on the relevant page with better wording or hover-text, so nothing has actually made it to this page yet đ Feel free to give suggestions for what I should add, though!
What are your next steps?
đ More scoring options! We can already calculate and show the Brier score for every market, but there are a lot more scoring methods! I plan to add logarithmic scores, spherical scores, and more.
đď¸ Even more platforms! After adding Polymarket (and then working with their dev team to get even more information) I paused adding new platforms to the site in order to get the user experience the way I wanted. Now, I think weâre in a great state and we can get moving on integrating even more data!
â Corpus of questions! The biggest issue with the site as it stands now is that youâre comparing apples to oranges - not all markets are comparably difficult! I donât want to punish sites for catering their questions to their communities, and I donât want to reward sites that might attempt to âgameâ statistics like calibration or accuracy. I want to build off of existing datasets, leverage tools to replicate questions across platforms, and build a large enough corpus of questions that users can confidently see how accurate different platforms were for key questions or in aggregate.
Is there anything others could help you with?
Absolutely: Iâm not a technical writer, so please review what Iâve written! The introduction dialog is a quick summary of how I understand calibration and accuracy, but it could be wrong or misleading in important ways. Thatâs why Iâve set up this bounty for users to submit their feedback on the site, especially anything that was confusing, unintuitive, or incorrect.
Iâm also interested if there is any prior art to grouping a corpus of questions like we will need to compare accuracy across platforms. Aside from doing it manually, we could leverage the markets that MirrorBot has replicated or use LLMs to determine when a market is âclose enoughâ to be identical, but each of those have drawbacks. Since this is going to be a big feature with a high future maintenance burden, I would be happy to hear any ideas that may make it simpler!
As before, Iâm always looking for ways to make the site better. If you have any ideas for things that would bring you to the site more often or features you would love to see, Iâm all ears.
â¤ď¸ Thanks!
wasabipesto
9 months ago
I'm a bit confused about this proposal. Are you looking for retroactive funding as compensation for past site moderation? To my knowledge the community fund is not continuing so I'm not sure how you would expect to gain investors for future work. Are you claiming that you'll stop the vast majority of your moderation if this is not filled? I think some context would be helpful.
wasabipesto
11 months ago
Working on categories now... take a peek at the new filter (đ¤Ť) at https://beta.calibration.city/
wasabipesto
11 months ago
What progress have you made since your last update?
đ Calibration City 2.0 is live! You can find it at https://calibration.city/
We've integrated 2 more platforms - Kalshi and Metaculus - and completely rewritten the backend to be hundreds of times faster! Now you can experiment to you heart's content without waiting ten seconds for each reload.
What are your next steps?
We've got plenty ahead of us, which you can follow along with here at the repo. A few things we're working on:
đď¸ More platforms! Three is good, but we want all of them. Polymarket is up next, and we have many more in our sights.
đś Categories! We want to be able to filter down to a category, and see those types of markets across every platform.
đ Market list! "What's that outlier?" Check the list of what markets are in each sample, and which are weighted the highest!
â Explainers! New users should have an easier way to understand forecasting, prediction markets, and calibration so the site can be more useful to them.
đŹ Brier analysis! Now that we have all of this information, calibration plots aren't the only thing we can build. We can plot the Brier scores per platform against time, number of traders, markets volume, and more.
Is there anything others could help you with?
Feedback, comments, and suggestions are always welcome. Tell me what you're using it for and what can be done to make it better!
wasabipesto
11 months ago
@Austin Thanks, but I'd rather keep going with Themis and let my prospective investors utilize their money elsewhere rather than lock up their bids an additional few weeks.
wasabipesto
11 months ago
@Sinclair If you're talking about the platform icon in the Themis repo I'll get that updated before I deploy anything. If you're talking about the brass foldies I'll use that as a starting point if I manage to get funded in the next 24 hours.
wasabipesto
12 months ago
Project update: During the proposal stage I have started working on a rewrite of Calibration City, currently under the name Themis. You can see my progress here: https://github.com/wasabipesto/themis
If this proposal is not funded I plan to continue working on Themis and have high hopes for it. If the proposal is funded I may spend some time on it while also working on whatever project is selected.
wasabipesto
12 months ago
@MarcusAbramovitch I don't plan to separate out any of these projects, but if this proposal is funded I will weigh my investors' advice on which to pursue much more heavily. Currently project #3 is ranked second-lowest on the market, so it would take a bit of convincing for me to pursue it over the others.
Transactions
For | Date | Type | Amount |
---|---|---|---|
Play money prediction markets | 2 months ago | project donation | 20 |
Manifund Bank | 2 months ago | withdraw | 3764 |
Adjacent News | 2 months ago | project donation | 224 |
Calibration City | 2 months ago | project donation | +10 |
Calibration City | 2 months ago | project donation | +20 |
Calibration City | 2 months ago | project donation | +25 |
Calibration City | 2 months ago | project donation | +10 |
Calibration City | 2 months ago | project donation | +50 |
Calibration City | 2 months ago | project donation | +2000 |
Calibration City | 2 months ago | project donation | +100 |
Calibration City | 2 months ago | project donation | +10 |
Calibration City | 2 months ago | project donation | +40 |
Calibration City | 2 months ago | project donation | +25 |
Calibration City | 2 months ago | project donation | +100 |
Calibration City | 2 months ago | project donation | +50 |
Calibration City | 2 months ago | project donation | +300 |
Calibration City | 2 months ago | project donation | +25 |
Calibration City | 2 months ago | project donation | +949 |
Calibration City | 2 months ago | project donation | +20 |
Calibration City | 2 months ago | project donation | +30 |
Manifund Bank | 3 months ago | deposit | +200 |
Manifund Bank | 8 months ago | withdraw | 3500 |
Wasabipesto's Umbrella Project | 8 months ago | user to user trade | +3500 |
Isaac's power-user Manifold search and dashboard | 8 months ago | user to user trade | +44 |
Isaac's power-user Manifold search and dashboard | 12 months ago | user to user trade | 100 |
Manifund Bank | about 1 year ago | mana deposit | +100 |