Tracking the money flows in forecasting

By NunoSempere @ 2022-11-09T16:10 (+76)

This is a linkpost to https://nunosempere.com/blog/2022/11/06/forecasting-money-flows/

This list of forecasting organizations includes:

• A brief description of each organization
• A monetary estimate of value. This can serve as a rough but hard-to-fake proxy of value. Sometimes this is a flow (e.g., budget per year), and sometimes this is an estimate of total value (e.g., valuation).
• A more subjective, rough, and verbal estimate of how much value the organization produces.

This started as a breadth first evaluation of the forecasting system, and to some extent it still is, i.e., it might be useful to get a rough sense of the ecosystem as a whole. After some discussion on whether very rough evaluations are worth it (a), people who prefer their evaluations to have a high threshold of quality and polish might want to either ignore this post or just pay attention to the monetary estimates.

Summary table

Note that the monetary value column has different types of estimates. 

Larger entities

Flutter Entertainment

What it is: Flutter Entertainment (a) is one of the largest gambling companies. It was created from the merger of Betfair (a) and Paddy Power (a). It has a large presence in the EU and UK.

Monetary value (market cap): It has a ~$20B (~£18B) market cap.

Social value: Betfair and Paddy Power have some markets with public utilities, e.g., around elections, but focus mostly on sports. However, its social value is otherwise low or negative—e.g., by enabling or creating gambling addicts.

Note: There are many other betting companies, e.g., Draft Kings, which I will not review here because they tend to focus mostly on sports, though I will note that they tend to have a high market capitalization.

Kalshi

What it is: Kalshi is a US company which aims to provide prediction markets (a) to US consumers while complying and shaping US regulations on this topic.

Monetary value (VC funding, valuation): Kalshi (a) has received ~$30M in VC funding. My sense is that it’s probably worth much more than that ($100M+) due to being the first prediction market regulated by the CFTC. Previous markets—like PredictIt or Iowa Electronic Markets—were only operating with a no-action letter.

Social value: There has been a bit of drama around Kalshi recently. In recent times, the CFTC withdrew its letter of no action from PredictIt, and fined and banned Polymarket from operating in the US. These are two of Kalshi’s competitors, and my sense is that Kalshi probably contributed to these events. As such, I’d estimate Kalshi’s impact so far to probably be negative, on account of these anti-competitive practices. In the future, however, if Kalshi contributes to the adoption of prediction markets in the US, and this leads to better decision-making, it could end up having a large positive value. It is also possible that their contracts have value as hedging instruments.

Metaculus

What it is: Metaculus is a forecasting site and a public benefit corporation that spearheads forecasting initiatives.

Monetary value (grant flow): Recently, Open Philanthropy granted them $5.5M (a). With a 5% discount rate, and asuming the $5.5M grant is for 2 years will be indefinitely renewed, this makes¹ Metaculus' discounted cashflow be worth $55M. Metaculus previously received $308k (a) from the EA infrastructure Fund in 2021 and $65k (a) from the EA Long-term Future Fund in 2020.

Social value: With its $5.5M Open Philanthropy grant, Metaculus has been growing its already 12-person-strong team (a), and is hiring for a number of positions (a). So it’s one of the EA forecasting organizations most able to scale and deploy human capital in terms of sheer number. The community which makes up its forecasting site is also reasonably strong.

Polymarket

What it is: Polymarket is a crypto prediction market. It combined various technologies: magic.link, Polygon (a) and the Gnosis conditional contract in order to build a usable crypto prediction market at a time where the previous iteration, Augur, had become unusable in practice due to high fees on the Ethereum blockchain.

Monetary value (VC funding): It has received $4M in VC funding (a), according to Crunchbase. It’s unclear to me what its valuation is.

Social value: Polymarket has had a range of public interest markets, on politics, COVID infection numbers, or the Russian invasion of Ukraine.

Cultivate Labs

What it is: Cultivate Labs is a company which offers infrastructure for forecasting tournaments. They currently host Good Judgment Open, INFER, the Cosmic Bazaar, and a Czech Prediction platform—and probably a few more platforms that I don’t know of.

Monetary value (valuation): I’m pretty uncertain about how much they are worth, and I’d give a guess of $5M to $80M.

Social value: I used to view them as a bit clunky and outdated, but I’ve come to respect that they are one of the only few groups willing to offer a full package, with support for multiple years.

Good Judgment

What it is: Good Judgment" I mean the set of organizations which administer the public prediction market tournament GJOpen (a), and provide forecasts and training through the Good Judgment brand (a)

Monetary value (valuation): I’m fairly uncertain how much they would be worth. Maybe a very wide interval would be $3M to $50M.

Social value: Good Judgment has helped build awareness of superforecasting practices. Their forecasts provided to their subscribers have probably influenced the decisions of their readers and of those who commissioned them. Their training at government and corporations probably marginally increased the decision quality of clients, but it’s unclear to me to what extent and for how long.

Manifold Markets

What it is: Manifold Market is a play-money prediction market site with a really nice user experience.

Monetary value (VC funding, valuation): Manifold Market has received $2M in funding, ($2M in seed funding (a), of which $1M from the FTX Future Fund’s regrantor program (a)), at a $15M valuation. They have a superb software engineering team, and one of the fastest development speeds, but on the other hand their user acquisition has been a bit slow (a). My sense is that their valuation would be somewhere between $10M and $50M

Social value: So far, by making an eminently usable site, they allow users to signal their beliefs and challenge others to do the same. They have also hosted a few tournaments, and at least the Clearer Thinking tournament (a) was probably decision-relevant. But their value is linked to their future growth.

Insight Prediction

What it is: Insight Prediction (a) is a new prediction market which accepts crypto.

Monetary value (valuation): This would depend on their usage numbers, which aren’t public. I’d say it’s worth at least $5M, but whether the upper bound is $50M or $500M probably depends on the team’s execution over the coming year.

Social value: It is one of the few prediction sites which hosts real-money somewhat liquid markets on the Ukraine-Russia conflict.

Tetlock’s research group

What it is: Tetlock’s research group is a loose collaboration of academics working in collaboration with Phil Tetlock (a)—author of the Superforecasting book.

Monetary value (valuation, grant flow): They have received at least $2M from Open Philanthropy (1 (a), 2 (a)), and probably more than that through the University of Pennsylvania. It’s unclear to me how much to value a research group, but I’d give a ballpark of $1M to $20M.

Social value: Tetlock’s work on expert political judgment and on superforecasting was seminal work that kick-started the current forecasting community. In recent times, it seems likely that work stemming from their work on estimating existential risk (a) could be action-guiding for large organizations and governmental bodies, but most of the output from that research direction hasn’t been published yet.

PredictIt

What it is: PredictIt is a US-based prediction market. Historically, a community of sharp prediction market players coalesced around it. But in recent times, the CFTC has withdrawn its permission for it to operate in the US.

Monetary value (valuation): Insight Prediction gives it only an 11% chance of it surviving (a). My guess is that it was worth $5M to $50M when alive, implying a $0.5M to $5M valuation right now.

Social value: PredictIt was fairly instrumental in the creation of a passionate if sometimes a bit cut-throat prediction market community. Its future value depends on its survival.

Epoch

What it is: Epoch (a) is a research team investigating AI progress.

Monetary value (grant flow): They have received around $2M (a) from Open Philanthropy.

Social value: My sense is that their research has usefully informed some OpenPhilanthropy decisions. Open Philanthropy donated around $80M in 2021 to the “Potential Risks from Advanced AI” cause area. A 5% improvement in that decision-making would imply a value of $4M/year.

Swift Centre For Applied Forecasting

What it is: The Swift Centre (a) publishes publicly available forecasts from accurate forecasters on topics of importance.

Monetary value (grant flow): They have received $2M from the FTX future fund regranting program (a).

Social value: Their forecasts (a) lead to better decisions. My sense is that the organization is still getting up to speed, and that when running at full operational capacity, they’ll be producing many more predictions.

Quantified Uncertainty Research Institute (QURI)

What it is: QURI is a “new initiative to advance forecasting and epistemics to improve the long-term future of humanity. We write research and make software.” In particular, we are developing Squiggle, a web-capable programming language for better estimation, and doing research around scalable forecasting and estimation.

Monetary value (grants flow): Overall we have received around $780k so far (1 (a), 2 (a), 3 (a)), maybe a bit more.

Social value: Most of QURI’s value is coming from the value of Squiggle (a), which people are now starting to use for cost-effectiveness estimates, and from the value of my own research (a), which might move the Effective Altruism movement more in the direction of using quantified estimates even for speculative stuff.

Note: This is the organization that I work for.

Samotsvety Forecasting

What it is: Samotsvety Forecasting (a) is a somewhat ad-hoc group of highly accurate forecasters of which I am a member.

Monetary value: My sense is that we could find some funder to sacrifice $200k to $5M if our group’s existence depended on it. But I haven’t actually tried that!

Social value: We’ve published a variety of public forecasts, and we find ourselves more and more to be a part of Effective Altruism’s general epistemic infrastructure.

Czech Priorities

What it is: Czech Priorities (a) is a Czech group working to advance forecasting within the Czech Republic.

Monetary value: I couldn’t find grantmaking information from a quick online search.

Social value: It’s possible that they could find generalizable lessons about using forecasting to influence policy and improve decision-making that could then be extended to other countries. But this is not certain.

Hedgehog Markets

What it is: Hedghog Markets (a) is a crypto-prediction market

Monetary value: They raised $3.5M in funding back in the crypto bull days.

Social value: They have been particularly innovative around no-loss markets, and peer-to-peer markets. But value depends on future usage volume.

INFER

What it is: INFER is a forecasting platform which seeks to influence US government policy around transformative technologies.

Monetary value: INFER is hosted at ARLIS (a), at the University of Maryland, which received an ~$8M grant from Open Philanthropy specifically for forecasting (a) (over two years, over two platforms), which I find very confusing because I don’t think they have deployed much of that capital yet.

Social value: INFER aims to influence decisions by the US government around technology policy. To the extent that this is the case, it could be valuable indeed, but this is hard to evaluate from the outside.

Smaller forecasting entities

Nathan Young

Nathan Young is a smallish forecasting influencer followed by 6k people on Twitter. He recently got $182k from the FTX Future Fund regranting program (a) to build a platform for forecasting question generation, an alpha version of which can be seen here (a).

Sage

Sage is an organization dedicated to forecasting tooling and research. It got $700k from the FTX Future Fund, and has built the tools on quantifiedintuitions.org (a). But it hasn’t deployed most of its capital yet.

Global Guessing

Global Guessing (a) is a forecasting site which made and gathered forecasts early on the Russian invasion of Ukraine, but which seems pretty dormant since then. They have received ~$330k from the FTX Future Fund. (a)

Social Science Prediction Platform

The Social Science Prediction Platform (a) collects forecasts from academics on upcoming papers. This can have a range of public benefits, such as estimates of how surprising results in the social sciences are. Personally I’d be hopeful about the ability to construct a more legible or objective Bayesian prior. I could quickly find that they have received $346k (a) from the Survival and Flourishing Fund, and an additional $492k from the FTX Future Fund's regranting program (a)

Augur

Augur was a pioneering prediction market way back when, but isn’t up to much these days. Somehow the implied market cap (a) of their coin is still ~$60M (?!?).

Confido

Confido (a) is a three-person strong Czech organization which develops forecasting tooling. They received $190k from the FTX Future Fund regranting program (a).

Hypermind

Hypermind (a)is a French forecasting organization which hosts some low-stakes markets with play money but real-money rewards.

Replication Markets

Replication Markets (a) was a super-interesting experiment in having forecasters predict the outcome of replications. It has now ended, and it’s uncertain whether the creators will follow up with something as interesting. They had ~$150k in forecaster rewards.

PredictionBook

PredictionBook (a)is an old, old site used by people to keep track of their probabilities, with no rewards involved. Gwern still uses it (a).

Gnosis

Gnosis is much like Augur, but less successful. They also have a GnosisDAO (a). The implied market cap of their token is worth $230M (a), based on the Gnosis safe (a). But it doesn’t do much forecasting these days.

Thoughts on this ranking

On the money flows

Money flows are interesting, but flawed as a measure of value. The spending rate or VC money raised does not correspond to the value a project creates. The high valuations for crypto projects could be mostly illusory, and for the case of Gnosis and Augur, they probably are.

Still, a monetary ranking has some advantages

• There may be some ambiguity to it, but it allows one to differentiate between <$500k projects, $500k to $5M projects, and $5M to $50M projects.
• It has an honesty to it, in that money is a somewhat hard-to-fake signal
• It is a fast proxy to estimate

On the estimate of social value

The estimate of social value isn’t a quantified estimate, but rather a few lines of observations, and maybe a path to impact. I think I could translate this into a numerical relative value scale, but it would take a fair amount of time.

On the different pathways to impact

I’m seeing mainly three pathways of impact for forecasting projects:

1. Providing consumer value, e.g., betting markets or hedging utilities that people want to use
2. Providing informative markets about events of interest, which can apply pressure to decision-makers inside governments and corporations to make better decisions
3. Providing tooling that people making decisions can make and incorporate as part of their decision-making

I’m mostly bullish on the third option, but this deserves more elaboration.

On how this estimate could be improved

This estimate could be improved by having numerical estiamates of impact. This would allow for comparisons around efficiency &c. It could also be presented better, i.e., in a table, or more creatively, in a map.

Significance given FTX's current troubles

Given FTX's current troubles, it's very possible that there will be less money floating around for altruistic forecasting projects. At the same time, some of the projects FTX has donated to may  prove successful, and other funders may want to step in.