Higher-Order Forecasts
By Ozzie Gooen @ 2024-05-22T21:49 (+35)
Higher-order forecasting could be a useful concept for prediction markets and forecasting systems more broadly.
The core idea is straightforward:
Nth-order forecasts are forecasts about (N-1)th order forecasts.
Examples
Here are some examples:
0-Order Forecasting (i.e., the ground truth)
- Biden won the 2020 U.S. presidential election
- The US GDP in 2023 was $27 trillion
1st-Order Forecasting (i.e., regular forecasting)
- What is the chance that Trump will win the 2024 U.S. presidential election?
- What will be the GDP of the US in 2024?
2nd-Order Forecasting
- How much will the forecasts for US GDP in 2024 and 2025 be correlated over the next year?
- How many forecasts will the question "What will be the GDP of the US in 2024?" receive in total?
- If the question “What is the chance that a Republican will win the 2028 Presidential Election?” was posted to Manifold, with a subsidy of 100k Mana, what would the prediction be, after 1 month?”
3rd-Order Forecasting
- How much will the forecasts, [How much will the forecasts for US GDP in 2024 and 2025 be correlated over the next year?] and [How many forecasts will the question "What will be the GDP of the US in 2024?" receive in total?], be correlated, from now until 2024?
- How valuable were all the forecasts for the question, [‘How many forecasts will the question "What will be the GDP of the US in 2024?" receive in total?’]
As forecasting systems mature, higher-order forecasts could play a role analogous to financial derivatives in markets. Derivatives allow for more efficient pricing, risk transfer, and information aggregation by letting market participants express views on the relationships between assets. Similarly, higher-order forecasts could allow forecasters to express views on the relationships between predictions, leading to a more efficient and informative overall forecasting ecosystem.
Benefits
Some potential benefits of higher-order forecasting include:
- Identify Overconfidence
- Improve the accuracy of forecasts by having participants directly predict and get rewarded for estimating overconfidence or poor calibration in other forecasts.
- "How overconfident is [forecast/forecaster] X"
- Prioritize Questions
- Prioritize the most important and decision-relevant questions by forecasting the value of information from different predictions.
- "How valuable is the information from forecasting question X?"
- Surface Relationships
- Surface key drivers and correlations between events by letting forecasters predict how different questions relate to each other.
- "How correlated will the forecasts for questions X and Y be over [time period]?"
- Faster Information Aggregation
- Enable faster aggregation of information by allowing forecasts on future forecast values, which may update more frequently than the underlying events.
- "What will the forecast for question X be on [future date], conditional on [other forecasts or events]?"
- Leverage Existing Infrastructure
- Leverage the existing infrastructure and resolution processes of prediction platforms, which are already designed to handle large numbers of forecasting questions.
We've already seen some early examples of higher-order forecasts on platforms like Manifold Markets. For example, with the recent questions:
- Will there be a Manifold bot that makes profitable bets on random 1-month markets by December 2024? (Ṁ3,000 subsidy!)
- Manifold Top Traders Leaderboard Ranking Prediction (2024)
- If Manifold begins allowing real-money withdrawals, will its accuracy improve?
- Is Manifold's P(Doom) by 2050 currently between 10% and 90%? [Resolves to Poll]
- Will Manifold be more accurate than real-money markets in forecasting the 2024 election?
Challenges
Of course, there are also challenges and risks to consider with higher-order forecasts:
- The accuracy of higher-order forecasts depends on the accuracy of the lower-order forecasts they build on. If the underlying forecasts are poorly calibrated or noisy, that will limit the value of higher-order forecasts.
- Higher-order forecasts inherently add complexity to forecasting systems, which could create challenges for participation, interpretation, and managing systemic risks.
- A substantial base of lower-order forecasting questions is needed before higher-order forecasts can be productively created on top of them.
Alternative Names
I considered a few options for names, asked around a bit, and settled on "higher-order" for this term. Here are some other options I considered:
- Derivatives: In the financial market, "markets about markets" are called derivatives. However, "derivative" is often understood as a term specific to markets, which could make it more confusing for forecasting.
- Meta-forecasts: I used this term before. It's a catchy term, but it doesn't differentiate between layers easily. There's no straightforward way to refer to "meta-layer 1."
- Higher-Layer: Similar to "higher-order," but less precise.
If there's contention on this later, it could be useful to have some formal discussion, to make sure that we share consistent terminology. Right now, I doubt many people care about it.
Conclusion
Over time, I expect higher-order forecasts to go from a niche idea to a key component of mature forecasting systems. Just as financial markets would be far less efficient without derivatives, forecasting platforms could see substantial accuracy and liquidity gains from higher-order forecasts.
Sanjay @ 2024-05-22T23:29 (+10)
An application I was expecting you to mention was longer term forecasts. E.g. if there was a market about, say, something in 2050, for example, the incentives for forecasters are perhaps less good, because the time until resolution is so long. But a "chained" forecast capturing something like "what will next year's forecast say" (and next year's forecast is about the following year's forecast, and so until you hit 2050, when it resolves to the ground truth).
This assumes that forecasters are less effective when it comes to markets which don't resolve for a long time.
Ben Millwood @ 2024-05-25T18:37 (+2)
In principle, normal markets should work this way. That is, if there's a market that won't settle for a year, but you think next week the price is going to go up a bunch, you will want to buy it now, and then sell it when the price goes up. If lots of people do this, the price goes up now, instead of next week (and in fact, if everyone saw that coming, it went up last week instead, and so on). If the market is reasonably liquid and/or there are market makers, you're not committing yourself until settlement, you can just sell out of your position when the price corrects (or when you give up on it doing so).
If, on the other hand, the market is not reasonably liquid, then I don't think iterated markets fix your problem, because people don't have a strong reason to expect the next market forecast to match the actual probability, so they can't profit by trading on that basis.
Ozzie Gooen @ 2024-05-22T23:39 (+2)
Good point.
I think this sort of use case is very important. I think that this problem is probably better addressed with other methods rather than straightforward sequences of separate questions, but I imagine the latter will occasionally be useful as a solution.
I've also been thinking about this use case recently. I think I like the term "progressive" forecast, but I'm curious to get more takes!
Bolton Bailey @ 2024-05-23T02:49 (+3)
Readers of this comment thread might be interested in this blog post from Johnathan Mann and this followup, where this concept is called "iterative markets".
Ozzie Gooen @ 2024-05-23T14:43 (+2)
I did find that interesting, thanks for the links!
Arepo @ 2024-05-23T03:02 (+2)
What other methods do you have in mind for it?
Ozzie Gooen @ 2024-05-23T14:45 (+4)
You can imagine strategies like,
"There's just one question. However, people will get paid out over time, if the future aggregate agrees with their earlier forecasts. These payments can trigger at arbitrary times, and can feature a lot of flexibility regarding how far back the forecasts are that they reward."
The effect is very similar to doing it by formally having separate questions.
(I'm sure many would consider this a minor difference)
Ben Millwood @ 2024-05-23T17:12 (+9)
A potential downside is that markets about markets are generally easier to manipulate than markets about ground truth. You may even find that second-order markets create an incentive to distort first-order markets to an extent that makes the existing markets less reliable.
Ozzie Gooen @ 2024-05-23T17:22 (+2)
I think that higher-order markets definitely make things more complicated, in part by creating feedback loops and couplings that are difficult to predict.
That said, there are definitely a few ways in which higher-order markets could potentially make markets more reliable.
My guess is that useful higher-order markets will take a lot of experimentation, but with time, we'll learn techniques that are more useful than harmful.
Gustavo Lacerda @ 2024-05-24T22:31 (+3)
Another simple type of 2nd order forecast is "What will [Forecaster]'s forecast be on [Date]?"
Bolton Bailey @ 2024-05-24T17:29 (+3)
In addition to the links I provided in the other comment, people might be interested in reading about a small competition I ran on Manifold last year which tries to capture exactly this "surface correlations" concept.
SummaryBot @ 2024-05-23T15:47 (+1)
Executive summary: Higher-order forecasts, which are forecasts about lower-order forecasts, could improve the efficiency and information aggregation of prediction markets and forecasting systems, analogous to the role of derivatives in financial markets.
Key points:
- Higher-order forecasts are defined as forecasts about lower-order forecasts (e.g., 2nd-order forecasts predict 1st-order forecasts).
- Potential benefits include identifying overconfidence, prioritizing important questions, surfacing relationships between events, enabling faster information aggregation, and leveraging existing prediction platform infrastructure.
- Challenges include the dependence on accuracy of lower-order forecasts, added complexity, and the need for a substantial base of lower-order forecasting questions.
- Alternative names considered include "derivatives," "meta-forecasts," and "higher-layer forecasts."
- The author expects higher-order forecasts to become a key component of mature forecasting systems over time, potentially leading to substantial accuracy and liquidity gains.
This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.