Accuracy Agreements: A Flexible Alternative to Prediction Markets

By Ozzie Gooen @ 2023-04-20T03:09 (+37)

This is a linkpost to https://quri.substack.com/p/accuracy-agreements-a-flexible-alternative

Epistemic Status: This concept is early and speculative

TLDR: Accuracy Agreements are contracts in which one party commits to remunerate the other based on a predetermined rate, proportional to the accuracy of a series of forecasts supplied by the contract owner. This incentivizes the owner to find cost-effective methods for producing calibrated and accurate forecasts.
 

The Basics

The problem: Prediction Markets are effective at determining outcomes for binary questions, but they struggle with probability distributions and more expressive situations, such as those involving functions. Also, often there are only a few potential organizations that manage estimation, and it’s simpler to pay one for estimation than it is to set up a market.

Proposed solution: A client publishes an "Accuracy Agreement" with an associated "Type Specification."

A simple agreement's terms are as follows:

1. A purchaser of the agreement provides a one-time or continuously-updating forecast that complies with the established Type Specification. 
2. Upon reaching a prearranged future date, the Agreement will undergo Resolution, which entails:
   1. The client will determine the answers to the forecasting questions.
   2. The client will evaluate and assign a score to the submitted forecast.
   3. The contract holder at the time of resolution will receive payment based on a function of a scoring rule. An example of this could be: "The average log score of the forecast minus the prior, multiplied by $10,000. Stated differently, $10,000 per bit of information.”

Holding an "Accuracy Agreement" with a specific forecast has a calculable value. If the holder trusts the forecast, the value would equal the expected resolution payment. (Easily calculated using expected loss).

One elegant way the client could ensure a reasonable price is by selling the contract, initially, to the winner of a bidding system. When a question is proposed, potential buyers bid for the Agreement. If a bidder believes their forecast has an EV of "$25,000," they might bid "$23,000" for the Agreement. The winner would either need to submit a prediction immediately, or would have time to generate one.

An Example:

A client publishes:

1. A list of 50 continuous variables, to be resolved in 1 year.
   1. A set of (weak) priors for those variables.
   2. A simple scoring function, e.g., $1K * total logScore sum.
   3. A future date for bidding to commence (1 month out).
2. Bidders submit:
   1. A list of forecasts for the 50 variables, each as a probability distribution.
   2. A maximum price for purchasing the agreement with their forecast.
3. The top bidder is chosen and sold the contract.
4. In 1 year, the contract resolution occurs.
   1. All (or some) of the variables are resolved.
   2. The contracts’ forecasts get scored.
   3. The agreement owner is paid according to the predetermined scoring rule

Potential Changes / Improvements

1. An active marketplace. Purchasers can trade these contracts with each other over time, after the initial purchase.
2. Introduce a market with multiple shares, allowing buyers and sellers to purchase only a portion of the market. This can help ensure that no one purchaser can overbid for the privilege of promoting their (bad) forecast.
3. Buyers can update their forecasts over time. The contracts cover the average accuracy over that time. This incentivizes buyers to continue to improve their contracts.
4. Forecasting questions can be resolved and scored by agreed-upon third parties.
5. There’s an extra reward for a forecast that comes with a good explanation or similar. This could be part of the scoring function.
6. In some cases, it might make sense for bidders not to have to put any money down. This might be considered gambling. One alternative, if the bidders are trusted, is that they get payments in two parts: one fixed fee, and one Accuracy Agreement. For example, some EA org puts out a call for a forecasting consultancy to forecast 1000 variables - but most of the payment is in the form of an Accuracy Agreement.

Powering Accuracy Agreements with Prediction Markets

Would information purchasers get a better return by subsidizing Prediction Markets, than they would by selling Accuracy Agreements?

I think this difference should be bounded. If/when it is the case that Prediction Markets are more efficient than “Accuracy Agreements”, then the owner of the Accuracy Agreement could just set up Prediction Markets once they own the agreement. This owner could subsidize the market in whatever way would do the best job.

For example, say a purchaser sees an agreement that could make them $50,000 profit without a Prediction Market, or $100,000 if they spend $20,000 on setting up and subsidizing a corresponding Prediction Market. They might bid $60,000 on the contract, and if they get it, then spend $10,000 on the market.

This setup would help encourage both good Prediction Markets, and encourage actors who are savvy about properly setting up and subsidizing said markets.

Bottlenecks

The first bottleneck to actually selling Accuracy Agreements is the legality. It might be tricky to do in a way that’s not considered gambling.

The second bottleneck is finding buyers. Accuracy agreements will be somewhat complicated assets. You’d need sophisticated estimation agencies to do competent estimation and to have the flexibility to take the necessary financial risk.

The third challenge is just the infrastructural and logistical expenses. Setting up software to make this process smooth, and making sure that questions get resolved by third parties, would require some overhead, especially early on.

Next Steps

To make progress, I’d like to see this sort of setup be tested in simple ways. First, talk with lawyers to best maneuver around the first bottleneck. If any readers of this have insight here, please leave comments!

Second, experiment with small contracts (maybe <$5k) and small consultancies. This should be achievable with a small team with a decent reputation with potential estimation consultancies. I could see QURI doing this in the future but I would be happy to see other groups do so first. Early contracts don’t need to allow for themselves to be sold, they can be simple agreements with individual consultancies.

One nice property of these contracts is that they should be able to scale gracefully, in that you could do them on small scales or large scales. So, start small, and as we gain experience, we’ll be able to better scale these setups with larger sums and more diverse Accuracy Agreement purchasers.

Thanks to Javier Prieto and Nuño Sempere for feedback