My current best guess on how to aggregate forecasts

It's not clear to me that "fitting a Beta distribution and using one of it's statistics" is different from just taking the mean of the probabilities.

I fitting a beta distribution to Metaculus forecasts and looked at:

• Median forecast
• Mean forecast
• Mean log-odds / Geometric mean of odds
• Fitted beta median
• Fitted beta mean

Scattering these 5 values against each other I get:

We can see fitted values are closely aligned with the mean and mean-log-odds, but not with the median. (Unsurprising when you consider the ~parametric formula for the mean / median).

The performance is as follows:

My intuition for what is going on here is that the beta-median is an extremized form of the beta-mean / mean, which is an improvement

Looking more recently (as the community became more calibrated), the beta-median's performance edge seems to have reduced:

Hmm good question.

For a quick foray into this we can see what would happen if we use our estimate the mean of the max likelihood beta distribution implied by the sample of forecasts $p_1,...,p_N$.

The log-likelihood to maximize is then 

$\log L(\alpha ,\beta )=(\alpha -1){\sum }_i\log p_i+(\beta -1){\sum }_i\log (1-p_i)-N\log B(\alpha ,\beta )$ 

The wikipedia article on the Beta distribution discusses this maximization problem in depth, pointing out that albeit no closed form exists if $\alpha$ and $\beta$ can be assumed to be not too small the max likelihood estimate can be approximated as $\hat{\alpha }\approx \frac{1}{2}+\frac{{\hat{G}}_X}{2(1-{\hat{G}}_X-{\hat{G}}_{1-X})}$  and $\hat{\beta }\approx \frac{1}{2}+\frac{{\hat{G}}_{1-X}}{2(1-{\hat{G}}_X-{\hat{G}}_{1-X})}$, where $G_X={\prod }_i{p}_i^{1/N}$ and $G_{1-X}={\prod }_i(1-p_i{)}^{1/N}$.

The mean of a beta with these max likelihood parameters is $\frac{\hat{\alpha }}{\hat{\alpha }+\hat{\beta }}=\frac{(1-G_{1-X})}{(1-G_X)+(1-G_{1-X})}$.

By comparison, the geometric mean of odds estimate is:

$p=\frac{{\prod }_{i=1}^N{p}_i^{1/N}}{{\prod }_{i=1}^N{p}_i^{1/N}+{\prod }_{i=1}^N(1-p_i{)}^{1/N}}=\frac{G_X}{G_X+G_{1-X}}$

Here are two examples of how the two methods compare aggregating five forecasts

I originally did this to convince myself that the two aggregates were different. And they seem to be! The method seems to be close to the arithmetic mean in this example. Let's see what happens when we extremize one of the predictions:

We have made p3 one hundred times smaller. The geometric mean is suitable affected. The maximum likelihood beta mean stays close to the arithmetic mean, unperturbed. 

This makes me a bit less excited about this method, but I would be excited about people poking around with this method and related ones!

Thanks for this post - I think this was a very useful conversation to have started (at least for my own work!), even if I'm less confident than you in some of these conclusions 

Thank you for your kind words! To dismiss any impression of confidence, this represents my best guesses. I am also quite confused.

I've heard other people give good-sounding arguments for other conclusions

I'd be really curious if you can dig these up!

You later imply that you think [the geo mean of probs outperforming the geo mean of odds] is at least partly because of a specific bias among Metaculus forecasts. But I'm not sure if you think it's fully because of that or whether that's the right explanation

I am confident that the geometric mean of probs outperformed the geo mean of odds because of this bias. If you change the coding of all binary questions so that True becomes False and viceversa then you are going to get worse performance that the geo mean of odds.

This is because the geometric mean of probabilities does not map consistently predictions and their complements. With a basic example, suppose that we have $p_1=0.01,p_2=0.3$. Then $\sqrt{p_1\ast p_2}+\sqrt{(1-p_1)\ast (1-p_2)}\approx 0.89<1$.

 So the geometric mean of probabilities in this sense it's not a consistent probability - it doesn't map the complement of probabilities to the the complement of the geometric mean as we would expect (the geometric mean of odds, the mean of probabilities and the median all satisfy this basic property).

So I would recommend viewing the geometric mean of probabilities as a hack to adjust the geometric mean of odds down. This is also why I think better adjustments likely exist, since this isn't a particularly well motivated adjustment. It does however seem to slighly improve Metaculus predictions, so I included it in the flowchart.

To drill this point even more, here is what we would get if we aggregated the predictions in the last 860 resolved metaculus binary questions by mapping each prediction to their complement, taking the geo mean of probs and taking the complement again:

As you can see, this change (that would not affect the other aggregates) significantly weakens the geo mean of probs.

Interesting case. I can see the intuitive case for the median.

I think the mean is more appropriate - in this case, what this is telling you is that your uncertainty is dominated by the possibility of a fat tail, and the priority is ruling it out.

I'd still report both for completeness sake, and to illustrate the low resilience of the guess.

Very much enjoyed the posts btw

(speculating) The key property you are looking for IMO is to which degree people are looking at different information when making forecasts. Models that parcel reality into neat little mutually exclusive packages are more amenable , while forecasts that obscurely aggregate information from independent sources will work better with geomeans. 

In any case, this has little bearing on aggregating welfare IMO. You may want to check out geometric rationality as an account that lends itself more to using geometric aggregation of welfare. 

Depends on whether you are aggregating distributions or point estimates.

If you are aggregating distributions, I would follow the same procedure outlined in this post, and use the continuous version of the geometric mean of odds I outline in footnote 1 of this post.

If you are aggregating point estimates, at this point I would use the procedure explained in this paper, which is taking a sort of extremized average. I would consider a log transform depending on the quantity you are aggregating. (though note that I have not spent as much time thinking about how to aggregate point estimates)