What stops you doing more forecasting?
By Nathan Young @ 2021-11-16T00:26 (+11)
What stands in the way of you doing more forecasting on EA topics? If you started, what made you stop?
Nathan Young @ 2021-11-16T00:31 (+8)
Scoring systems
I like to feel like additional effort makes a difference. The easiest way to get to the top of the metaculus leaderboard is to forecast on everything. Not well, but just to do it. This is pretty dispiriting.
JoshYou @ 2021-11-16T01:58 (+1)
I did Metaculus for a while but I wasn't quite sure how to assess how well I was doing and I lost interest. I know Brier score isn't the greatest metric. Just try to accumulate points?
Nathan Young @ 2021-11-16T00:27 (+8)
Not being able to mark predictions as quick and slow.
Sometimes I spend a long time on a forecast (10-20 mins) sometimes I spend about 30s. It frustrates me that all the current sites don't let me track these two quite different types.
Jackson Wagner @ 2021-11-16T02:39 (+2)
Forecasts are rather time-intensive -- I used to make a bunch of metaculus forecasts, but putting serious effort into researching something, just to come up with a number to toss into the community mix, rarely seems worth the effort these days. Instead, I have a backlog of blog posts and other projects that seem higher expected value. (But going forwards, I should try to contribute forecasts when I find myself already researching something EA-adjacent anyways.)
Related to this, unclear impact -- as Nuno judges here, it's hard for forecasts to really hit the sweet spot of being at the right difficulty level and also being decision-relevant. If I had more of an understanding of who finds forecasts useful, I might be more interested in contributing.
D0TheMath @ 2021-11-17T01:00 (+1)
Overthinking forecasts, causing writing them down & tracking them diligently to be too much of a mental-overhead for me to bother with.
Jackson Wagner @ 2021-11-16T02:48 (+1)
As you say, there's a tension between effortful vs informal forecasts: Out of humility / modesty / imposter syndrome, sometimes I don't want to mess up a pool of expert forecasts by tossing out an ill-informed hunch.
I think Metaculus could benefit from adopting the more stock-market / prediction-market style mechanism of letting players stake different amounts of their total forecasting "budget" on individual questions... prediction-market style, experienced players or players who wanted to make a very confident prediction could pour lots of points into the system. Then I would be better able to distinguish my confident from my less-confident predictions, and small numbers of experts could bet their beliefs to overturn an ill-judged majority opinion.
Although I recognize that Metaculus's current, more egalitarian style probably helps to grow the userbase and encourage participation.