Proposal + Demo: Connect Guesstimate and Metaculus and Turn them into Trees

By Charlie_Guthmann @ 2023-03-25T17:15 (+15)

Partially inspired by this post earlier today. 

Problem: Metaculus and guesstimate don't save information from previous questions for future use in a simple and easy way. 

Let's solve both problems at once with an automated protocol.  

TLDR; Turn Guesstimate into a Tree of Models rather than local models stored together. Integrate this into Metaculus. Create a voting protocol on Metaculus that allows people to weigh how much a model goes into the final forecast. Skip down a bit to see a partial example

Pre Notes: 

• As of now, Guestimate models/sheets don't have the option to have multiple worksheets in one spreadsheet which is a bit problematic. 
• Below my protocol, I'm going to go through an example. It may be easier to just skip to that to understand. 
• We are going to either need (1) much better forecast naming conventions (2) an AI to moderate this (3) A group of dedicated volunteers cleaning the entire system

STEP 1: Open A Forecast

Go to Metaculus and open up a forecast. 

Under my proposal, Metaculus pings guesstimate and asks for all the forecasts it has models on. If this forecast is already a sheet in Guesstimate, Metaculus will pull the sheet and any models in the sheet (1 spreadsheet = 1 question, 1 worksheet = 1 model) put it in a section called the "Model Section" below the comment section. 

If there isn't an existing model, Metaculus will post a new spreadsheet with the forecast title on guesstimate. 

STEP 2: Create a Model

anyone can now go to guesstimate, open up a worksheet in the spreadsheet, and create a model. The Model *COULD* be as simple as citing a single source and have no further inputs or relationships. However, it could be much more complicated. 

Here's where my aha moment occurred. If you create a model to predict something, the (exogenous) inputs that you have in your model are themselves forecasts also. Hence, every time someone creates a new input in their model, it... OPENS another spreadsheet named after that forecast (and on and on). Then, it pings back to Metaculus and asks Metaculus if this forecast already exists. If it doesn't It opens a new forecast on Metaculus.

In order for this to work, every time you add an input to your model, you must make it well-defined, so that there is agreement on what the forecast is saying. A gold standard would be building each question as a series of statements and defining it all the way to the bottom a la lean, but this is too complicated for now. 

STEP 3: Vote for the Models on Metaculus

Every work sheet in the spreadsheet will now show up on metaculus. In addition to forecasting on the question itself, you can now vote on the models. This gets a little bit technical now but I won't get into detail. In short, at first, the guesses on the top level question will get most of the weight in terms of the final forecast. As people vote on the models, and as the inputs of the models are forecasted on, the models will gain a larger share of the total weight that goes into the final forecast.

Alternatively, voting could just serve to bring attention to the best models and we could leave the group-level forecasting as is on Metaculus. 

A Step By Step Example of Guesstimate Tree

I (chatgpt) created a shitty script that you can put in the google scripts API on top of google sheets to vaguely recreate this. Please keep in mind that in my protocol, each new question creates a new spreadsheet. However, for the sake of simplicity, my script just opens new worksheets in the same spreadsheet.

We are going to recreate the 'Deep Work' Guesstimate, which aims to guess how many hours of work someone has left in their life. 

First, we are going to open up a new spreadsheet, and put the top level question in the first worksheet

It’s completely blank except for the border in the top left. This is where we will put the final answer. 

The deep work model has three exogenous inputs (Current Age of Human, Potential Age for Death of Mind, Deep Work Output per Day). What we are going to do is write these inputs down in the first column. My code will then check if the name we write down matches the name of any other worksheet in this spreadsheet. It will only run this functionality in the first column. 

Since none of these inputs had worksheets already, a new one is automatically opened. In theory, we would also define these inputs somewhere. Then, this would also open a question in Metaculus for each of these things (well not the age one, for obvious reasons) and use the definition we provided. (the yellow should just be a border, but it’s the answer box all the same). Now we can create a model on this sheet as before. For simplicity, I’m just going to fill out the answer boxes instead. But the main point is that we have a ever-branching tree of models, so try to use your imagination.

Now that we have put a number into the answer box, the code will check if there are any boxes that have strings that match the name of the sheet (in theory, my code doesn’t actually do this - you have to rewrite the string). If they do, the number in the answer box of the worksheet autopopulates the box to the right of said boxes. 
 

Finally, we add the finishing touches - intermediate inputs (don't put these in the first column unless you want to open up another sheet), and the final.

Now any model that uses Hours of deep work left will automatically pull 21900. 

Integrating this into guesstimate and then integrating that into Metaculus is above my paygrade but hopefully, you can kind of see how this comes together.