Research idea: Evaluate the IGM economic experts panel

If you read the expert comments, very often they complain that the question is poorly phrased. It's typically about wording like "would greatly increase" where there's not even an attempt to define "greatly". So if you want to improve the panel or replicate it, that is my #1 recommendation.

...My #2 recommendation is to create a Metaculus market for every IGM question and see how it compares.

This does seem useful. At least one similar survey does exist for other fields: the TRIP survey for international relations scholars. I've found this somewhat useful for my research, though often the questions in IR seem less specific than the questions asked of economists.

Thanks for this post.

Here's a copy of some quick notes I wrote earlier (and don't plan to take any next steps on), in case this is useful to anyone:

• "I see results from the IGM economics expert panels cited somewhat often, and it seems like that's probably a really useful thing to have.
• Idea 1: Does anyone know if there are similar things for other disciplines? 
  • E.g., historians, psychologists, international relations scholars?
  • And if there isn't, might it be highly valuable per unit effort to set that up?
  • Obviously lots of individual experts comment on individual things, and there'll sometimes be surveys or the like. But it seems like it could make sense for this sort of thing to be institutionalised in the way the IGM panels are, rather than being ad hoc.
• Idea 2: Does anyone know if it's possible to submit questions to the IGM panel or other panels (if they exist), or to in some other way get them to respond to questions on topics of particular interest to EAs?
• (If people think this sort of thing doesn't exist yet but might be worth thinking more about, I might write a quick EA Forum post/question post/shortform on this.)"

Replies from colleagues of mine included:

• “In the US, the National Academy of Sciences, Engineering and Medicine does expert opinion in a more formal way for relevant issues e.g., https://www.nap.edu/catalog/25665/heritable-human-genome-editing”
• “You hear a lot in the US from the US Army Corps of Engineers.”
• “PhilPapers seems like a natural place to expand polls of philosophers.”
• “I don't know about historians, but wouldn't it be cool if macrohistory/predictive history were more of a thing and they took surveys of their members?”

I think the IGM panel is great, and expanding it within economics or extending it to other fields would be great. Some quick thoughts:

It's impossible to objectively retrospectively evaluate panel responses, because all of the interesting questions are about causal effects of X on Y, where the answer doesn't become clear over time because the counterfactual is never observed. Maybe the best you could do is run a retrospective survey where you ask panel members (or other economists) whether they think the panel responses were right in hindsight.

I'm not surprised that Tyler Cowen has a negative view of many of the panel results; he's a libertarian and the panel is broadly left-leaning, as is the broader economics profession. I think a better critique is one that Cowen expresses, e.g., in his interview on Spencer Greenberg's podcast: that the views of economists seem to covary pretty closely with the political beliefs of their demographic group (highly educated urban dwellers), suggesting that their views aren't purely reflective of professional expertise.

I also don't think it's concerning that the IGM forum "excludes certain schools of economics, like the Austrian school." The economics profession doesn't really have "schools" anymore, and I don't think the "Austrian school" has ever had much mainstream credibility. The first and third bullet points from the 2016 paper are more concerning.

I don't think the IGM panel influences future research topics, as the questions are typically about topics already highly salient to academic economists. I don't know whether it influences public discourse or policymakers. The panel has high buy-in from academics because its members are all highly regarded in the profession.