Features that make a report especially helpful to me

By lukeprog @ 2022-04-12T13:57 (+107)

Follow-up to: EA needs consultancies

Below is a list of features that make a report on some research question more helpful to me, along with a list of examples.

I wrote this post for the benefit of individuals and organizations from whom I might commission reports on specific research questions, but others might find it useful as well. Much of what's below is probably true for Open Philanthropy in general, but I've written it in my own voice so that I don't need to try to represent Open Philanthropy as a whole.

For many projects, some of the features below are not applicable, or not feasible, or (most often) not worth the cost, especially time-cost. But if present, these features make a report more helpful and action-informing to me:

  1. The strongest forms of evidence available on the question were generated/collected. This is central but often highly constrained, e.g. we generally can't run randomized trials in geopolitics, and major companies won't share much proprietary data. But before commissioning a report, I'd typically want to know what the strongest evidence that could in theory be collected is, and how much that might cost to gather or produce.

  2. Thoughtful cost-benefit analysis, where relevant.

  3. Strong reasoning transparency throughout, of this particular type. In most cases this might be the most important feature I'm looking for, especially given that many research questions don't lend themselves to more than 1-3 types of evidence anyway, and all of them are weak. In many cases, especially when I don't have much prior context and trust built up with the producers of a report, I would like to pay for a report to be pretty "extreme" about reasoning transparency, e.g. possibly:

    • a footnote or endnote indicating what kind of support nearly every substantive claim has, including lengthy blockquotes of the relevant passages from primary sources (as in a GiveWell intervention report[1]).
    • explicit probabilities (from authors, experts, or superforecasters) provided for dozens or hundreds of claims and forecasts throughout the report, to indicate degrees of confidence. (Most people don't have experience giving plausibly-calibrated explicit probabilities for claims, but I'll often be willing to provide funding for explicit probabilities about some of a report's claims to be provided by companies that specialize in doing that, e.g. Good Judgment, Metaculus, or Hypermind.)
    • Lots of appendices that lay out more detailed reasoning and evidence for claims that are argued more briefly in the main text of the report, a la my animal consciousness report, which is 83% appendices and endnotes (by word count).

  4. Authors and other major contributors who have undergone special training in calibration and forecasting,[2] e.g. from Hubbard and Good Judgment. This should help contributors to a report to "speak our language" of calibrated probabilities and general Bayesianism, and perhaps improve the accuracy/calibration of the claims in the report itself. I'm typically happy to pay for this training for people working on a project I've commissioned.

  5. External reviews of the ~final report, including possibly from experts with different relevant specializations and differing/opposed object-level views. This should be fairly straightforward with sufficient honoraria for reviewers, and sufficient time spent identifying appropriate experts.

Some of the strongest examples of ideal reports of this type that I've seen are:

  1. GiveWell's intervention/program reports[3] and top charity reviews.[4]

  2. David Roodman's evidence reviews, e.g. on microfinance, alcohol taxes, and the effects of incarceration on crime (most of these were written for Open Philanthropy).

Other examples include:

  1. The most-developed topic articles by Our World in Data, e.g. on COVID-19, population growth, life expectancy, and extreme poverty.
  2. Various Open Philanthropy reports that are more speculative and/or less thorough than e.g. Roodman's reports, for example our "medium-depth" cause reports here, some reports speculating about the future of AI (e.g. 1, 2, 3, 4, 5), and my report speculating about animal consciousness.
  3. A few reports by Rethink Priorities, e.g. on agricultural land redistribution, charter cities, lead exposure, and risks from nuclear weapons.
  4. Some of the systematic reviews by Cochrane and Campbell.

Notes

  1. E.g. see the footnotes at the bottom of this page. ↩︎

  2. Of course, training in statistics, causal inference, critical evidence assessment, and the specific domains relevant to the report are also important, but that training is more difficult to acquire rapidly. ↩︎

  3. Click the "(More)" links in the "Description of program" column here. ↩︎

  4. Click the "Full Research Report" links here. ↩︎

Marcel D @ 2022-04-12T14:40 (+7)

Re: thoughtful cost benefit analysis:

One of my pet peeves is when a report from e.g., a think tank spends about 80% of the report body describing the problems/situation, then concludes with a ~short segment on “policy recommendations,” which often lacks the kind of basic structured analysis/assumption support that is commonplace in e.g., some high school competitive policy debate leagues.

I’m often not asking for some formalized cost-benefit analysis with e.g., discount rates—especially since some policies/outcomes are not amenable to neat quantification, and some think tank reports are meant to be more exploratory rather than definitive (although if that is the case then the report should not push the policy recommendations so forcefully, IMO). However, I’ve seen many reports where the recommendation section barely considers alternative analysis, and I suspect that this is partially because there is not a Schelling/focal point for audience and writer expectations/norms.

In contrast, various leagues of competitive policy debate (e.g., in high school) have developed norms around concepts like the “stock issues”, which I generalized beyond policy debate in this post. Specifically, for each advantage (or disadvantage) claimed for a given proposal (e.g., “this plan reduces pollution of X and thus saves Y lives”), one should critically consider whether/to what extent:

  1. the outcome would occur counterfactually;
  2. the proposal would be implemented/feasible as described;
  3. a given implementation of the proposal would lead to the outcome; and
  4. how morally significant the outcome is.

Similar to how the importance-tractability-neglectedness framework is common parlance among EAs for broad cause prioritization, I think it would be good to see a similar common framework be developed/used to e.g., check against one’s own biases and break down alleged pros and cons into smaller pieces. And I certainly would like to see more think tank/etc. reports at least toying with questions like “would this proposal actually mitigate the problem (beyond what would counterfactually occur)?”