Summary: Evidence, cluelessness and the long term | Hilary Greaves

By Vasco Grilo @ 2022-04-15T17:22 (+6)

This is a summary of the talk Evidence, cluelessness and the long term | Hilary Greaves | EA Student Summit 2020. The full transcript is available here.

Part 1: effectiveness, cost-effectiveness, and the importance of evidence

• Effectiveness:
  • Many well-meaning interventions do not work at all.
  • Examples:
    • Playpums are inferior to the original hand pumps they replaced.
    • At least in one study, distributing free sanitary products had no effect on the school attendance of teenage girls.
• Cost-effectiveness:
  • Of the interventions that do work, there are differences in cost-effectiveness spanning many orders of magnitude.
  • So it is critical to focus on what works best.
• The importance of evidence:
  • It is very hard to know a priori which interventions are going to cause which outcomes.
  • If you want to know what actually works (and how well it works), you have to pay a close attention to the evidence (ideally, randomised controlled trials).
  • Examples:
    • In medicine, there has been a tendency towards evidence-based treatments, where decisions are backed up by careful attention to randomised controlled trials.
    • In the field of altruistic endeavour, there has been a tendency towards paying close attention to randomised controlled trials (spearheaded by organisations such as GiveWell).

Part 2: the limits of evidence

• "Simple cost-effectiveness analysis":
  • In such analysis, one only measures the immediate intended effect of the intervention: e.g. child years spent in school.
  • This allows two kinds of comparisons:
    • Intra-cause comparisons: e.g. which intervention increases child years spent in school the most?
    • Cross-cause comparisons: e.g. how should we trade off additional child years spent in school against improvements in clean water consumption?
• What the simple analyses omit:
  • Knock-on/Flow-through effects:
    • These are effects that are causally downstream of the intended effect: e.g. an intervention whose goal is increasing child years spent in school has downstream consequences on economic prosperity, population size, and politics.
  • Side effects:
    • These are effects of the intervention that do not go via the intended causal route: e.g. provision of healthcare services by Western funded charities might decrease the tendency of the local population to lobby their own governments for adequate health services.
• Cluelessness:
  • The unmeasured knock-on and side effects are, in expectation, almost certainly greater in aggregate than the measured effects.
    • Hopefully, if all goes well, there are thousands of future generations.
    • It seems extremely unlikely that the mere 60 (or so) life years gained in the life of one person whose premature death was averted are more valuable than the effects on population size down the millennia.
  • These further future (causally downstream or otherwise) events are much harder to estimate: e.g. you cannot do a randomised controlled trial to ascertain what the effect of your intervention is going to be in 100 years).
  • These further future and relatively unforeseeable effects, in principle, matter from an altruistic point of view: e.g. you would have overwhelming moral reason not to press a button which you knew (never mind how) would trigger a nuclear explosion going off in two thousand years time, killing millions of people.

Part 3: five possible responses to cluelessness

1. Make the analysis more sophisticated:
   • GiveWell is very much to be applauded for having done this.
   • However, it only relatively slightly shifts the boundary between the things that we know about and the things that we are clueless about (e.g. it says basically nothing about the effects on population size down the generations).
2. Give up the effective altruist enterprise:
   • "Why would I make big sacrifices, if I am radically uncertain whether they would even do any good at all?".
   • Let us think (and hope) this is not ultimately the right response, but it is an understandable one.
3. Make bolder estimates:
   • In principle, one could build a model that takes into account all distant future effects (e.g. effects on population size, and value of changes to the population size).
     • There would be questions where there is relatively little guidance from evidence, and where one feels much more like guessing.
   • Intra-personal issue:
     • One is not going to be able to shake the feeling that, when writing down a particular uber-analysis, some really arbitrary decisions were made.
     • It was pretty arbitrary, perhaps, that one came down on the side of increasing population size being good rather than bad.
   • Inter-personal issue:
     • Suppose one person chooses to go all out on increasing future population size, and another chooses to go all out on decreasing future population size.
     • They would perhaps have done something much more productive if they got together and had a conversation, and decided to instead fund some third thing that at least the two could agree upon.
4. Ignore things we cannot even estimate:
   • Consider the most sophisticated, plausible cost-effectiveness analysis, perhaps like the GiveWell 2020 analysis.
     • The analysis stopped at the point where we are making some educated guesses, and we can also do our sensitivity analysis to check that our important conclusions are not too sensitive to reasonable variations in the input parameters for this medium complexity cost-effectiveness model.
   • This is tempting, but does not seem right.
5. Go longtermist:
   • If longer-term effects supply most of the expected value, but longer-term effects for "short-termist" interventions are utterly unpredictable, then perhaps we would do better to focus on interventions whose effects on the further future are more predictable.
   • Examples:
     • Reducing the chance of premature human extinction.
     • Improving very long run average future welfare, conditional on the supposition that humanity does not go prematurely extinct, perhaps by improving the content of key, long lasting political institutions.

Summary

• Part 1: effectiveness, cost-effectiveness, and the importance of evidence.
  • Most well-intentioned things do not work.
  • Even among the things that do work, some work hundreds of times better than others.
  • We have to pay attention to evidence if we want to know which are which.
• Part 2: the limits of evidence.
  • Evidence, kind of necessarily, only tracks relatively near term effects.
  • Plausibly, the bulk of even the expected value of our interventions comes from their effects on the very far future (which are not measured in even the more complicated, plausible cost-effectiveness analysis).
• Part 3: five possible responses to cluelessness.
  • Responses:
    • Make the cost-effectiveness analyses somewhat more sophisticated.
    • Give up effective altruism.
    • Do the uber-analysis.
    • Adopt a parochial form of morality where you only care about the near-term, predictable effects.
    • Shift to interventions that are explicitly aimed at improving, as much as we possibly can, the expected course of the very long run future.
  • We need to do a lot more thinking and research about this, what motivates the enterprise that we call global priorities research.