Five Ways to Handle Flow-Through Effects

By Peter Wildeford @ 2016-07-28T03:39 (+40)

As effective altruists, we are interested in finding opportunities to produce the highest expected moral value per unit of cost. This is hard enough even when just looking at short-term effects --good giving opportunities are scarce,good science is difficult, errors can go uncorrected for awhile, and it’s hard to even figure out what kind of metric we really want to be improving.

But even if we were to somehow figure out an accurate view of the short-term effects of a particular opportunity, we have no guarantee that effects other than the direct, measurable short-term effects will also work out in our favor. These additional effects -- such as medium-term effects, long-term effects, and other indirect effects not captured in the initial impact analysis -- are collectively referred to here as “flow-through effects” -- and can significantly change how we think about a particular cause. (This terminology may be subpar , but I use it anyway because it is what is commonly used in the EA movement right now and there’s no agreed upon replacement.)

For example, even while AMF likely has very valuable short-term effects, could it be creating harmful overpopulation in the medium-term ( likely no)? Or could it be creating a net population loss, thereby decreasing the amount of happy people in the world ( possibly yes)? And in the long-term, could the economic development caused by AMF just be making it more likely we die in some technological catastrophe ? Or is economic development positive for the world in the long-run?

Additionally, some have argued that people and nonhuman animals who will live in the far future (but don’t yet exist) still have significant moral weight. Given that they will (hopefully) exist in much more massive quantities than those currently alive, whatever impact we have on those in the far future, however miniscule now, may add up to a much larger overall effect than our effect on those alive now . This would also suggest we need increased attention to long-term effects.

Dealing with these potential problems is a very difficult challenge because these problems are often not well resolved by direct empirical analysis (you can’t just easily wait 100 years to see how things turn out) and because such crucial considerations can wildly affect our choices, making AMF either highly beneficial or even net negative.

While flow-through effects will still require significant analysis, it is important to make sure they do not paralyze our decisions. Thus, in the meantime, I suggest five non-exclusive potential ways to handle flow-through effects going forward. This isn’t intended to be a fully comprehensive evaluation, but hopefully offers an initial perspective that can start discussion about how to better handle flow-through effects, which I think is still a developing part of EA discourse.

Ignoring flow-through effects

The simplest strategy and possibly the most common one in the charity sector is to not take any flow-through effects into account when making decisions. This could be because of a belief that flow-through effects are too complicated and intractable to be worth working with and/or because having a good short-term effect is taken as the best proxy for having good other effects. One might also think that the flow-through effects are likely to be much smaller in magnitude than the direct, short-term effects, perhaps because these effects diffuse quickly. Proximate consequentialism, a version of consequentialism that looks only at immediate effects, basically codifies this “ignore” reasoning into moral philosophy.

Pros

• Very quick and easy

• Avoids adding very speculative elements into your impact calculations

Cons

• Very likely to miss large, clear, and important effects that greatly affect expected value

Qualitative evaluation

Is improving economic development net negative? While differential technological progress may be a concern, GiveWell has offered a few reasons to think economic development is positive. This approach involves making a best guess about how a particular flow-through effect will adjust your impact calculation and adjusting accordingly, perhaps using a “many weak arguments” framework. Making such a guess can outperform ignoring without requiring a huge amount of effort on a potentially intractable issue.

Pros

• Relatively quick to perform

• Intuitively appealing

Cons

• Probably less accurate than more refined methods

• Guesses may provide no real value but give a false sense of confidence

• Can lead to only taking into account the most noticeable flow-through effects

• May invite bias

Weighted quantitative modeling

Similar to guessing, one can implement a weighted model based on our confidence in the direction (positive vs. negative) and magnitude of each potential flow-through effect. Basically it entails weighing the flow-through effects based on not only how important they are but also how much evidence or confidence one can have in them.

To use some fictitious numbers, imagine that AMF has a short-term impact of +1, a medium-impact of +10, and potential long-term impacts of either +100 or -100 depending on whether economic development is good or bad. If we’re 60% sure economic development is net good, we can transform the long-term impact as (0.6)(100) + (0.4)(-100) = 20. Additionally, if we’re 10x more confident in the short-term projections as the medium-term projections and 2x more confident in the medium-term projections as the long-term projections, we could adjust by this to create a total impact of +1 + (1/10)(+10) + (1/10)(1/2)(+20) = +3.

Of course, these two adjustments are very simplistic and more time should be put into turning this into some sort of model, perhaps based on priors and Bayesian adjustment. There are many such ways to make these adjustments and it is not clear yet which way is the most correct.

Pros

• Very speculative flow-through effects do not dominate the cost-effectiveness calculation

• Takes a medium amount of time relative to other options

• Seems like a good trade off between time and results

Cons

• Could mistakenly minimize the effect of a very important flow-through effect

• May lead to estimates based on biased, wishful thinking

Conducting research

Another approach would be to list the potential flow-through effects and try to research them as much as is feasible, even if the questions risk being intractable. For example, those worried about overpopulation could expand upon GiveWell’s review. Other questions may be much harder to research, such as differential technological progress and economic development, though GiveWell did come up with an initial framework for thinking about it , and one could potentially try to do more careful historical analysis on similar situations (e.g., improvement in nuclear capabilities vs. improvement in nuclear safety / disarmament, improvements in DNA research vs. safeguards against genetically engineered pandemics). Carl Shulman has offered numerous proxy variables we could attempt to measure and connect causally with various interventions. There is also the potential for meta-research into ways others have dealt with flow-through effects.

Pros

• More empirically grounded than guesswork

• More verifiable and specific than guesswork

• Can potentially make use of existing empirical research

Cons

• Very slow

• Very costly, both in time and money

• May be practically impossible for some (or even most) questions

• May not produce an answer that significantly improves upon guesswork

Prioritizing robustness

Finally, another approach to handling flow-through effects is to pick a cause that is believed to be net positive , even if a wide variety of flow-through effects end up being important. For example, it seems much less likely that work spent to improve developed world education, international cooperation, or philosophical thoughtfulness will be net negative (even if they do have a sizable risk of being net zero).

Pros

• Safer from many different value and epistemological stances

• Least likely to end up doing massive harm

Cons

• It’s hard to determine what is a robust cause

• We don’t yet have a good framework for navigating trade-offs between cause effectiveness and robustness

• Does not take into account relative scale of flow-through effects

• Might lead to overall worse cost-effectiveness if we trade away direct impact to avoid the risk of negative flow-through effects. For example, many developed world education initiatives end up being very expensive and produce little to no results, even if they may have reduced risk of negative flow-through effects.