Flimsy Pet Theories, Enormous Initiatives

Very interesting - I've been thinking about a generalized theory of bikeshedding that also applies to careers, where some people will have initial exposure to a career through, say, an internship, and because they then know that topic very well and are ambiguity averse, they'll just continue with it until the end of their lives. Because they do value impact they'll post-hoc rationalize their choice as very important and then fall prey to the sunk cost fallacy.

I had similar thoughts on Gates recently after watching his netflix documentary:

"The Gates foundation focuses on Water, Sanitation and Hygiene (WASH), because diarrheal deaths are about 1m/year. 

They invested quite heavily in this and also seem to routinely leverage money from governments, and influence the discourse on the relative priority of WASH within global development. This could be net negative because global health might not be as effective as other economic development interventions (c.f. the work by Lant Pritchett).

He seems to have spent an extraordinary amount of money on WASH and just generally global development.

What caused him to focus on this? And what is thus the more distal cause for the EA focusing on global health? Thinking about this might uncover non-optimal path dependency. 

There seem to be a few causes:

• because he read a NYT article by Nicholas Christofis on diarrheal disease, which because it affects people directly. 
• because he experienced burnout at Microsoft and wanted to do something more meaningful and direct
• He personally went to India and vaccinated children himself giving him an emotional attachment to the cause

I used to be quite the fan of Gates until now, and though I thought his foundation could have done better if it were more flexible, I always thought he gets things roughly right."

I really like this post, and I think it is a useful way to frame putting EA thinking on a foundation that is minimal and light on more substantive philosophical commitments. You don't have to be a total utilitarian, master of expected utility theory and metanormativism to be an EA. You just have to commit to the premise: if I am going to try to do good, I should  spend some time rationally thinking about what would be the best way to do that. As it is, the vast majority of people don't do this. I think that on reflection most people would agree that the way people think about doing good is really pathological. Most departures from EA thinking are not, then, grand philosophical disagreements; they just stem from the fact that people have gone with their passion or fallen into an area or done what feels right

Small suggestion: drop the Facebook example and find a better one. Facebook was obviously not founded out of a grand prosocial vision, that was a pretty clear case of “greenwashing“ afterwards.

I recently gave a talk on one of my own ambitious projects at my organization, and gave the following outside view outcomes in order of likelihood.

1. The project fails to gain any traction or have any meaningful impact on the world.
2. The project has an impact on the world, but despite intentions the impact is negative, neutral or too small to matter.
3. The project has enough of a positive outcomes to matter.

In general, I'd say that outside view this is the most likely order of outcomes of any ambitious/world-saving project. And I was saying it specifically to elicit feedback and make sure people were red-teaming me morally.

However, it's not specifically clear to me that putting more money into research/thinking improves it much? 

For one thing, again the most likely outcome is that the project fails to gain any traction or have any impact at all, so you need to be de-risking the likelihood of that through classic lean-startup MVP style stuff anyway, you shouldn't wait on that, and spend a bunch of money figuring out the positive or negative effects of an intervention at scale that won't actually be able to scale (most things won't scale).

For another, I think that a lot of the benefit of potentially world changing projects is through hard to reason about flow through effects. For instance, in your example about Andrew Carnegie and libraries, a lot of the benefits would be some hard to gesture at stuff related to having a more educated populace and how that effects various aspects of society and culture. You can certainly create Fermi estimates and systems models but ultimately people's models will be very different, and missing one variable or relationship in a complex systems model of society can completely reverse the outcome.

Ultimately, it might be better to use the types of reasoning/systems analyis that work under Knightian Uncertainty, things like "Is this making us more anti-fragile?  is this effectuative and allowing us to continually build towards more impact? Is this increasing our capabilities in an asymmetric way?" 

This is the exact type of reasoning that would cause someone intuitively to think that space settlements are important - it's clearly a thing that increases the anti-fragility of humanity, even if you don't have exact models of the threats that it may help against. By increasing anti-fragility, you're increasing the ability to face unknown threats.  Certainly, you can get into specifics, and you can realize it doesn't make you as anti-fragile as you thought, but again, it's very easy to miss some other specifics that are unknown unknowns and totally reverse your conclusion.

I ultimately think what makes sense is a sort of culture of continuous oversight/thinking about your impact, rather than specific up front research or a budget. Maybe you could have "impact-analysisathons" once a quarter where you discuss these questions. I'm not sure exactly what it would look like, but I notice I'm pretty skeptical at the idea of putting a budget here or creating a team for this purpose. I think they end up doing lots of legible impact analysis which ultimately isn't that useful for the real questions you care about.

Meta: thanks for turning this into a forum post! It seems like it's sparked good discussion that wouldn't have happened if it was solely on Facebook.

Great article!

At risk of losing my "EA Card" and being permanently cast out of this community with no recourse, could we perhaps maybe "red team" EA itself? [ducks]

My take is that visionary engineers tend to start by imagining an interesting mechanism they could build, and then hunting for inspiring justifications so that people will give them the resources to do it.

Facebook, space colonies, neurolink, and OpenAI all sort of fit the bill.

There’s a trust that technology is usually good, and that the hard thing is to find an interesting mechanism, coordinate, inspire, and accomplish building it. Once done, people will find uses for it. At least the sales team will find clients, anyway.

Engineering is, in what I’ve experienced as the culture of engineers, the process of realizing the things that can be build.

World betterment happens on its own, because technology is usually good. If not, can be debugged or improved with further technology.

I think this is true-ish. It’s really pretty hard to imagine a novel mechanism that has any utility at all, and get it working. Especially if you need lots of money and other people to work with you.

In a way, it’s even better to lean on a simple story you can think up in 10 minutes to justify your project. Because that’s how much time your funders will spend. And the public. And regulators. There’s a strong short-term incentive to pursue projects that seem best after the least amount of thought.

The long term is made out of a series of short terms.

Of course, we know the failure modes. Externalities, existential threats, hijacking of human psychology, and regulatory capture are a few.

But if you want to understand the reason why engineers act this way, perhaps one place to start is by imagining that tractable, interesting, juicy ideas in a person’s skill set are not very fungible and pretty uncommon.

If you then wanted to better steer these sorts of decisions, how would you do that? How would you get the engineers on board?

I strongly agree with this post and it's message.

I also want to respond to Jason Crawford's response. We don't necessarily need to move to a situation where everyone tries to optimize things as you suggest, but at this point it seems that almost no one tries to optimize for the right thing. I think even changing this to a few percents of entrepreneurial work or philanthropy could have tremendous effect, without losing much of the creative spark people worry we might lose, or maybe gain even more, as new directions open.

Also, Facebook does research its own technology’s effects on users. I can’t vouch for the quality of the research, but it exists.

https://www.google.com/amp/s/www.wsj.com/amp/articles/facebook-knows-instagram-is-toxic-for-teen-girls-company-documents-show-11631620739

I actually find it quite easy to believe that Musk's initiatives are worth more than the whole EA movement - though I'm agnostic on the point. Those ideas exist in a very different space from effective altruism, and if you fail to acknowledge deep philosophical differences and outside view reasons for scepticism about overcommitting to one worldview you risk polarising the communities and destroying future value trades between them. For example:

• Where EA starts (roughly) from an assumption that you have a small number of people philosophically committed to maximising expected welfare, Musk's companies start with a vision a much larger group of people find emotionally inspiring, and a small subset of them find extremely inspiring. Compare the 35-50 hour work weeks of typical EA orgs' staff vs the 80-90 common among Tesla/SpaceX employees - the latter seem to be far more driven, and I doubt that telling them to go and work on AI policy would a) work or b) inspire them to anywhere near comparable productivity if it did.
• Musk's orgs are driven by a belief that they can one day make a profit from what they do, and that if they can't, they shouldn't succeed.
• Most EA orgs have no such market mechanism, even in the long term. And EA research has perverse incentives that we rarely seem to recognise - researchers gain prestige for raising 'interesting questions' that might minimally if at all affect anyone's behaviour (eg moral uncertainty, cluelessness, infinite ethics, doomsday arguments etc), and they're given money and job security for failing to answer them in favour of ending every essay with 'more research needed'.
• In particular they're incentivised to produce writings that encourage major donors to fund them. One plausible way of doing this for eg, is to foster an ingroup mentality, encouraging the people who take them seriously to think of themselves as custodians of a privileged way of thinking (cf the early rationality movement's dismissal of outsiders as 'NPCs'). I don't know of any meta-level argument that this should lead to more reliable understanding of the world than, say the wisdom of crowds.
• As Halffull discussed in detail in another comment, Musk's initiatives are immensely complicated, and a priori reasoning about them might essentially be worthless. We could spend lifetimes considering them and still not have meaningfully greater confidence in their outcomes - and we'd have marked ourselves as irrelevant in the eyes of people driven to work on them. Or we could work with the people who're motivated by the development of such technologies and encourage what Halffull's 'culture of continuous oversight/thinking about your impact' - which those companies seem to have, at least compared to other for-profits .
• Empirically, the EA movement has a history of ignoring or rejecting certain causes as being not worthy of consideration, then coming to view them as significant after all. See GWWC's original climate change research which basically dismissed the cause vs Founders Pledge's more recent research which takes it seriously as one of their top causes, Open Philanthropy Project's explicit acknowledgement of their increasing concern with 'minor' global catastrophic risks, or just see all the causes OPP have supported with their recent grants (how many EAs would have taken you seriously 10 years ago if you'd thought about donating to US criminal justice reform?). I would say we have a much better track record of unearthing important causes that were being neglected than of providing good reasons to neglect causes.

I think that it's not always possible to check that a project is "best use, or at least decent use" of its resources. The issue is that these kinds of checks are really only good on the margin. If someone is doing something that jumps to a totally different part of the pareto manifold (like building a colony on Mars or harnessing nuclear fission for the first time), conventional cost-benefit analyses aren't that great. For example a standard post-factum justification of the original US space program is that it accelerated progress in materials science and computer science in a way that payed off the investment even if you don't believe that manned space exploration is worthwhile. Whether or not you agree with this (and I doubt this counterfactual can be quantified with any confidence), I don't think that the people who were working on it would have been able to make this argument convincingly at the time. I imagine that if you ran a cost-benefit analysis at the time, it would have found that a better investment would be to put money into incremental materials research. But without the challenge of having to develop insulators, etc., that work in space, there would have plausibly been fewer new materials discovered. 

I think that here there is an important difference between SpaceX and facebook, since SpaceX is an experiment that just burns private money if it fails to have a long-term payoff, whereas facebook is a global institution whose negative aspects harm billions of people. There's also a difference between something like Mars exploration, which is a simple and popular idea that's expensive to implement,  and more kooky vanity projects which consist of rich people imagining that their being rich also makes them able to solve  hairy problems that more qualified people have failed to solve for ages (an example that comes to mind, which thankfully doesn't have billions of dollars riding on it, is Wolfram's project to solve physics: https://blog.wolfram.com/2021/04/14/the-wolfram-physics-project-a-one-year-update/). I think that many big ambitious initiatives by billionaires are somewhere in between kooky ego-trip and genuinely original/pareto-optimal experiment, but it seems important to recognize that these are different things. Given this point of view, along with the general belief that large systems tend to err on the side of being conservative, I think that it's at least defensible to support experiments like SpaceX or Zuckerberg's big Newark school project, even when (like Zuckerberg's school project) they end up not being successful.

Open questions:

What's the incentive structure here? If I'm following the money, it seems likely that there's a much higher likely return if you hype up your plausibly-really-important product, and if you believe in the hype yourself. I don't see why Musk or Zuckerberg should ask themselves the hard questions about their mission given that there's not, as far as I can see, any incentive for them to do so. (Which seems bad!)

What can be done? Presumably we could fund two FTE in-house at any given EA research organization to red-team any given massive corporate effort like SpaceX. But I don't have a coherent theory of change as to what that would accomplish. Pressure the SEC to require annual updates to SEC filings? Might be closer...

Somebody ought to start an independent organization specifically dedicated to red-teaming other people and groups' ideas.

I could start this after I graduate in the Fall, or potentially during the summer.

DM me if you want to discuss organization / funding.

FWIW, Elon Musk famously kiiiiiiinda had a theory-of-change/impact before starting SpaceX. In the biography (and the WaitButWhy posts about him), it notes how he thought about funding a smaller mission of sending mice to Mars, and used a material cost spreadsheet to estimate the adequacy of existing space travel technology. He also aggressively reached out to experts in the field to look for the "catch", or whether he was missing something.

This is still nowhere near good red-teaming/proving-his-hunch-wrong, though. He also didn't seem to do nearly as much talking-to-experts knowledge-base-building for his other projects (e.g. Neuralink).

And most groups don't even do that.

I'm pretty sure Mark Zuckerberg still thinks Facebook is a boon to humanity, based on his speculation on the value of "connecting the planet".

This seems a bit naive to me. Most big companies come up with some generic nice-sounding reason why they're helping people. That doesn't mean the people in charge honestly believe that; it could easily just be marketing.

I consistently enjoy your posts, thank you for the time and energy you invest.

Robin Hanson is famous for critiques in the form of “X isn’t about X, it’s about Y.” I suspect many of your examples may fit this pattern. To wit, Kwame Appiah wrote that “in life, the challenge is not so much to figure out how best to play the game; the challenge is to figure out what game you’re playing.” Andrew Carnegie, for instance, may have been trying to maximize status, among his peers or his inner mental parliament. Elon Musk may be playing a complicated game with SpaceX and his other companies. To critique assumes we know the game, but I suspect we only have a dim understanding of ”the great game” as it’s being played today.

When we see apparent dysfunction, I tend to believe there is dysfunction, but more deeper in the organizational-civilizational stack than it may appear. I.e. I think both Carnegie and Musk were/are hyper-rational actors responding to a very complicated incentive landscape.

That said, I do think ideas get lodged in peoples’ heads, and people just don’t look. Fully agree with your general suggestion, “before you commit yourself to a lifetime’s toil toward this goal, spend a little time thinking about the goal.”

That said— I’m also loathe to critique doers too harshly, especially across illegible domains like human motivation. I could see how more cold-eyed analysis could lead to wiser aim in what things to build; I could also see it leading to fewer great things being built. I can’t say I see the full tradeoffs at this point.