Biorisk researcher Gregory Lewis's criticism of EA's initial response to covid-19

To clarify, are you saying that, in retrospect, the process through which people in EA did research on epidemiology, public health, and related topics looks any better to you now that it looked to you back in April 2020 when you did this interview? 

I think I understand your point that it would probably be nearly impossible to score the conclusions in a way that people in EA would agree is convincing or fair — there's tons of ambiguity and uncertainty, hence tons of wiggle room. (I hope I'm understanding that right.)

But in the April 2020 interview, you said that many of these conclusions were akin to calling a coin flip. Crudely, many interventions that experts were still debating could be seen as roughly having a 50-50 chance of being good or bad (or maybe it's anywhere from 70-30 to 30-70, doesn't really matter), so any conclusion that an intervention is good or bad has a roughly 50-50 chance of being right. You said a stopped clock is right twice a day, and it may turn out that Donald Trump got some things right about the pandemic, but if so, it will be through dumb luck rather than good science. 

So, I'm curious: leaving aside the complicated and messy question of scoring the conclusions, do you now think the EA community's approach to the science — particularly, the extent to which they wanted to do it themselves, as non-experts, rather than just trying to find the expert consensus on any given topic, or even seeing if any expert would talk to them about it (e.g. in 2020, you suggested some names of experts to have on the 80,000 Hours Podcast) — was any less bad than you saw it in 2020?

That seems like a perfectly reasonable, fair challenge to put to GiveWell. That’s the right question for people to ask! My impression is that GiveWell, starting early on and continuing today, has acted with humility and put in the grinding hard work to build credibility over time. 

I don’t think GiveWell would have been a meaningfully useful project if a few people just spent a bit of their spare time over a few weeks to produce the research and recommendations. (Yet that seems sufficient in some cases, such as covid-19, for people in EA to decide to overrule expert opinion.)

It could have been different if there were already a whole field or expert community doing GiveWell-style cost-effectiveness evaluations for global health charities. Part of what GiveWell did was identify a gap in the “market” (so to speak) and fill it. They weren’t just replicating the effort of experts. 

[Edited on Dec. 16 at 8:50 PM Eastern to add: to clarify, I mean the job GiveWell did was more akin to science journalism or science communication than original scientific research. They were building off of expert consensus rather than challenging it.]

As I recall, GiveWell initially was a project the founders tried to do in their spare time, and then quickly realized was such a big task it would have to be a full-time job. I also hazily recall them doing a lot of work to get up to speed, and also that they had, early on, (and still have) a good process for taking corrections from people outside the organization.

I think as a non-expert waltzing into an established field, you deserve the skepticism and the challenges you will initially get. That is something for you to overcome, that you should welcome as a test. If that is too hard, then the project is too hard. 

After all, this is not about status, esteem, ego, or pride, right? It’s about doing good work, and about doing right by the aid recipients or the ultimate beneficiaries of the work. It's not about being right, it's about getting it right.

Based on my observations and interactions, EA had much more epistemic modesty and much less of a messiah complex in the early-to-mid-2010s. Although I saw a genuinely scary and disturbing sign of a messiah complex from a LEAN employee/volunteer as early as 2016, who mused on a Skype call about the possibility of EA solving the world's problems in priority sequence. (That person, incidentally, also explicitly encouraged me and my fellow university EA group organizer in the Skype meeting to emotionally manipulate students into becoming dedicated effective altruists in a way that I took to be plainly unethical and repugnant. So, overall, a demented conversation.) 

The message and tone in the early-to-mid-2010s was more along the lines of: 'we have these ideas about our obligation to give to charity and about charity cost-effectiveness that might sound radical and counterintuitive, but we want to have a careful, reasonable conversation about it and see if we can convince people we're on the right track'. Whereas by now, in the mid-2020s, the message and tone feels more like: 'the EA community has superior rationality to 99.9% of people on Earth, including experts and academics in fields we just started thinking about 2 weeks ago, and it's up to us to save the world — no, the lightcone!'. I can see how the latter would be narratively and emotionally compelling to some people, but it's also extremely off-putting to many other people (like me, perhaps also to Gregory Lewis and other people he knows with backgrounds in public health/medicine/epidemiology/virology/etc., although I don't want to speak for him or the people he knows). 

This sounds like outcome bias to me, i.e., believing in retrospect that a decision was the right one because it happened to turn out well. For example, if you decide to drive home drunk and don't crash your car, you could believe based on outcome bias that that was the right decision. 

There may also be some hindsight bias going on, where in retrospect it's easier to claim that something was absolutely, obviously the right call, when, in fact, at the time, based on the available evidence, the optimally rational response might have been to feel a significant amount of uncertainty. 

I don't know if you're right that someone taking their advice from Gregory Lewis in this interview would have put themselves more at risk. Lewis said it was highly uncertain (as of mid-April 2020) whether medical masks were a good idea for the general population, but to the extent he had an opinion on it at the time, he was more in favour than against. He said it was highly uncertain what the effect of cloth masks would ultimately turn out to be, but highlighted the ambiguity of the research at the time. There was a randomized controlled trial that found cloth masks did worse than the control, but many people in the control group were most likely wearing medical masks. So, it's unclear.

The point he was making with the cloth masks example, as I took it, was simply that although he didn't know how the research was ultimately going to turn out, people in EA were missing stuff that experts knew about and that was stated in the research literature. So, rather than engaging with what the research literature said and deciding based on that information, people in EA were drawing conclusions from less complete information.

I don't know what Lewis' actual practical recommendations were at the time, or if he gave any publicly. It would be perfectly consistent to say, for example, that you should wear medical masks as a precaution if you have to be around people and to say that the evidence isn't clear yet (as of April 2020) that medical masks are helpful in the general population. As Lewis noted in the interview, the problem isn't with the masks themselves, it's that medical professionals know how to use them properly and the general population doesn't. So, how does that cash out into advice?

You could decide: ah, why bother? I don't even know if masks do anything or not. Or you could think: oh, I guess I should really make sure I'm wearing my mask right. What am I supposed to do...? 

Similarly, with cloth masks, when it became better-known how much worse cloth masks were than medical masks, everyone stopped wearing cloth masks and started wearing KN95, N95 masks, or similar. If someone's takeaway from that April 2020 interview with Lewis was that the efficacy of cloth masks was unclear and there's a possibility they might even turn out to be net harmful, but that medical masks work really well if you use them properly, again, they could decide at least two different things. They could think: oh, why bother wearing any mask if my cloth mask might even do more harm than good? Or they could think: wow, medical masks are so much better than cloth masks, I really should be wearing those instead of cloth masks.

Conversely, people in EA promoting cloth masks might have done more harm than good depending on, well, first of all, if anyone listened to that advice in the first place, but, second, on whether any people who did listen decided to go for cloth masks rather than no mask, or decided to wear cloth masks instead of medical masks.

Personally, my hunch is that if very early on in the pandemic (like March or April 2020), there had been less promotion of cloth masks, and if more people had been told the evidence looked much better for medical masks than for cloth masks (slight positive overall story for medical masks, ambiguous evidence for cloth masks, with a possibility of them even making things worse), then people would have really wanted to switch from cloth masks to medical masks — because this is what happened later on when the evidence came in and it was much clearer that medical masks were far superior. 

The big question mark above that is I don't know/don't remember at what point the supply chain was able to provide enough medical masks for everybody. 

I looked into it and found a New York Times article from April 3, 2020 that discusses a company in Chicago that had a large supply of KN95 masks, although this is still in the context of providing masks to hospitals:

One Chicago-based company, iPromo, says it has been in the KN95 importing business for a month. It had previously developed relationships with Chinese supplies for its main business, churning out custom logo-adorned promotional knickknacks like mugs, water bottles, USB flash drives and small containers of hand sanitizer.

The company’s website advertises KN95 masks at $2.96 apiece for hospitals, with delivery in five to seven days, although its minimum order is 1,000 masks.

More masks are available because coronavirus transmission in China has been reduced. “They have so much stock,” said Leo Friedman, the company’s chief executive, during an interview Thursday. “They ramped up and now it’s a perfect storm of inventory.”

I found another source that says, "Millions of KN95 masks were imported between April 3 and May 7 and many are still in circulation." But this is also still in the context of hospitals, not the general public. That's in the United States.

I found a Vice article from July 13, 2020 that implies by that point it was easy for anyone to buy KN95 masks. Similarly, starting on or around June 30, 2020, there were vending machines selling KN95 masks in New York City subway stations. Although apparently KN95 masks were for sale in vending machines in New York as early as May 29.

In any case:

1. It seems quite plausible that pushing people toward medical masks rather than cloth masks sooner would have been beneficial.
2. Even if people who don't understand the science and don't know how to properly read the research occasionally stumble into the correct conclusion from time to time through sheer luck, that doesn't imply this is the best strategy over a long timespan, over a large sample size of examples. 

Is this a response to what Gregory Lewis said? I don’t think I understand. 

Maybe this is subtle/complicated… Are the examples you’re citing the actual consensus views of experts? Or are they examples of governments and institutions like the World Health Organization (WHO) misunderstanding what the expert conensus and/or misrepresenting the expert consensus to the public? 

This excerpt from the Nature article you cited makes it sound like the latter:

The change brings the WHO’s messaging in line with what a chorus of aerosol and public-health experts have been trying to get it to say since the earliest days of the outbreak. Many decry the agency’s slowness in stating — unambiguously — that SARS-CoV-2 is airborne. Interviews conducted by Nature with dozens of specialists on disease transmission suggest that the WHO’s reluctance to accept and communicate evidence for airborne transmission was based on a series of problematic assumptions about how respiratory viruses spread.

Did random members of the EA community — as bright and eager as they might be — with no prior education, training, or experience with relevant fields like public health, epidemiology, virology, and medicine outsmart the majority of relevant experts on this question (airborne vs. not) or any others, not through sheer luck or chance, but by actually doing better research? This is a big claim, and if it is to be believed, it needs strong evidentiary support. 

In this Nature article, there is an allegation that the WHO wasn’t sufficiently epistemically modest or deferential to the appropriate class of experts:

Other criticisms are that the WHO relies on a narrow band of experts, many of whom haven’t studied airborne transmission, and that it eschews a precautionary approach that could have protected countless people in the early stages of the pandemic.

And there is an allegation around communications, rather than around the science itself:

Having shifted its position incrementally over the past two years, the WHO also failed to adequately communicate its changing position, they say. As a result, it didn’t emphasize early enough and clearly enough the importance of ventilation and indoor masking, key measures that can prevent airborne spread of the virus.

So, let’s say there’s another pandemic. Which is the better strategy? 

Strategy A: Read forum posts and blog posts by people in the EA community doing original research and opining on epidemiology, virology, and public health who have never so much cracked open a relevant textbook.

Strategy B: Survey sources of expert opinion, including publications like Nature, open letters written on behalf of expert communities, statements by academic and scientific organizations and so on, to determine if a particular institution like the WHO is accurately communicating the majority view of experts, or if they’re a weird outlier adhering to a minority view, or just communicating the science badly. 

I would say the Nature article is support for Strategy B and not at all support for Strategy A. 

You could even interpret it as evidence against Strategy A. If you believe the criticism in Nature is right, even experts in an adjacent field or subfield, who have prestigious credentials like advising the WHO, can get things catastrophically wrong by being insufficiently epistemically modest and not deferring enough to the experts who know the most about a subject, and who have done the most research on it. If this is true, then that should make you even more skeptical about how reliable the research and recommendations will be from a non-expert blogger with no relevant education who only started learning about viruses and pandemics for the first time a few weeks ago.

What is described in the Nature piece sound like incredibly subtle mistakes (if we can for sure call them mistakes at this point). Lewis’ critique of the EA community is that it was making incredibly obvious, elementary mistakes. So, why think the EA community can outperform experts on avoiding subtle mistakes if it can’t even avoid the obvious mistakes?

One of the recent examples of hubris I saw on the EA Forum was someone asserting that they (or the EA community at large) could resolve, within the next few months/years, the fundamental uncertainty around the philosophical assumptions that go into cost-effectiveness estimates comparing shrimp welfare and human welfare. Out of the topics I know well, the philosophy of consciousness might be #1, or at least near the top. I don’t know how to convey the level of hubris that comment betrays. 

It would be akin to saying that a few people in EA, with no training in physics, could figure out, in a few months or years, the correct theory of quantum gravity and reconcile general relativity and quantum mechanics. Or that a few people in EA, with no training in biology or medicine, would be able to cure cancer within a few months or years. Or that, with no background in finance, business, or economics, that they’d be able to launch an investment fund that consistently, sustainably achieves more alpha than the world’s top-performing investment funds every year. Or that, with no background in engineering or science, they’d be able to beat NASA and SpaceX in sending humans to Mars.  

In other words, this is a level of hubris that’s unfathomable, and just not a serious or credible way to look at the world or your own abilities, in the absence of strong, clear evidence that you possess even average abilities relative to the relevant expert class. 

I don’t know the first thing about epidemiology, virology, public health, or medicine. So, I can’t independently evaluate how appropriate or correct it is/was for Gregory Lewis to be so aggravated by the EA community’s initial response to covid-19 that he considered distancing himself from the movement. I can believe that Lewis might be correct because a) he has the credentials, b) the way he’s describing it is how it essentially always turns out when non-experts think they can outsmart experts in a scientific or medical field without first becoming experts themselves, and c) in areas where I do know enough to independently evaluate the plausibility of assertions made by people in the EA community on the object level, I feel as infuriated and incredulous as Lewis described feeling in that 80,000 Hours interview. 

I see these sort of wildly overconfident claims about being able to breezily outsmart experts on difficult scientific, philosophical, or technical problems as moderately, but not dramatically, more credible than the people talking about UFOs or ESP or whatever. (Which apparently is not that rare.)

I see a general rhetorical or discursive strategy employed across many people with fringe views, be they around pseudoscience, fake medicine, or conspiracy theories. First, identify some scandal or blunder or internecine conflict within some scientific expert community. Second, say, “Aha! They’re not so smart, after all!” Third, use this as support for whatever half-cocked pet theory you came up with. This is obviously a logically invalid argument, as in, obviously the conclusion does not logically follow from the premises. The standard for scientific experts should not be perfection; the standard for amateurs, dilettantes, and non-expert iconoclasts should be showing they can objectively do better than the average expert — not on a single coin flip, but on an objective, unbiased measure of overall performance.

There is a long history in the LessWrong community of opposition to institutional science, with the typical amount of intellectual failure that usually comes with opposition to institutional science. There is a long history of hyperconfidently scorning expert consensus and being dead wrong. Obviously, there is significant overlap between the LessWrong community and the EA community, and significant influence by the former on the latter. What I fear is that this anti-scientific attitude and undisciplined iconoclasm has become a mainstream, everyday part of the EA community, in a way that was not true, or at least not nearly as true, in my experience, in the early-to-mid-2010s. 

The obvious rejoinder is: if you really can objectively outperform experts in any field you care to try your hand at for a few weeks, go make billions of dollars right now, or do any other sort of objectively impressive thing that would provide evidence for the idea that you have the abilities you think you do. Surely, within a few months or years of effort, you would have something to show for it. LessWrong has been around for a long time, EA has been around for a long time. There’s been plenty of time. What’s the excuse for why people haven’t done this yet?

And, based on base rates, what would you say is more likely: people being misunderstood iconoclastic self-taught geniuses who are on the cusp of greatness or people just being overly confident based on a lack of experience and a lack of understanding of the problem space? 

Huh? I think everything mentioned in this post (either by me or by Gregory Lewis) are behaviours that people can consciously choose to engage in or not. I don’t think any of them are personality traits.

What do you think is an example of a personality trait mentioned in this post?