My tentative best guess on how EAs and Rationalists sometimes turn crazy

My theory is that while EA/rationalism is not a cult, it contains enough ingredients of a cult that it’s relatively easy for someone to go off and make their own. 

Not everyone follows every ingredient, and many of the ingredients are actually correct/good, but here are some examples:

• Devoting ones life to a higher purpose (saving the world)
•  High cost signalling of group membership (donating large amounts of income)
• The use of in-group shibboleths (like “in-group, and “shibboleths”)
• The use of weird rituals and breaking social norms  (Bayesian updating, “radical honesty”, etc)
• A tendency to isolate oneself from non-group members (group houses, EA orgs)
• the belief that the world is crazy, but we have found the truth (rationalist thinking)
• the following of sacred texts explaining the truth of everything (the sequences)
• And even the belief in an imminent apocalypse (AI doom)

These ingredients do not make EA/rationalism in general a cult, because it lacks enforced conformity and control by a leader. Plenty of people, including myself, have posted on Lesswrong critiquing the sequences and Yudkowsky and been massively upvoted for it. It’s decentralised across the internet, if someone wants to leave there’s nothing stopping them. 

However, what seems to have happened is that multiple people have taken these base ingredients and just added in the conformity and charismatic leader parts. You put these ingredients in a small company or a group house, put an unethical or mentally unwell leader in charge, and you have everything you need for an abusive cult environment. Now it’s far more difficult to leave, because your housing/income is on the line, and the leader can use the already established breaking of social norms as an excuse to push boundaries and consent in the name of the greater good.  This seems to have happened multiple times already. 

I don’t know what to do if this theory is correct, besides to take extra scrutiny of leaders of sub-groups within EA, and maybe ease up on unnecessary rituals and jargon. 

I think another contributing factor is that as a community we a) prize (and enjoy!) unconventional thinking, and b) have norms that well-presented ideas should  be discussed on their merits, even if they seem strange. That means that even the non-insane members of the community often say things that sound insane from a common-sense perspective, and that many insane ideas are presented at length and discussed seriously by the community.

I think experiencing enough of this can disable your "insane idea alarm". Normally, if someone says something that sounds insane, you go "this person is insane" and run away. But in the EA/rationalist community you instead go "hm, maybe this is actually a great but out-oft-the-box idea?".

And then the bad state is that someone says something like "you should all come and live in a house where I am the ruler and I impose quasi-military discipline on you" and you go "hm, maybe this is an innovative productivity technique for generating world-saving teams" instead of immediately running away screaming.

Thanks for this, I agree with almost all of it. Just one observation, which I think is important. You say "Now one might think that because we have a lot of smart people, we might be able to avoid the worst outcomes here, by just not enforcing extreme standards that seem pretty crazy."

I think the main factor that could contain craziness is social judgment rather intelligence. So, I think the prevalence of autism in the community means that EAs will be less able to than other social groups to contain craziness. 

Vasco Grilo @ 2023-06-22T18:00 (+10)

Hi John,

Is there concrete data on the prevalence of autism in the EA community?

David_Moss @ 2023-06-23T13:03 (+22)

The 2020 SlateStarCodex survey has some data on this. Obviously it is quite limited by the fact that the sample is pre-selected for the (rationalist-leaning) SSC audience already, which you might also expect to be associated with autism.

This survey asked about identification as EA (No/Sorta/Yes) and autism (I don't have this condition and neither does anyone in my family/I have family members (within two generations) with this condition/I think I might have this condition, although I have never been formally diagnosed/I have a formal diagnosis of this condition). Unfortunately, these leave a lot of researcher degrees of freedom (e.g. what to do with the "Sorta" and the "never been formally diagnosed" respondents).

Just looking at the two raw measures straightforwardly, we see no significant differences, though you can see that there are more people with a formal diagnosis in the "Yes" category. 

I'd be wary of p-hacking from here, though fwiw, with different, more focused analyses, the results are borderline significant (both sides of the borderline), e.g. looking at whether someone said "Yes" they are an EA * whether they are formally diagnosed with autism, there is a significant association (p=0.022).

I have previously considered whether it would be worth including a question about this in the EAS, but it seems quite sensitive and relatively low value given our high space constraints.

John G. Halstead @ 2023-06-22T19:33 (+8)

Not to my knowledge, though I think it's pretty clear

David Mathers @ 2023-06-23T11:31 (+5)

This sounds kind of plausible to me, but couldn't you equally say that you'd expect autism to be mitigating factor to cultiness, because cults are about conformity to group shibboleths for social reasons which autistic people are better at avoiding ? (At least, I'd have thought we are. Maybe only I have that perception?) That kind of makes me think that actually it is just easy to generate plausible-sounding hypothesis about the effects of a fairly broad and nebulous thing like "autistic traits" and maybe none of them should be taken that seriously without statistical evidence to back them up. 

I think this sounds sensible in general, but FTX might have just been greed. Probably only a handful of people knew about the fraud, after all.

I'd be interested in more thoughts if you have them on evidence or predictions one could have made ahead of time that would distinguish this model from other (like maybe a lot of what's going on is youth and will evaporate over time (youth still has to be mediated by things like what you describe, but as an example).

Also, my understanding is that SBF wasn't very insecure? Does that affect your model or is the point that the leader / norm setter doesn't have to be?

Seems like the forces that turn people crazy are the same ones that lead people to do anything good and interesting at all. At least for EA, a core function of orgs/elites/high status community members is to make the kind of signaling you describe highly correlated with actually doing good. Of course it seems impossible to make them correlate perfectly, and that’s why setting with super high social optimization pressure (like FTX) are gonna be bad regardless.

But (again for EA specifically) I suspect the forces you describe would actually be good to increase on the margin for people not living in Berkeley and/or in a group house which is probably a majority of self-identified EAs but a strong minority of the people-hours OP interacts with irl.

This could explain the psychology of followers, but I'm not sure it speaks to the psychology of people who lead the cults, and how we can and should deal with these people.

Social dynamics seem important, but I also think Scott Alexander in "Epistemic Learned Helplessness" put his finger on something important in objecting to the rationalist mission of creating people who would believe something once it had been proven to them. Together with "taking ideas seriously"/decompartmentalization, attempting to follow the rules of rationality itself can be very destabilizing.

I think trying to figure out the common thread "explaining datapoints like FTX, Leverage Research, [and] the LaSota crew" won't yield much of worth because those three things aren't especially similar to each other, either in their internal workings or in their external effects. "World-scale financial crime," "cause a nervous breakdown in your employee," and "stab your landlord with a sword" aren't similar to each other and I don't get why you'd expect to find a common cause. "All happy families are alike; each unhappy family is unhappy in its own way."

There's a separate question of why EAs and rationalists tolerate weirdos, which is more fruitful. But an answer there is also gonna have to explain why they welcome controversial figures like Peter Singer or Eliezer Yudkowsky, and why extremely ideological group houses like early Toby Ord's [EDIT: Nope, false] or more recently the Karnofsky/Amodei household exercise such strong intellectual influence in ways that mainstream society wouldn't accept. And frankly if you took away the tolerance for weirdos there wouldn't be much left of either movement.

And the short answer to that is “well, the group standard is what everyone else believes the group standard is”. And this is the exact context in which social miasma dynamics come into play. To any individual in a group, it can easily be the case that they think the group standard seems dumb, but in a situation of risk aversion, the important part is that you do things that look to everyone like the kind of thing that others would think is part of the standard.

1. Combat general social miasma dynamics (e.g. by running surveys or otherwise collapsing a bunch of the weird social uncertainty that makes things insane). Public conversations seem like they should help a bunch, though my sense is that if the conversation ends up being less about the object-level and more about persecuting people (or trying to police what people think) this can make things worse. 

My general sense is that we can do a lot to surface what we as a community actually believe and that will help a lot here. 

eg Polls like this where we learn what the median respondent actually thinks https://viewpoints.xyz/polls/ea-strategy-1/analytics 

Do you intend to build on this by writing about the roles played by those other dynamics you mentioned but didn't write about in this post?

I have many years of being in multiple kinds of religious communities and I can say I've seen this kind of dynamic...this is great thinking. 

I remember reading about a group of people early in Alameda, before FTX started and that had a good number of EA's among them, who stood up against SBF and demanded a better contract and he dismissed them and they all left...one can't help but think that helped create an evaporative concentration of people willing to go along with his crazy.

As an aside, for pondering, the evaporative cooling dynamic is something I experience in cooking almost daily where you reduce the liquid in order to make it a sauce with more concentrated flavors...they always refer to this as reduction, or a reduction sauce. In this case it's a good thing...but knowing this dynamic in cooking helped me understand your argument.