Samo Burja: What the collapse of FTX means for effective altruism

By peterhartree @ 2022-11-17T12:16 (+46)

This is a linkpost to https://brief.bismarckanalysis.com/p/what-the-collapse-of-ftx-means

Samo Burja has the best analysis I've seen so far.


CONTENT WARNING: Samo's analysis may be upsetting and demoralising.

If you're feeling low, anxious, or otherwise in a bad way, I strongly recommend you bookmark this post and come back when you're on better form.

If you're ready for a calm attempt to understand what is happening, and what this all means, read on.


This post is written in a personal capacity

I'm sharing this in a personal capacity.

I am not speaking on behalf of any current or past employer.

I asked a couple of people for their thoughts before I posted, but the decision to post is mine alone.


Disclosure


Personal comment


Selected excerpts

The moral authority and intellectual legitimacy of EA will be reduced

Most importantly, the collapse of FTX will reduce the moral authority and intellectual legitimacy of the “Effective Altruism” (EA) movement, which has become a rising cultural force in Silicon Valley and a rising financial force in global philanthropy. Sam Bankman-Fried was a highly visible example of an “Effective Altruist.” He was a major donor both to the movement and to causes favored by the movement, for example committing $160 million in grants through the FTX Future Fund. Bankman-Fried also frequently boosted its ideology in press interviews and reportedly only earned money so that he could donate more of it to the cause, an ethos known as “earn to give” that was invented by the EA movement.

Good summary of SBF political activity

Bankman-Fried has also shown a serious interest in engaging with mainstream U.S. politics. He has funded political candidates aligned with his own views in congressional primary races and ballot initiatives in California. In 2020, he became President Joseph Biden’s second-largest campaign donor and, in 2022, the Democratic Party’s second-largest individual donor behind only George Soros, having donated a total of $39.8 million. While other tech and crypto companies tend to do too little political lobbying for their own good, Bankman-Fried pursued the opposite strategy, with the result that he was both the highest-profile Effective Altruist and the highest-profile cryptocurrency entrepreneur in Washington.

Effective Altruism’s claims to authority and legitimacy rest on the idea that the EA movement is better than others at allocating time & resources towards saving & improving human lives.

With the collapse of FTX, the regulatory outlook for the cryptocurrency sector in the U.S. has become worse, and it seems unlikely that the venture capital industry will soon see a return on the tens of billions invested into cryptocurrency startups since 2020. This will reduce the financial resources and strategic autonomy available to tech elites in the near future. But it will also reduce the appeal of the fastest-rising philanthropic vehicle in Silicon Valley, which has increasingly become a cultural and even political force in its own right. As its name suggests, Effective Altruism’s claims to authority and legitimacy rest on the idea that the EA movement is better than others at allocating time and resources towards saving and improving human lives. In Sam Bankman-Fried, perhaps its most visible proponent has demonstrated the opposite.

Totalising ideology provides EAs with justification to intervene in any area of human activity

Effective Altruism has become both a cultural force in Silicon Valley and among ambitious young people in the English-speaking world, as well as a growing financial force in both global philanthropy and U.S. electoral politics. Affiliating with the EA movement is attractive not just because it offers new adherents a potentially high-status career and professional network, but because it also offers power. The totalizing ideology of EA provides a justification to intervene in any area of human activity—whether electoral politics, technological development, or anything in between—so long as it can be justified on the basis of altruism. This power can be used insofar as the EA movement can build up its moral authority and intellectual legitimacy, on which the power ultimately rests. Effective Altruism has been well on its way to becoming not just a charitable movement, but a numerically small yet influential political faction.

Nishad Singh: Alameda "couldn't have taken off without EA" because "all the employees, all the funding—everything was EA to start with"

Bankman-Fried served briefly as director of development at the Centre for Effective Altruism before founding Alameda Research and then FTX, and has been transparent that the appeal of cryptocurrency is its ability to accumulate money fast, which, until recently, he had certainly achieved. Alameda’s earliest trades were enabled by Bankman-Fried’s friendships in the Effective Altruism community, through which Bankman-Fried found connections in Japan willing and able to help exploit the Bitcoin price arbitrage, as well as the funding to increase the scale of the trades. According to FTX’s director of engineering Nishad Singh, Alameda “couldn’t have taken off without EA,” because “all the employees, all the funding—everything was EA to start with.”

EA movement finances will be no worse than they were in 2019

In 2022, the FTX Future Fund committed over $160 million in grants, including tens of millions to EA organizations, and the rest strongly influenced by EA ideas of effective charity. On November 10, the FTX Future Fund team resigned and announced that “it looks likely that there are many committed grants that the Future Fund will be unable to honor.” FTX’s arrival on the scene funded a profusion of professional EA organizers and a growing patronage network of “regranting” funds given to respected community members to redistribute as they think best. All this is now threatened, but should these new positions evaporate entirely, the movement as a whole will be no worse off financially than it was in 2019.

[...]

The loss of current grants and especially of expected future donations will be a blow to EA institutions, but they are in little danger of collapse.

Careful observers & live players will no longer take EA’s claims of moral supremacy at face value, and the opinion of these groups has influence far in excess of their low numbers.

The bigger blow to EA will be to its moral authority and intellectual legitimacy. EA has long benefited from a very strong position in ethical discourse and the presumption that its institutions are doing “the most good,” especially among top intellectuals and ambitious live players making a name for themselves. EA’s alliance with Bankman-Fried and his use of EA philosophy to justify his activities will make this presumption much harder to take for granted. The scandal will most likely fade from discourse over the next few months as news cycles move on and, like any scandalized movement, EAs will be especially motivated to downplay the events or simply ignore them. However, careful observers and live players will remember the relationship and will no longer take EA’s claims of moral supremacy at face value, and the opinion of these groups has influence far in excess of their low numbers.

Like most ideologies, Effective Altruism makes it easy for the ends to justify the means

Like most ideologies, Effective Altruism makes it easy for the ends to justify the means: if doing more good requires acquiring more money, then perhaps even questionable means of acquiring more money can be easily justified or excused as a smaller evil in the service of the greater good. The phenomenon of wealthy businessmen taking to charity to polish their image and justify questionable business practices is also not a new one. Before the liquidity crunch at FTX, this worked well for Bankman-Fried, earning him prestige, glowing media profiles, and useful allies such as EA luminary Will MacAskill.

Whether or not Sam Bankman-Fried genuinely believes in Effective Altruism is ultimately moot, since his career path has been perfectly compatible with a genuine commitment to the cause. In other words, earnest intentions should not be ruled out. A genuine belief in the risks of rapid development of artificial general intelligence might well have been in part responsible for his high-risk approach to building wealth. The possibility of near-term global disaster, whether from AI or pandemics, may have represented a real imperative to grow disposable funds as quickly as possible in order to combat these immediate dangers.

Effective Altruism is the rising moral and charitable vehicle in the tech culture of software engineers, venture capital-funded entrepreneurs, and the cryptorich. With his artfully sloppy t-shirts, unkempt hair, and stunts like playing video games during meetings, Bankman-Fried doubled down on membership in this culture and deyance of traditional professional culture. It is this cultural package that has him donating to EA causes instead of, for example, building a dorm at his alma mater like a traditional philanthropist might. But the collapse of FTX also reveals that Effective Altruism is in fact more similar to its philanthropic predecessors than either would like to admit: it is just as willing to convert “dirty” money from cryptocurrency casinos into legitimacy, in the same way universities or foundations are willing to accept large donations from Goldman Sachs or Philip Morris.

SBF raised the status of EA, and will now lower it in the eyes of careful observers

The legitimization also went both ways. By implicitly crediting his career path to the pure altruistic motives of EA, Bankman-Fried lent it the aura of financial success, which is a potent source of prestige and a draw for ambitious young recruits. He is a savvy and energetic self-promoter, and EA benefited from his brand-building in the news media. Before the collapse of FTX, the movement was happy to go along with his mythology and hold him up as a respected role model. The collapse of FTX and Sam Bankman-Fried’s fortunes will only moderately impact the material situation of the EA movement. But for careful observers it deals a serious blow to the moral authority, intellectual legitimacy, and prestige that the movement has cultivated thus far, and on which its appeal and power ultimately rest.


AndrewDoris @ 2022-11-17T18:44 (+22)

"Moral authority" and "intellectual legitimacy" are such fuzzy terms that I'm not really sure what this post is arguing.

Insofar as they just denote public perceptions, sure: this is obviously bad PR for the movement. It shows we're not immune from big mistakes, and raises fair questions about the judgment of individual EAs, or certain problematic norms/mindsets among the living breathing community of humans associated with the label. We'll probably get mocked a bit more and be greeted with more skepticism in elite circles. There are meta-EA problems that need fixing, that I've been introspecting on the past two weeks.

But "careful observers" also knew that before the FTX scandal, and it's unclear to me which specific ideas in EA philosophy are less intellectually legitimate or authoritative than they were before. When a prominent Democrat politician has a scandal, Democrats get mocked - but nobody intelligent thinks that reduces the moral authority of being pro-choice or supporting stricter gun control, etc. The ideas are right or wrong independent of how their highest-profile advocates behave.

Perhaps SBF's fraud indicts the EA community's lack of scrutiny or safeguards around how to raise money. But to me, it does not at all indict EA's ability to allocate resources once they've been raised. It's not as if the charities SBF was funding were proven ineffective. "The idea that the EA movement is better than others at allocating time and resources toward saving and improving human lives" could be right or wrong, but this incident isn't good evidence either way.

Sarah Levin @ 2022-11-17T20:53 (+8)

See Samo's essay series here for the definition of "intellectual legitimacy" as it's being used in the OP:

An idea has intellectual legitimacy insofar as it is recognized by society as respectable and reasonable. An intellectually legitimate idea does not need to be recognized as credible by all people, or even by very many people at all. There only needs to be a general perception that society at large holds the idea to be legitimate. Powerful institutions and individuals are seen as tolerating or endorsing it. Such a perception isn’t necessarily coupled to whether an idea is true.

...

individuals routinely use legitimacy as a shortcut for evaluating the quality of the ideas around them. What may intuitively feel like evaluating an idea on its merits has oftentimes already factored in how an idea is communicated, and who is communicating it. We do this because it is harder for us to assess claims in fields that are outside our areas of expertise and so, instead, we learn from experience which sources to rely on. Evaluating an idea’s intellectual legitimacy is often safer and easier than evaluating the idea itself, and in a healthy society, the shortcut works. This makes the shortcut an efficient and effective heuristic for individuals. Even then, though, an intellectually valid idea that is correctly perceived as legitimate might ultimately still turn out to be false.

Tobias Dänzer @ 2022-11-17T12:55 (+8)

Where do the multiple "inzuence" typos come from? A weird effect of copy&pasting? (I've seen this stuff in pdfs when "fl" was displayed as a separate character.) A typo in the original that was fixed there but not here?

Greg_Colbourn @ 2022-11-17T18:20 (+7)

A genuine belief in the risks of rapid development of artiycial general intelligence might well have been in part responsible for his high-risk approach to building wealth. The possibility of near-term global disaster, whether from AI or pandemics, may have represented a real imperative to grow disposable funds as quickly as possible in order to combat these immediate dangers.

I really hope this wasn't a factor (I don't think it was - Sam's megalomania, seeming borderline sociopathy, and pride -- stealing customer funds rather than let Alameda fail -- seem much more likely to be  prominent causes of the approach he took, which went far beyond "high-risk" into outright criminality). But if it was, it was going against everything Yudkowsky -- arguably the biggest and most doomiest proponent of AGI x-risk -- has said

Ironically, if AI timelines really are short (or biorisk great), the FTX crisis has likely significantly increased existential risk with its setting back of the reputation of EA and Longtermism. As a proponent for urgent action on AGI x-risk, I am saddened by the association with SBF/FTX. And the fact remains that we need to address the risk.

Sharmake @ 2022-11-17T18:58 (+3)

Also, there's this link on April 1, 2022, several months before, warning against this type of thinking. Go to Q4 for the answer.

https://www.lesswrong.com/posts/j9Q8bRmwCgXRYAgcJ/miri-announces-new-death-with-dignity-strategy

Greg_Colbourn @ 2022-11-17T19:38 (+2)

Thanks, had forgotten about that. Added.

Greg_Colbourn @ 2022-11-17T18:00 (+7)

CONTENT WARNING: Samo's analysis may be upsetting and demoralising.

The whole situation is upsetting and demoralising. But there is a way forward. We should be humbled, keep in mind important things, and learn lessons. Then get back to work.