Latest comments on the EA Forum

The amount of knowledge outside your bubble outweighs the knowledge within it by a truly gargantuan margin. If you entering a field that you are not already an expert in, it is foolish to not consult with the existing experts in that field. This includes paying them for consulting time, if necessary. This doesn’t mean every expert is automatically right, but if you are disagreeing with the expert about their field, you should be able to come up with a very good explanation as to why.

 

@Lukeprog posted this few decades ago "neglected rationalist virtue of scholarship" 

Within Rationalism, the obvious filter is the reverence for “the sequences”, an extremely large series of pop science blogposts. In it’s initial form, Lesswrong.com was basically a fanforum for these blogs. So obviously people that like the sequences are far more likely to be in the community than those that don’t. As a result, there is a consensus within rationalism that the core ideas of sequences are largely true.

I think it's mostly just cognitive science which is daniel kanehman and others(which is well known), good bunch of linguistics (which I have heard are well known), and anti-philosophy (because we dislike philosophy as it is done), rest is just ethics and objective bayesianism, with a quinean twist.

Rationalism loves jargon, including jargon that is just completely unnecessary. For example, the phrase “epistemic status” is a fun technique where you say how confident you are in a post you make. But it could be entirely replaced with the phrase “confidence level”

I think there is a difference between epistemic status and confidence level, I could be overtly confident and still buy the lottery ticket while knowing it won't work. I think there is a difference between social and epistemic confidence, so it's better to specify.

Hmm. I'm imagining a monogamous bisexual person who prefers het relationships, but settles for a gay one because they really love their partner and reasonably believe they wouldn't be able to find a better het relationship if they were back on the market (such that they are not avoiding suffering and also maximising utility by being in this relationship). This person would opt to take the pill that makes them exclusively gay in order to feel more life satisfaction (or even SWB), even though it destroys their preferences.

I assume this person is in your latter bucket of preferring greater life satisfaction per se? If so, I don't think this situation is as uncommon as you imply—lots of people have hard to satisfy or unsatisfiable preferences that they would rather be rid of in favour of greater life satisfaction; in some sense, this is what it means to be human (Buddhism again).

Life satisfaction is typically considered to be a kind of (or measure of) subjective well-being, and the argument would be the same for that as a special case. Just make the number go up enough after taking the pill, while replacing what they care about. (And I'm using subjective well-being even more broadly than I think normally used.)

For example, I wonder if people who have preferences that are hard to satisfy might actually want to take such a life-satisfaction pill, if it meant their new preferences were easier to satisfy.

In my view, it only makes sense to do if they already have or were going to otherwise have preferences/attitudes that would be more satisfied by taking the pill. If they would suffer less by taking the pill, then it could make sense. If they prefer to have greater life satisfaction per se, then it can make sense to take the pill.

I am a bit unenlightened when it comes to moral philosophy so I would appreciate if you can help me understand this viewpoint better. Does it change if you replace 'subjective well-being' with 'life satisfaction' (in the sense of SWB being experiential and satisfaction being reflective/prospective)? i.e. are there conceptions of 'life satisfaction' that sort of take into account what this person wants for themselves?

For example, I wonder if people who have preferences that are hard to satisfy might actually want to take such a life-satisfaction pill, if it meant their new preferences were easier to satisfy. (Is this, in some sense, what a lot of Buddhist reframing around desire is doing?)

I like the hybrid approach, and discuss its implications for replaceability a bit here. (Shifting to the intrapersonal case: those of us who reject preference theories of well-being may still recognize reasons not to manipulate preferences, for example based on personal identity: the more you manipulate my values, the less the future person is me. To be a prudential benefit, then, the welfare gain has to outweigh the degree of identity loss. Moreover, it's plausible that extrinsic manipulations are typically more disruptive to one's degree of psychological continuity than voluntary or otherwise "natural" character development.)

It seems worth flagging that some instances of replacement seem clearly good! Possible examples include:

• Generational turnover
• not blindly marrying the first person you fall in love with
• helping children to develop new interests

I guess even preference-affecting views will support instrumental replacement, i.e. where the new desire results in one's other desires being sufficiently better satisfied (even before counting any non-instrumental value from the new desire itself) to outweigh whatever was lost.

I am a bit unenlightened when it comes to moral philosophy so I would appreciate if you can help me understand this viewpoint better. Does it change if you replace 'subjective well-being' with 'life satisfaction' (in the sense of SWB being experiential and satisfaction being reflective/prospective)? i.e. are there conceptions of 'life satisfaction' that sort of take into account what this person wants for themselves?

For example, I wonder if people who have preferences that are hard to satisfy might actually want to take such a life-satisfaction pill, if it meant their new preferences were easier to satisfy. (Is this, in some sense, what a lot of Buddhist reframing around desire is doing?)

I like the hybrid approach, and discuss its implications for replaceability a bit here. (Shifting to the intrapersonal case: those of us who reject preference theories of well-being may still recognize reasons not to manipulate preferences, for example based on personal identity: the more you manipulate my values, the less the future person is me. To be a prudential benefit, then, the welfare gain has to outweigh the degree of identity loss. Moreover, it's plausible that extrinsic manipulations are typically more disruptive to one's degree of psychological continuity than voluntary or otherwise "natural" character development.)

It seems worth flagging that some instances of replacement seem clearly good! Possible examples include:

• Generational turnover
• not blindly marrying the first person you fall in love with
• helping children to develop new interests

I guess even preference-affecting views will support instrumental replacement, i.e. where the new desire results in one's other desires being sufficiently better satisfied (even before counting any non-instrumental value from the new desire itself) to outweigh whatever was lost.

I like the hybrid approach, and discuss its implications for replaceability a bit here. (Shifting to the intrapersonal case: those of us who reject preference theories of well-being may still recognize reasons not to manipulate preferences, for example based on personal identity: the more you manipulate my values, the less the future person is me. To be a prudential benefit, then, the welfare gain has to outweigh the degree of identity loss. Moreover, it's plausible that extrinsic manipulations are typically more disruptive to one's degree of psychological continuity than voluntary or otherwise "natural" character development.)

It seems worth flagging that some instances of replacement seem clearly good! Possible examples include:

• Generational turnover
• not blindly marrying the first person you fall in love with
• helping children to develop new interests

I guess even preference-affecting views will support instrumental replacement, i.e. where the new desire results in one's other desires being sufficiently better satisfied (even before counting any non-instrumental value from the new desire itself) to outweigh whatever was lost.

Change your mind in what way? Could you elaborate a bit?

I don’t know the exact dates, but: a)proof-based methods seem to be receiving a lot of attention b) def/acc is becoming more of a thing c) more focus on concentration of power risk (tbh, while there are real risks here, I suspect most work here is net-negative)

Hi Remmelt,

Joining the bet on which side?

The article is enough to change my mind, personally. 

Where do they say the handpicked line?

Thanks for the comment, Stephen.

Vasco, how do your estimates account for model uncertainty?

I tried to account for model uncertainty assuming 10^-6 probability of human extinction given insufficient calorie production.

I don't understand how you can put some probability on something being possible (i.e. p(extinction|nuclear war) > 0), but end up with a number like 5.93e-14 (i.e. 1 in ~16 trillion). That implies an extremely, extremely high level of confidence.

Note there are infinitely many orders of magnitude between 0 and any astronomically low number like 5.93*10^-14. At least in theory, I can be quite uncertain while having a low best guess. I understand greater uncertainty (e.g. higher ratio between the 95th and 5th percentile) holding the median constant tends to increase the mean of heavy-tailed distributions (like lognormals), but it is unclear to which extent this applies. I have also accounted for that by using heavy-tailed distributions whenever I thought appropriate (e.g. I modelled the soot injected into the stratosphere per equivalent yield as a lognormal).

As a side note, 10 of 161 (6.21 %) forecasters of the Existential Risk Persuasion Tournament (XPT), 4 experts and 6 superforecasters, predicted a nuclear extinction risk until 2100 of exactly 0. I guess these participants know the risk is higher than 0, but consider it astronomically low too.

Putting ~any weight on models that give higher probabilities would lead to much higher estimates.

I used to be persuaded by this type of argument, which is made in many contexts by the global catastrophic risk community. I think it often misses that the weight a model should receive is not independent of its predictions. I would say high extinction risk goes against the low prior established by historical conflicts.

I am also not aware of any detailed empirical quantitative models estimating the probability of extinction due to nuclear war.

They seem really good! I feel like an idiot for asking this, but where on their website can I subscribe to the newsletter? 

Phib @ 2024-07-27T00:32 (+1)

IDK if this actually works since I only just signed up, but, the "Join us" button in top right leads to, "https://sentinel-team.org/contact/"

Seems you can add yourself to mailing list from there.

I think in the eyes of many (including substantial fraction of the public) participating in such an investigation will be seen as importantly heroic. I think it's wrong to assume that people cannot reward and understand the difficulty of such a choice, and cannot assign respect appropriately.

Anthropic issues questionable letter on SB 1047 (Axios). I can't find a copy of the original letter online. 

I think I basically agree with you, and I am definitely not saying we should just shrug. We should instead try to shape the future positively, as best we can. However, I still feel like I'm not quite getting my point across. Here's one more attempt to explain what I mean.

Imagine if we achieved a technology that enabled us to build physical robots that were functionally identical to humans in every relevant sense, including their observable behavior, and their ability to experience happiness and pain in exactly the same way that ordinary humans do. However, there is just one difference between these humanoid robots and biological humans: they are made of silicon rather than carbon, and they look robotic, rather than biological.

In this scenario, it would certainly feel strange to me if someone were to suggest that we should be worried about a peaceful robot takeover, in which the humanoid robots collectively accumulate the vast majority of wealth in the world via lawful means. 

By assumption, these humanoid robots are literally functionally identical to ordinary humans. As a result, I think we should have no intrinsic reason to disprefer them receiving a dominant share of the world's wealth, versus some other subset of human-like beings. This remains true even if the humanoid robots are literally "not human", and thus their peaceful takeover is equivalent to "human disempowerment" in a technical sense.

There ultimate reason why I think one should not worry about a peaceful robot takeover in this specific scenario is because I think these humanoid robots have essentially the same moral worth and right to choose as ordinary humans, and therefore we should respect their agency and autonomy just as much as we already do for ordinary humans. Since we normally let humans accumulate wealth and become powerful via lawful means, I think we should allow these humanoid robots to do the same. I hope you would agree with me here.

Now, generalizing slightly, I claim that to be rationally worried about a peaceful robot takeover in general, you should usually be able to identify a relevant moral difference between the scenario I have just outlined and the scenario that you're worried about. Here are some candidate moral differences that I personally don't find very compelling:

• In the humanoid robot scenario, there's no possible way the humanoid robots would ever end up killing the biological humans, since they are functionally identical to each other. In other words, biological humans aren't at risk of losing their rights and dying.
  • My response: this doesn't seem true. Humans have committed genocide against other subsets of humanity based on arbitrary characteristics before. Therefore, I don't think we can rule out that the humanoid robots would commit genocide against the biological humans either, although I agree it seems very unlikely.
• In the humanoid robot scenario, the humanoid robots are guaranteed to have the same values as the biological humans, since they are functionally identical to biological humans.
  • My response: this also doesn't seem guaranteed. Humans frequently have large disagreements in values with other subsets of humanity. For example, China as a group has different values than the United States as a group. This difference in values is even larger if you consider indexical preferences among the members of the group, which generally overlap very little.

It could be better for the world, and you might care about that.

It could be that you expect enough other people will talk to them that it's good for them to hear your side of the story too.

It could be that you expect it would be bad for your reputation to refuse to talk to them (or to give details which are non-concordant with the picture they're building from talking to other people).

I think it depends what sort of risks we are talking about. The more likely Dustin is to turn out to be perpetrating a fraud (which I think is very unlikely!) the more the marginal person should be earning to give. And the more projects should be taking approaches that conserve runway at the cost of making slower progress toward their goals.

I think that an eventual AI-driven ecosystem seems likely desirable. (Although possibly the natural conception of "agent" will be more like supersystems which include both humans and AI systems, at least for a period.)

But my alarm at nonviolent takeover persists, for a couple of reasons:

• A feeling that some AI-driven ecosystems may be preferable to others, and we should maybe take responsibility for which we're creating rather than just shrugging
• Some alarm that nonviolent takeover scenarios might still lead to catastrophic outcomes for humans
  • e.g. "after nonviolently taking over, AI systems decide what to do humans, this stub part of the ecosystem; they conclude that they're using too many physical resources, and it would be better to (via legitimate means!) reduce their rights and then cull their numbers, leaving a small population living in something resembling a nature reserve"
  • Perhaps my distaste at this outcome is born in part from loyalty to the human tribe? But I do think that some of it is born from more robust moral intuitions

Anthropic issues questionable letter on SB 1047 (Axios). I can't find a copy of the original letter online. 

This will be unpopular here, but these calls for public flagellation of "EA leaders" amounts to nothing more than scapegoating and trying to direct general disappointment to someone, even though it's not the right people.

Yes it's obviously interesting to speculate about who knew what, but no amount of "institutional reform" can change these things.

Also, this post has quite a conspiratorial tone which strikes me as unhelpful and not particularly truth seeking.

Have you directly asked these people if they're interested (in the headhunting task)? It's sort of a lot to just put something like this on someone's plate (and it doesn't feel to me like a-thing-they've-implicitly-signed-up-for-by-taking-their-role).

I have not. While nobody in EA leadership has weighed in on this explicitly, the general vibe I get is “we don’t need an investigation, and in any case it’d be hard to conduct and we’d need to fund it somehow.” So I’m focusing on arguing the need for an investigation, because without that the other points are moot. And my assumption is that if we build sufficient consensus on the need for an investigation, we could sort out the other issues. If leaders think an investigation is warranted but the logistical problems are insurmountable, they should make that case and then we can get to work on seeing if we can actually solve those logistical problems.

I think the principal challenge for an independent investigation is getting folks with useful information to disclose it, given these people will usually (to some kind and degree) also have 'exposure' to the FTX scandal themselves. 

If I was such a person I would expect working with the investigation would be unpleasant, perhaps embarrassing, plausibly acrimonious, and potentially disastrous for my reputation. What's in it for me?

That may well have been OP’s thinking and they may have been correct about the relative cost effectiveness of community building in GCR vs. GHW. But that doesn’t change the fact that this funding strategy had massive (and IMO problematic) implications for the incentive structure of the entire EA community.

I think it should be fairly uncontroversial that the best way to align the incentives of organizations like CEA with the views and values of the broader community would be if they were funded by organizations/program areas that made decisions using the lens of EA, not subsets of EA like GCR or GHW. OP is free to prioritize whatever it wants, including prioritizing things ahead of aligning CEA’s incentives with those of the EA community. But as things stand significant misalignment of incentives exists, and I think it’s important to acknowledge and spread awareness of that situation.

I don’t see why Effective Altruism is responsible for not detecting fraud at a company before the market writ large did. If those warnings had been so clear and obvious, I would have suggested you invest in FTX’s competitors and bring their malfeasance to light. Otherwise, it is carping after the fact.

I don’t think the comparison about investigating Dustin is particularly apt, as he didn’t have all the complaints/red flags that SBF did. 

And -- if we are talking about 2024 -- there's another reason it doesn't seem like a great comparison to me. Researching catastrophic risks (to one's movement or otherwise) is generally only compelling to the extent that you can mitigate the likelihood and/or effect of those risks. Given the predominance of a single funder, investigating certain risks posed by that funder may not lead to actionable information to reduce risk no matter what the facts are.^[1] At some level of vulnerability, the risk becomes akin to the risk of a massive life-extinguishing asteroid crashing into Earth in the next week; I'm just as dead if I know about it a week in advance rather than seconds in advance.

1. ^{^}

   Of course, certain ethical duties would still exist.

Ideally, the questions this investigation would seek to answer would be laid out and published ahead of time.

Not sure I buy this, on principle -- surely the investigation should have remit to add questions as it goes if they're warranted by information it's turned up? Maybe if the questions for this purpose are more broad principles than specific factual ones it would make sense to me.

Would also be good to pre-publish the principles that would determine what information would be redacted or kept confidential from public communication around findings.

This checks out to me.

As things stand, my low-conviction take is that [headhunting for investigators] would be a reasonable thing for the new non-OP connected EV board members to take on, or perhaps the community health team.

Have you directly asked these people if they're interested (in the headhunting task)? It's sort of a lot to just put something like this on someone's plate (and it doesn't feel to me like a-thing-they've-implicitly-signed-up-for-by-taking-their-role). 

In general my instinct would be more like "work out who feels motivated to have an investigation happen, and then get one (or more) of them to take responsibility for the headhunting".

Ideally, the questions this investigation would seek to answer would be laid out and published ahead of time.

Not sure I buy this, on principle -- surely the investigation should have remit to add questions as it goes if they're warranted by information it's turned up? Maybe if the questions for this purpose are more broad principles than specific factual ones it would make sense to me.

Would also be good to pre-publish the principles that would determine what information would be redacted or kept confidential from public communication around findings.

This checks out to me.

As things stand, my low-conviction take is that [headhunting for investigators] would be a reasonable thing for the new non-OP connected EV board members to take on, or perhaps the community health team.

Have you directly asked these people if they're interested (in the headhunting task)? It's sort of a lot to just put something like this on someone's plate (and it doesn't feel to me like a-thing-they've-implicitly-signed-up-for-by-taking-their-role). 

In general my instinct would be more like "work out who feels motivated to have an investigation happen, and then get one (or more) of them to take responsibility for the headhunting".

I think the principal challenge for an independent investigation is getting folks with useful information to disclose it, given these people will usually (to some kind and degree) also have 'exposure' to the FTX scandal themselves. 

If I was such a person I would expect working with the investigation would be unpleasant, perhaps embarrassing, plausibly acrimonious, and potentially disastrous for my reputation. What's in it for me?

I think the principal challenge for an independent investigation is getting folks with useful information to disclose it, given these people will usually (to some kind and degree) also have 'exposure' to the FTX scandal themselves. 

If I was such a person I would expect working with the investigation would be unpleasant, perhaps embarrassing, plausibly acrimonious, and potentially disastrous for my reputation. What's in it for me?

Is there a site or calendar where we can see upcoming Forum events/ theme weeks? The events often take me by surprise (although draft amnesty week was announced in advance in the newsletter iirc). I believe that I would participate in some of them if I had a reminder about it in my calendar. 

They seem really good! I feel like an idiot for asking this, but where on their website can I subscribe to the newsletter? 

1. I don't think any EA group outside of FTX would take responsibility for having done a lot ($60k+ worth) of due-diligence and investigation of FTX. My impression is that OP considered this as not their job, and CEA was not at all in a position to do this (to biased, was getting funded by FTX). In general, I think that our community doesn't have strong measures in place to investigate funders. For example, I doubt that EA orgs have allocated $60k+ to investigate Dustin Moskovitz (and I imagine he might complain if others did!).
My overall impression was that this was just a large gap that the EA bureaucracy failed at. I similarly think that the "EA bureaucracy" is much weaker / less powerful than I think many imagine it being, and expect that there are several gaps like this. Note that OP/CEA/80k/etc are fairly limited organizations with specific agendas and areas of ownership. 

I’m very sympathetic to the idea that OP is not responsible for anything in this case. But CEA/EV should have done at least the due diligence that fit their official policies developed in the aftermath of Ben Delo affair. I think it’s reasonable for the community to ask whether or not that actually happened. Also, multiple media outlets have reported that CEA did do an investigation after the Alameda dispute. So it would be nice to know if that actually happened and what it found.

I don’t think the comparison about investigating Dustin is particularly apt, as he didn’t have all the complaints/red flags that SBF did. CEA received credible warnings from multiple sources about a CEA board member, and I’d like to think that warrants some sort of action. Which raises another question: if CEA received credible serious concerns about a current board member, what sort of response would CEA’s current policies dictate?

Re: gaps, yes, there are lots of gaps, and the FTX affair exposed some of them. Designing organizational and governance structures that will fix those gaps should be a priority, but I haven’t seen credible evidence that this is happening. So my default assumption is that these gaps will continue to cause problems.

2. I think there were some orange/red flags around, but that it would have taken some real investigation to figure out how dangerous FTX was. I have uncertainty in how difficult it would have been to notice that fraud or similar were happening (I previously assumed this would be near impossible, but am less sure now, after discussions with one EA in finance). I think that the evidence / flags around then were probably not enough to easily justify dramatically different actions at the time, without investigation - other than the potential action of doing a lengthy investigation - but again, that doing one would have been really tough, given the actors involved.

Note that actually pulling off a significant investigation, and then taking corresponding actions, against an actor as powerful as SBF, would be very tough and require a great deal of financial independence.

I very much agree that we shouldn’t be holding EA leaders/orgs/community to a standard of “we should have known FTX was a huge fraud”. I mentioned this in my post, but want to reiterate it here. I feel like this is point where discussions about EA/FTX often get derailed. I don’t believe the people calling for an independent investigation are doing so because they think EA knew/should have known that FTX was a fraud; most of us have said that explicitly.

That said, given what was known at the time, I think it’s pretty reasonable to think that it would have been smart to do some things differently on the margin, e.g. 80k putting SBF on less on a pedestal. A post-mortem could help identify those things and provide insights on how to do better going forward.

3. My impression is that being a board member at CEA was incredibly stressful/intense, in the following months after the FTX collapse. My quick guess is that most of the fallout from the board would have been things like, "I just don't want to have to deal with this anymore" rather than particular disagreements with the organizations. I didn't get the impression that Rebecca's viewpoints/criticisms were very common for other board members/execs, though I'd be curious to get their takes.

This seems like a very important issue. I think one big problem is that other board members/execs are disincentivized to voice concerns they might have, and this is one of the things an independent investigation could help with. Learning that several, or none of, the other board members had concerns similar to Rebecca’s would be very informative, and an investigation could share that sort of finding publicly without compromising any individual’s privacy. 

4. I think that OP / CEA board members haven't particularly focused on / cared about being open and transparent with the EA community. Some of the immediate reason here was that I assume lawyers recommended against speaking up then - but even without that, it's kind of telling how little discussion there has been in the last year or so.

I suggest reading Dustin Moskovitz's comments for some specific examples. Basically, I think that many people in authority (though to be honest, basically anyone who's not a major EA poster/commenter) find "posting to the EA forum and responding to comments" to be pretty taxing/intense, and don't do it much.

Remember that OP staff members are mainly accountable to their managers, not the EA community or others. CEA is mostly funded by OP, so is basically similarly accountable to high-level OP people. (accountable means, "being employed/paid by" here)

Pretty much agree with everything you wrote here. Though I want to emphasize that I think this is a pretty awful outcome, and could be improved with better governance choices such as more community representation, and less OP representation, on CEA’s board. 

If OP doesn’t want to be accountable to the EA community, I think that’s suboptimal though their prerogative. But if CEA is going to take responsibility for community functions (e.g. community health, running effectivealtruism.org, etc.) there should be accountability mechanisms in place. 

I also want to flag that an independent investigation would be a way for people in authority to get their ideas (at least on this topic) out in a less taxing and/or less publicly identifiable way than forum posting/commenting.

5. In terms of power, I think there's a pretty huge power gap between the funders and the rest of the EA community. I don't think that OP really regards themselves as responsible for or accountable to the EA community. My impression is that they fund EA efforts opportunistically, in situations where it seems to help both parties, but don't want to be seen as having any long-term obligations or such. We don't really have strong non-OP funding sources to fund things like "serious investigations into what happened." Personally, I find this situation highly frustrating, and think it gets under-appreciated.

Very well put!

6. My rough impression is that from the standpoint of OP / CEA leaders, there's not a great mystery around the FTX situation, and they also don't see it happening again. So I think there's not that much interest here into a deep investigation.
 

I think Zvi put it well: “a lot of top EA leaders ‘think we know what happened.’ Well, if they know, then they should tell us, because I do not know. I mean, I can guess, but they are not going to like my guess. There is the claim that none of this is about protecting EA’s reputation, you can decide whether that claim is credible.” 

I think the principal challenge for an independent investigation is getting folks with useful information to disclose it, given these people will usually (to some kind and degree) also have 'exposure' to the FTX scandal themselves. 

If I was such a person I would expect working with the investigation would be unpleasant, perhaps embarrassing, plausibly acrimonious, and potentially disastrous for my reputation. What's in it for me?

That may well have been OP’s thinking and they may have been correct about the relative cost effectiveness of community building in GCR vs. GHW. But that doesn’t change the fact that this funding strategy had massive (and IMO problematic) implications for the incentive structure of the entire EA community.

I think it should be fairly uncontroversial that the best way to align the incentives of organizations like CEA with the views and values of the broader community would be if they were funded by organizations/program areas that made decisions using the lens of EA, not subsets of EA like GCR or GHW. OP is free to prioritize whatever it wants, including prioritizing things ahead of aligning CEA’s incentives with those of the EA community. But as things stand significant misalignment of incentives exists, and I think it’s important to acknowledge and spread awareness of that situation.

I can highly recommend following Sentinel's weekly minutes, a weekly update from superforecasters on the likelihood of any events which plausibly could cause worldwide catastrophe.

Perhaps the weekly newsletter I look the most forward to at this point. Read previous issues here: 

https://sentinel-team.org/blog/

What was EV’s official policy post-Ben Delo?

I think the principal challenge for an independent investigation is getting folks with useful information to disclose it, given these people will usually (to some kind and degree) also have 'exposure' to the FTX scandal themselves. 

If I was such a person I would expect working with the investigation would be unpleasant, perhaps embarrassing, plausibly acrimonious, and potentially disastrous for my reputation. What's in it for me?

I think the correct interpretation of this is that OP GHW doesn’t think general community building for its cause areas is cost effective, which seems quite plausible to me. [Edit: note I'm saying community-building in general, not just the EA community specifically - so under this view, the skewing of the EA community is less relevant].

I don’t see why Effective Altruism is responsible for not detecting fraud at a company before the market writ large did. If those warnings had been so clear and obvious, I would have suggested you invest in FTX’s competitors and bring their malfeasance to light. Otherwise, it is carping after the fact.

I've been reviewing some old Forum posts for an upcoming post I'm writing, and incidentally came across this by Howie Lempel for noticing in what spirit you're engaging with someone's ideas:

"Did I ask this question because I think they will have a good answer or because I think they will not have a good answer?"

I felt pretty called out :P

To be fair, I think the latter is sometimes a reasonable persuasive tactic, and it's fine to put yourself in a teaching role rather than a learning role if that's your endorsed intention and the other party is on board. But the value of this quote to me is that it successfully highlights how easily we can tell ourselves we're being intellectually curious, when we're actually doing something else.

Executive summary: Democracies are crucial for societal resilience and forward-thinking decision-making, but their global decline threatens our ability to handle future catastrophic risks.

Key points:

1. Democracies tend to be more resilient and better at managing crises, with processes like national risk assessments and citizens' assemblies.
2. Democratic countries are less likely to go to war with each other, enhancing global stability.
3. The number of democracies worldwide is stagnating or decreasing, particularly liberal democracies.
4. Factors contributing to democratic decline include globalization, inequality, individualism, and corporate influence.
5. Historical transitions to democracy were often unintended, suggesting democracy is not an inevitable outcome.
6. To preserve and strengthen democracies, increased citizen participation and addressing inequality are recommended.

This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.

Ok, so I think we converge pretty much then -- essentially what I am saying is that people concerned about compounding risks would argue that these are not modeled correctly in GBD and that there is much more uncertainty there (and that the estimate is probably an underestimate, from the perspective of taking the compounding risk view seriously).

Coming back to this a year later, I notice that I used to treat EAGs being emotionally exhausting as inevitable, but my most recent EAG (London 2024) was much less difficult.

My best guess about why is that I had previously been misinterpreting my emotional feedback and had an incorrect intuition about what part of the conference was most difficult for me. I wrote more about it on my shortform.

Seeking help from other forum users on projects.

Thanks for writing this Alex, I enjoyed reading it! 
I can relate to the experience of precarious funding situation. It's definitely something to get used to, and the actual experience can vary a lot. 
I think my role is somewhat similar but in the online environment. I have definitely enjoyed the variety of the role a lot and a chance to do more high level work like strategy, which seems like it's been your experience too! 

(Apologies in advance for the messiness from my lack of hyperlinks/footnotes as I'm commenting from my phone and I can't find how to include them using my phone. If anyone knows how to, please let me know)

This is something which has actually briefly crossed my mind before. Looking (briefly) into the subject, I believe an alliance with the climate advocacy movement could be a plausible route of action for the AI safety movement. Though this of course would depend on whether AI would be net-positive or net-negative for the climate. I am not sure yet which it will be, but it is something I believe is certainly worth looking into.

This topic has actually gained some recent media attention (https://edition.cnn.com/2024/07/03/tech/google-ai-greenhouse-gas-emissions-environmental-impact/index.html), although most of the media focus on the intersection of AI and climate change appears to be around how AI can be used as a benefit to the climate.

A few recent academic articles (1. https://www.sciencedirect.com/science/article/pii/S1364032123008778#b111, 2. https://drive.google.com/uc?id=1Wt2_f1RB7ylf7ufD8LviiD6lKkQpQnWZ&export=download, 3. https://link.springer.com/article/10.1007/s00146-021-01294-x, 4. https://wires.onlinelibrary.wiley.com/doi/full/10.1002/widm.1507) go in depth on the topic of how beneficial/harmful AI will be the climate, and basically the conclusions appear somewhat mixed (I admittedly didn't look into them in too much detail), with there being a lot of emissions from ML training and servers, though this paper (https://arxiv.org/pdf/2204.05149) has shown that

"published studies overestimated the cost and carbon footprint of ML training because they didn’t have access to the right information or because they extrapolated point-in-time data without accounting for algorithmic or hardware improvements."

There could also be reasons to think that this may not be the best move particularly if it's association with the climate advocacy movement fuels a larger right-wing x E/Acc counter-movement, though I personally feel it would be better to (at least partly) align with the climate movement to bring the AI slowdown movement further into the mainstream.

Another potential drawback might be if the climate advocates see many in the AI safety crowd as not being as concerned about climate change as they may hope, and they may see the AI safety movement as using the climate movement simply for their own gains, especially given the tech-sceptic sentiment in much of the climate movement (though definitely not all of it). However, I believe the intersection of the AI safety movement with the climate movement would be net-positive as there would be more exposure of the many potential issues associated with the expansion of AI capabilities owing to the climate movement being a very well established lobbying force, and in a sense the tech-scepticism in the climate movement may also be helpful in slowing down progress in AI capabilities.

Also, this alliance may spread more interest in and sympathy towards AI safety within the climate movement (as well as more climate-awareness among AI safety people), as has been done with the climate and animal movements, as well as the AI safety and animal welfare movements.

Another potential risk is if the AI industry is able to become much more efficient (net-zero, net-negative, or even just slightly net-positive in GHG emissions) and/or come up with many good climate solutions (whether aligned or not), which may make climate advocates more optimistic about AI (though this may also be a reason to align with the climate movement to prevent this case from ocurring).

Thanks for the details of your disagreement : )

1. Yeah I think this is a fair point. However, my understanding is that climate action is reasonably popular with the public - even in the US (https://ourworldindata.org/climate-change-support). It's only really when it comes to taking action that the parties differ. So if you advocated for restrictions on large training runs for climate reasons I'm not sure it is obvious that it would necessarily have a downside risk, only that you might get more upside benefits with a democratic administration.  

2.  Yes, I think the argument doesn't make sense if you believe large training runs will be beneficial. Higher emissions seem like a reasonable price to pay for an aligned superintelligence. However, if you think large training runs will result in huge existential risks or otherwise not have upside benefits then that makes them worth avoiding - as the AI slowdown advocacy community argues - and the costs of emissions are clearly not worth paying. 

I think in general most people (and policymakers) are not bought into the idea that advanced AI will cause a technological singularity or be otherwise transformative. The point of this strategy would be to get those people (and policymakers) to take a stance on this issue that aligns with AI safety goals without having to be bought into the transformative effects of AI. 

So while a "Pro-AI" advocate might have to convince people of the transformative power of AI to make a counter-argument, we as "Anti-AI" advocates would only have to point non-AI affiliated people towards the climate effects of AI without having to "AI pill" the public and policymakers. (PauseAI apparently has looked into this already and has a page which gives a sense of what the strategy in this post might look like in practice (https://pauseai.info/environmental))

3. Yes but the question - as @Stephen McAleese noted - "is whether this indirect approach would be more effective than or at least complementary to a more direct approach that advocates explicit compute limits and communicates risks from misaligned AI." So yes national security / competitiveness considerations may regularly trump climate considerations, but if they are trumped less than by safety considerations then they're the better bet. I don't know what the answer to this is but I don't think it's obvious.

I think I remain confused as to what you mean with "all deaths from non-optimal temperature".

I mean the difference between the deaths for the predicted and ideal temperature. From OWID:

The deaths from non-optimal temperature are supposed to cover all causes (temperature is a risk factor for death rather than a cause of death in GBD), not just extreme heat and cold (which only account for a tiny fraction of the deaths; see my last comment). I say "supposed" because it is possible the mortality curves above are not being modelled correctly, and this applies even more to the mortality curves in the future.

So to me it seems you are saying "I don't trust arguments about compounding risks and the data is evidence for that" whereas the data is inherently not set up to include that concern and does not really speak to the arguments that people most concerned about climate risk would make.

My understanding is that (past/present/future) deaths from non-optimal temperature are supposed to include conflict deaths linked to non-optimal temperature. However, I am not confident these are being modelled correctly.

I was not clear, but in my last comment I mostly wanted to say that deaths from non-optimal temperature account for the impact of global warming not only on deaths from extreme heat and cold, but also on cardiovascular or kidney disease, respiratory infections, diabetes and all others (including conflicts). Most causes of death are less heavy-tailed than conflict deaths, so I assume we have a better understanding of how they change with temperature.

I don’t see why Effective Altruism is responsible for not detecting fraud at a company before the market writ large did. If those warnings had been so clear and obvious, I would have suggested you invest in FTX’s competitors and bring their malfeasance to light. Otherwise, it is carping after the fact.

Something like this:

1. I think an obvious risk to this strategy is that it would further polarize AI risk discourse and make it more partisan, given how strongly the climate movement is aligned with Democrats.

2. I think pro-AI forces can reasonably claim that the long-term impacts of accelerated AI development are good for climate -- increased tech acceleration & expanded industrial capacity to build clean energy faster -- so I think the core substantive argument is actually quite weak and transparently so (I think one needs to have weird assumptions if one really believes short-term emissions from getting to AGI would matter from a climate perspective - e.g. if you believed the US would need to double emissions for a decade to get to AGI you would probably still want to bear that cost given how much easier it would make global decarbonization, even if you only looked at it from a climate maximalist lens).

3. If one looks at how national security / competitiveness considerations regularly trump climate considerations and this was true even in a time that was more climate-focused than the next couple of years, then it seems hard to imagine this would really constrain things -- I find it very hard to imagine a situation where a significant part of US policy makers decide they really need to get behind accelerating AGI, but then they don't do it because some climate activists protest this.

So, to me, it seems like a very risky strategy with limited upside, but plenty of downside in terms of further polarization and calling a bluff on what is ultimately an easy-to-disarm argument.

Perhaps Uhlman et al (2015) or Landy & Uhlmann (2018)?

From the latter:

Evidence for this assertion comes from studies involving two jilted lovers (Tannenbaum et al., 2011, Studies 1a and 1b).  Participants were presented with information about two men who had learned that their girlfriends were cheating on them.  Both men flew into a rage, and one beat up his unfaithful girlfriend, while the other beat up her cat.  Participants judged the former action as more immoral, but judged the catbeater as having worse character (specifically, as being more lacking in empathy) than the girlfriend-beater.  This is an example of an act-person dissociation. 

I think the correct interpretation of this is that OP GHW doesn’t think general community building for its cause areas is cost effective, which seems quite plausible to me. [Edit: note I'm saying community-building in general, not just the EA community specifically - so under this view, the skewing of the EA community is less relevant].

Is there a site or calendar where we can see upcoming Forum events/ theme weeks? The events often take me by surprise (although draft amnesty week was announced in advance in the newsletter iirc). I believe that I would participate in some of them if I had a reminder about it in my calendar. 

Thanks!

I think I remain confused as to what you mean with "all deaths from non-optimal temperature".

It is clear that the data source you cite (GBD, focused on current deaths) will not feature nor proxy what people concerned about compounding risks from climate are concerned about.

So to me it seems you are saying "I don't trust arguments about compounding risks and the data is evidence for that" whereas the data is inherently not set up to include that concern and does not really speak to the arguments that people most concerned about climate risk would make.

As said before, I think it is fine to say "I don't trust arguments about compounding risks" and I am probably with you there to a large degree at least compared to people most concerned about this, but I don't think the data from GBD is additional evidence for that mistrust, as far as I can tell.

By crude analogy, if you believed that COVID restrictions had a big toll on the young and this will affect long-run impacts somehow, pointing to few COVID deaths amongs this age cohort would not be evidence against this concern.

Vasco Grilo🔸 @ 2024-07-26T09:38 (+4)

I think I remain confused as to what you mean with "all deaths from non-optimal temperature".

I mean the difference between the deaths for the predicted and ideal temperature. From OWID:

The deaths from non-optimal temperature are supposed to cover all causes (temperature is a risk factor for death rather than a cause of death in GBD), not just extreme heat and cold (which only account for a tiny fraction of the deaths; see my last comment). I say "supposed" because it is possible the mortality curves above are not being modelled correctly, and this applies even more to the mortality curves in the future.

So to me it seems you are saying "I don't trust arguments about compounding risks and the data is evidence for that" whereas the data is inherently not set up to include that concern and does not really speak to the arguments that people most concerned about climate risk would make.

My understanding is that (past/present/future) deaths from non-optimal temperature are supposed to include conflict deaths linked to non-optimal temperature. However, I am not confident these are being modelled correctly.

I was not clear, but in my last comment I mostly wanted to say that deaths from non-optimal temperature account for the impact of global warming not only on deaths from extreme heat and cold, but also on cardiovascular or kidney disease, respiratory infections, diabetes and all others (including conflicts). Most causes of death are less heavy-tailed than conflict deaths, so I assume we have a better understanding of how they change with temperature.

Say you're worried about any take-over-the-world actions, violent or not -- in which case this argument about the advantages of non-violent takeover is of scant comfort;

This is reasonable under the premise that you're worried about any AI takeovers, no matter whether they're violent or peaceful. But speaking personally, peaceful takeover scenarios where AIs just accumulate power—not by cheating us or by killing us via nanobots—but instead by lawfully beating humans fair and square and accumulating almost all the wealth over time, just seem much better than violent takeovers, and not very bad by themselves.

I admit the moral intuition here is not necessarily obvious. I concede that there are plausible scenarios in which AIs are completely peaceful and act within reasonable legal constraints, and yet the future ends up ~worthless. Perhaps the most obvious scenario is the "Disneyland without children" scenario where the AIs go on to create an intergalactic civilization, but in which no one (except perhaps the irrelevant humans still on Earth) is sentient.

But when I try to visualize the most likely futures, I don't tend to visualize a sea of unsentient optimizers tiling the galaxies. Instead, I tend to imagine a transition from sentient biological life to sentient artificial life, which continues to be every bit as cognitively rich, vibrant, and sophisticated as our current world—indeed, it could be even moreso, given what becomes possible at a higher technological and population level.

Worrying about non-violent takeover scenarios often seems to me to arise simply from discrimination against non-biological forms of life, or perhaps a more general fear of rapid technological change, rather than naturally falling out as a consequence of more robust moral intuitions. 

Let me put it another way.

It is often conceded that it was good for humans to take over the world. Speaking broadly, we think this was good because we identify with humans and their aims. We belong to the "human" category of course; but more importantly, we think of ourselves as being part of what might be called the "human tribe", and therefore we sympathize with the pursuits and aims of the human species as a whole. But equally, we could identify as part of the "sapient tribe", which would include non-biological life as well as humans, and thus we could sympathize with the pursuits of AIs, whatever those may be. Under this framing, what reason is there to care much about a non-violent, peaceful AI takeover?

This seems like the wrong meta-level orientation to me. A meta-level orientation that seems better to me is something like "Truth and transparency have strong global benefits, but often don't happen enough because they're locally aversive. So assume that sharing information is useful even when you're not concretely sure how it'll help, and assume by default that power structures (including boards, social networks, etc) are creating negative externalities insofar as they erect barriers to you sharing information".

The specific tradeoff between causing drama and sharing useful information will of course be situation-dependent, but in this situation the magnitude of the issues involved feels like it should significantly outweigh concerns about "stirring up drama", at least if you make attempts to avoid phrasing the information in particularly-provocative or careless ways.

Thanks for this, Vasco, thought-provoking as always!

Likewise! Thanks for the thoughtful comment.

Insofar as is this a correct representation of your argument

It seems like a fair representation.

a. Dying from heat stress is a very extreme outcome and people will act in response to climate change much earlier than dying. For example, before people die from heat stress, they might abandon their livelihoods and migrate, maybe in large numbers.

b. More abstractly, the fact that an extreme impact outcome (heat death) is relatively rare is not evidence for low impact in general. Climate change pressures are not like a disease that kills you within days of exposure and otherwise has no consequence.

Agreed. However:

• I think migration will tend to decrease deaths because people will only want to migrate if they think their lives will improve (relative to the counterfactual of not migrating).
• The deaths from non-optimal temperature I mentioned are supposed to account for all causes of death, not just extreme heat and cold. According to GBD, in 2021, deaths from environmental heat and cold exposure were 36.0 k (I guess this is what you are referring to by heat stress), which was just 1.88 % (= 36.0*10^3/(1.91*10^6)) of the 1.91 M deaths from non-optimal temperature. My post is about how these 1.91 M deaths would change.

a. You seem to suggest we are very uncertain about many of the effect signs. I think the basic argument why people concerned about climate change would argue that changes will be negative and that there be compounding risks is because natural and human systems are adapted to specific climate conditions. That doesn't mean they cannot adapt at all, but that does mean that we should expect it is more likely that effects are negative, at least as short-term shocks, than positive for welfare.

This makes sense. On the other hand, one could counter global warming will be good because:

• There are more deaths from low temperature than from high temperature.
• The disease burden per capita from non-optimal temperature has so far been decreasing (see 2nd to last graph).

b. I think a lot of the other arguments on the side of "indirect risks are low" you cite are ultimately of the form (i) "indirect effects in other causes are also large" or (ii) "pointing to indirect effects make things inscrutable and unverifiable".  (i) might be true but doesn't matter, I think, for the question of whether warming is net-bad and (ii) is also true, but does nothing by itself on whether those indirect effects are real -- we can live in a world where indirect effects are rhetorically abused and still exist and indeed dominate in certain situations!

Agreed. I would just note that i) can affect prioritisation across causes.

I basically agree with this take. I think an investigation conducted from within the EA community (by someone(s) with a bit of distance from FTX) makes a lot more sense than Mintz v2. Ideally, the questions this investigation would seek to answer would be laid out and published ahead of time. Would also be good to pre-publish the principles that would determine what information would be redacted or kept confidential from public communication around findings.

This is then kind of a headhunting task; but who would take responsibility for that?

If we had one or more ombudspeople or explicit community representation on the CEA board (which I really wish we did), this would be a great role for them. As things stand, my low-conviction take is that this would be a reasonable thing for the new non-OP connected EV board members to take on, or perhaps the community health team. I have some reservations about having CEA involved in this, but also give a lot of weight to Rebecca saying "CEA is a logical place to house" an investigation.  

Personally, I’d consider Rethink Priorities to be kind of the default choice to do an investigation; I’ve seen others toss their name around too. It’d be nice to have some process for generating other candidates (e.g. community health coming up with a few) and then some method of finding which of the options had the most community buy-in (e.g. ranked choice voting among everyone who has filled out the EA survey sometime in the last three years; I don’t think this would be an ideal methodology but it’s at least loosely in the ballpark of ways of finding an investigator that the community would find credible).

I would be curious to hear what the nature of your disagreement is : )

I disagree with the substance, but I don't understand why it gets downvoted.

In 2023, 80% of CEA’s budget came from OP’s GCRCB team. This creates an obvious incentive for CEA to prioritize the stuff the GCRCB team prioritizes.

As its name suggests, the GCRCB team has an overt focus on Global Catastrophic Risks. Here’s how OP’s website describes this team:

We want to increase the number of people who aim to prevent catastrophic events, and help them to achieve their goals.

We believe that scope-sensitive giving often means focusing on the reduction of global catastrophic risks — those which could endanger billions of people. We support organizations and projects that connect and support people who want to work on these issues, with a special focus on biosecurity and risks from advanced AI. In doing so, we hope to grow and empower the community of people focused on addressing threats to humanity and protecting the future of human civilization.

The work we fund in this area is primarily focused on identifying and supporting people who are or could eventually become helpful partners, critics, and grantees.

This team was formerly known as “Effective Altruism Community Growth (Longtermism).”

CEA has also received a much smaller amount of funding from OP’s “Effective Altruism (Global Health and Wellbeing)” team. From what I can tell, the GHW team basically focuses on meta charities doing global poverty type and animal welfare work (often via fundraising for effective charities in those fields). The OP website notes: 

“This focus area uses the lens of our global health and wellbeing portfolio, just as our global catastrophic risks capacity building area uses the lens of our GCR portfolio... Our funding so far has focused on [grantees that] Raise funds for highly effective charities, Enable people to have a greater impact with their careers, and found and incubate new charities working on important and neglected interventions.”

There is an enormous difference between these teams in terms of their historical and ongoing impact on EA funding and incentives. The GCRCB team has granted over $400 million since 2016, including over $70 million to CEA and over $25 million to 80k. Compare that to the GHW which launched “in July 2022. In its first 12 months, the program had a budget of $10 million.” 

So basically there’s been a ton of funding for a long time for EA community building that prioritizes AI/Bio/other GCR work, and a vastly smaller amount of funding that only became available recently for EA community building that uses a global poverty/animal welfare lens. And, as your question suggests, this dynamic is not at all well understood.

1. I don't think any EA group outside of FTX would take responsibility for having done a lot ($60k+ worth) of due-diligence and investigation of FTX. My impression is that OP considered this as not their job, and CEA was not at all in a position to do this (to biased, was getting funded by FTX). In general, I think that our community doesn't have strong measures in place to investigate funders. For example, I doubt that EA orgs have allocated $60k+ to investigate Dustin Moskovitz (and I imagine he might complain if others did!).
My overall impression was that this was just a large gap that the EA bureaucracy failed at. I similarly think that the "EA bureaucracy" is much weaker / less powerful than I think many imagine it being, and expect that there are several gaps like this. Note that OP/CEA/80k/etc are fairly limited organizations with specific agendas and areas of ownership. 

I’m very sympathetic to the idea that OP is not responsible for anything in this case. But CEA/EV should have done at least the due diligence that fit their official policies developed in the aftermath of Ben Delo affair. I think it’s reasonable for the community to ask whether or not that actually happened. Also, multiple media outlets have reported that CEA did do an investigation after the Alameda dispute. So it would be nice to know if that actually happened and what it found.

I don’t think the comparison about investigating Dustin is particularly apt, as he didn’t have all the complaints/red flags that SBF did. CEA received credible warnings from multiple sources about a CEA board member, and I’d like to think that warrants some sort of action. Which raises another question: if CEA received credible serious concerns about a current board member, what sort of response would CEA’s current policies dictate?

Re: gaps, yes, there are lots of gaps, and the FTX affair exposed some of them. Designing organizational and governance structures that will fix those gaps should be a priority, but I haven’t seen credible evidence that this is happening. So my default assumption is that these gaps will continue to cause problems.

2. I think there were some orange/red flags around, but that it would have taken some real investigation to figure out how dangerous FTX was. I have uncertainty in how difficult it would have been to notice that fraud or similar were happening (I previously assumed this would be near impossible, but am less sure now, after discussions with one EA in finance). I think that the evidence / flags around then were probably not enough to easily justify dramatically different actions at the time, without investigation - other than the potential action of doing a lengthy investigation - but again, that doing one would have been really tough, given the actors involved.

Note that actually pulling off a significant investigation, and then taking corresponding actions, against an actor as powerful as SBF, would be very tough and require a great deal of financial independence.

I very much agree that we shouldn’t be holding EA leaders/orgs/community to a standard of “we should have known FTX was a huge fraud”. I mentioned this in my post, but want to reiterate it here. I feel like this is point where discussions about EA/FTX often get derailed. I don’t believe the people calling for an independent investigation are doing so because they think EA knew/should have known that FTX was a fraud; most of us have said that explicitly.

That said, given what was known at the time, I think it’s pretty reasonable to think that it would have been smart to do some things differently on the margin, e.g. 80k putting SBF on less on a pedestal. A post-mortem could help identify those things and provide insights on how to do better going forward.

3. My impression is that being a board member at CEA was incredibly stressful/intense, in the following months after the FTX collapse. My quick guess is that most of the fallout from the board would have been things like, "I just don't want to have to deal with this anymore" rather than particular disagreements with the organizations. I didn't get the impression that Rebecca's viewpoints/criticisms were very common for other board members/execs, though I'd be curious to get their takes.

This seems like a very important issue. I think one big problem is that other board members/execs are disincentivized to voice concerns they might have, and this is one of the things an independent investigation could help with. Learning that several, or none of, the other board members had concerns similar to Rebecca’s would be very informative, and an investigation could share that sort of finding publicly without compromising any individual’s privacy. 

4. I think that OP / CEA board members haven't particularly focused on / cared about being open and transparent with the EA community. Some of the immediate reason here was that I assume lawyers recommended against speaking up then - but even without that, it's kind of telling how little discussion there has been in the last year or so.

I suggest reading Dustin Moskovitz's comments for some specific examples. Basically, I think that many people in authority (though to be honest, basically anyone who's not a major EA poster/commenter) find "posting to the EA forum and responding to comments" to be pretty taxing/intense, and don't do it much.

Remember that OP staff members are mainly accountable to their managers, not the EA community or others. CEA is mostly funded by OP, so is basically similarly accountable to high-level OP people. (accountable means, "being employed/paid by" here)

Pretty much agree with everything you wrote here. Though I want to emphasize that I think this is a pretty awful outcome, and could be improved with better governance choices such as more community representation, and less OP representation, on CEA’s board. 

If OP doesn’t want to be accountable to the EA community, I think that’s suboptimal though their prerogative. But if CEA is going to take responsibility for community functions (e.g. community health, running effectivealtruism.org, etc.) there should be accountability mechanisms in place. 

I also want to flag that an independent investigation would be a way for people in authority to get their ideas (at least on this topic) out in a less taxing and/or less publicly identifiable way than forum posting/commenting.

5. In terms of power, I think there's a pretty huge power gap between the funders and the rest of the EA community. I don't think that OP really regards themselves as responsible for or accountable to the EA community. My impression is that they fund EA efforts opportunistically, in situations where it seems to help both parties, but don't want to be seen as having any long-term obligations or such. We don't really have strong non-OP funding sources to fund things like "serious investigations into what happened." Personally, I find this situation highly frustrating, and think it gets under-appreciated.

Very well put!

6. My rough impression is that from the standpoint of OP / CEA leaders, there's not a great mystery around the FTX situation, and they also don't see it happening again. So I think there's not that much interest here into a deep investigation.
 

I think Zvi put it well: “a lot of top EA leaders ‘think we know what happened.’ Well, if they know, then they should tell us, because I do not know. I mean, I can guess, but they are not going to like my guess. There is the claim that none of this is about protecting EA’s reputation, you can decide whether that claim is credible.” 

Can you say more about why the distinction between "Open Philanthropy" and "Open Philanthropy GCRCB team" matters? What subset of the community does this GCRCB team align with vs not? I've never heard this before

My model is that the "community" doesn't really have much power directly, at this point. OP has power, and to the extent that they fund certain groups (at this point, when funding is so centralized), CEA and a few other groups have power.

I more or less agree with this. Though I think some of CEA’s power derives not only from having OP funding, but also the type of work CEA does (e.g. deciding who attends and talks at EAG). And other orgs and individuals have power related to reputation, quality of work, and ability to connect people with resources (money, jobs, etc). 

Regarding how different parts of the community might be able to implement changes, it might be helpful to think about “top-down” vs. “bottom-up” reforms.

Top-down reforms would be initiated by the orgs/people that already have power. The problem, as you note, is that “OP and these other top EA groups feel like they just have a lot going on, and aren't well positioned to do other significant reforms/changes.” (There may also be an issue whereby people with power don’t like to give it up.) But some changes are already in the works, most notably the EV breakup. This creates lots of opportunities to fix past problems, e.g. around board composition since there will be a lot more boards in the post-breakup world. Examples I’d like to see include: 

• EA ombudsperson/people on CEA’s board and/or advisory panel of with representatives from different parts of the community. (There used to be an advisory panel of this sort, but from what I understand they were consulted for practically, or perhaps literally, nothing).
• Reduced reliance on OP’s GCRCB program as the overwhelmingly dominant funder of EA orgs. I’d like it even more if we could find a way to reduce reliance on OP overall as a funder, but that would require finding new money (hard) or making do with less money (bad). But even if OP shifted to funding CEA and other key EA orgs from a roughly even mix of its Global Catastrophic Risks and its Global Health and Wellbeing teams, that would be an enormous improvement IMO. 
  • The fact that key EA orgs (including ones responsible for functions on behalf of the community) are overwhelmingly funded by a program which has priorities that only align with a subset of the community is IMO the most problematic incentive structure in EA. Compounding this problem, I think awareness of this dynamic is generally quite limited (people think of CEA as being funded by OP, not by OP’s GCRCB program), and appreciation of its implications even more so.
• Vastly expanding the universe of people who serve on the boards of organizations that have power, and hopefully including more community representation on those boards.
• Creating, implementing, and sharing good organizational policies around COI, donor due diligence, whistleblower protections, etc. (Note that this is supposedly in process.^[1])

Bottom up reforms would be initiated by lay-EAs, the folks who make up the vast majority of the community. The obstacles to bottom up reforms are finding ways to fund them and coordinate them; almost by definition these people aren’t organized. 

Examples I’d like to see include: 

• People starting dedicated projects focused on improving EA governance (broadly defined)
  • This could also involve a contest to identify (and incentivize the identification of) the best ideas
• Establishment of some sort of coalition to facilitate coordination between local groups. I think “groups as a whole” could serve as a decentralized locus of power that could serve as a counterbalance to the existing centralized power bases. But right now, I don’t get the impression that there are good ways for groups to coordinate.
• EAIF focusing on and/or earmarking some percentage of grantmaking towards improving EA governance (broadly defined). As mentioned earlier, lack of funding is a bit obstacle for bottom up reforms, so the EAIF (~$20m in grants since start of 2020) could be a huge help.  
• Individual EAs acting empowered to improve governance (broadly defined), e.g. publicly voicing support for various reforms, calling out problems they see, incorporating governance issues into their giving decisions, serving on boards, etc) 

1. ^{^}

   In December, Zach Robinson wrote: “EV also started working on structural improvements shortly after FTX’s collapse and continued to do so alongside the investigation. Over the past year, we have implemented structural governance and oversight improvements, including restructuring the way the two EV charities work together, updating and improving key corporate policies and procedures at both charities, increasing the rigor of donor due diligence, and staffing up the in-house legal departments. Nevertheless, good governance and oversight is not a goal that can ever be definitively ‘completed’, and we’ll continue to iterate and improve. We plan to open source those improvements where feasible so the whole EA ecosystem can learn from EV’s challenges and benefit from the work we’ve done.” 

   Open sourcing these improvements would be terrific, though to the best of my knowledge this hasn’t actually happened yet, which is disappointing. Though this stuff has been shared and I've just missed it.

I've been contemplating writing a post about my side of the issue. I wasn't particularly close, but did get a chance to talk to some of the people involved.

Here's my rough take, at this point:
1. I don't think any EA group outside of FTX would take responsibility for having done a lot ($60k+ worth) of due-diligence and investigation of FTX. My impression is that OP considered this as not their job, and CEA was not at all in a position to do this (to biased, was getting funded by FTX). In general, I think that our community doesn't have strong measures in place to investigate funders. For example, I doubt that EA orgs have allocated $60k+ to investigate Dustin Moskovitz (and I imagine he might complain if others did!).
My overall impression was that this was just a large gap that the EA bureaucracy failed at. I similarly think that the "EA bureaucracy" is much weaker / less powerful than I think many imagine it being, and expect that there are several gaps like this. Note that OP/CEA/80k/etc are fairly limited organizations with specific agendas and areas of ownership. 

2. I think there were some orange/red flags around, but that it would have taken some real investigation to figure out how dangerous FTX was. I have uncertainty in how difficult it would have been to notice that fraud or similar were happening (I previously assumed this would be near impossible, but am less sure now, after discussions with one EA in finance). I think that the evidence / flags around then were probably not enough to easily justify dramatically different actions at the time, without investigation - other than the potential action of doing a lengthy investigation - but again, that doing one would have been really tough, given the actors involved.

Note that actually pulling off a significant investigation, and then taking corresponding actions, against an actor as powerful as SBF, would be very tough and require a great deal of financial independence.

3. My impression is that being a board member at CEA was incredibly stressful/intense, in the following months after the FTX collapse. My quick guess is that most of the fallout from the board would have been things like, "I just don't want to have to deal with this anymore" rather than particular disagreements with the organizations. I didn't get the impression that Rebecca's viewpoints/criticisms were very common for other board members/execs, though I'd be curious to get their takes.

4. I think that OP / CEA board members haven't particularly focused on / cared about being open and transparent with the EA community. Some of the immediate reason here was that I assume lawyers recommended against speaking up then - but even without that, it's kind of telling how little discussion there has been in the last year or so.

I suggest reading Dustin Moskovitz's comments for some specific examples. Basically, I think that many people in authority (though to be honest, basically anyone who's not a major EA poster/commenter) find "posting to the EA forum and responding to comments" to be pretty taxing/intense, and don't do it much.

Remember that OP staff members are mainly accountable to their managers, not the EA community or others. CEA is mostly funded by OP, so is basically similarly accountable to high-level OP people. (accountable means, "being employed/paid by" here)

5. In terms of power, I think there's a pretty huge power gap between the funders and the rest of the EA community. I don't think that OP really regards themselves as responsible for or accountable to the EA community. My impression is that they fund EA efforts opportunistically, in situations where it seems to help both parties, but don't want to be seen as having any long-term obligations or such. We don't really have strong non-OP funding sources to fund things like "serious investigations into what happened." Personally, I find this situation highly frustrating, and think it gets under-appreciated.
 

6. My rough impression is that from the standpoint of OP / CEA leaders, there's not a great mystery around the FTX situation, and they also don't see it happening again. So I think there's not that much interest here into a deep investigation.
 

So, in summary, my take is less, "there was some conspiracy where a few organizations did malicious things," and more, "the EA bureaucracy has some significant weaknesses that were highlighted here." 

Note: Some of my thinking on this comes from my time at the reform group. We spent some time coming up with a list of potential reform projects, including having better investigative abilities. My impression is that there generally hasn't been much concern/interest in this space.
 

I think this case it's ok (but happy to change my mind) - afaict he owns the connection now and the two names are a bit like separate personas. He's gone on podcasts under his true name, for instance.

we haven’t changed the community’s incentive structures in a way that will prevent those same sorts of mistakes going forward

I'm curious what your model is of the "community" - how would it significantly change on this issue?

My model is that the "community" doesn't really have much power directly, at this point. OP has power, and to the extent that they fund certain groups (at this point, when funding is so centralized), CEA and a few other groups have power.

I could see these specific organizations doing reforms, if/when they want to. I could also see some future where the "EA community" bands together to fund their own, independent, work. I'm not sure what other options there are.

Right now, my impression is that OP and these other top EA groups feel like they just have a lot going on, and aren't well positioned to do other significant reforms/changes. 

Congrats! These are great results and it looks like you're scaling really well for a very early-stage org :)

I'm curious about that 9-point PHQ-9 reduction goal. How did you decide on it? Do you think it's achievable (especially since you saw a much larger reduction in your pilot)? Why do you think you saw such a large difference in reductions between the pilot and now? And finally, do you think focusing on increasing effect size will take effort away from cost-reduction efforts?

My read is that you can apply the framework two different ways:

• Say you're worried about any take-over-the-world actions, violent or not -- in which case this argument about the advantages of non-violent takeover is of scant comfort;
• Say you're only worried about violent take-over-the-world actions, in which case your argument fits into the framework under "non-takeover satisfaction": how good the AI feels about its best benign alternative action.

I don't think you're wrong exactly, but AI takeover doesn't have to happen through a single violent event, or through a treacherous turn or whatever. All of your arguments also apply to the situation with H sapiens and H neanderthalensis, but those factors did not prevent the latter from going extinct largely due to the activities of the former:

1. There was a cost to violence that humans did against neanderthals
2. The cost of using violence was not obviously smaller than the benefits of using violence -- there was a strong motive for the neanderthals to fight back, and using violence risked escalation, whereas peaceful trade might have avoided those risks
3. There was no one human that controlled everything; in fact, humans likely often fought against one another
4. You allow for neanderthals to be less capable or coordinated than humans in this analogy, which they likely were in many ways

The fact that those considerations were not enough to prevent neanderthal extinction is one reason to think they are not enough to prevent AI takeover, although of course the analogy is not perfect or conclusive, and it's just one reason among several. A couple of relevant parallels include:

• If alignment is very hard, that could mean AIs compete with us over resources that we need to survive or flourish (e.g., land, energy, other natural resources), similar to how humans competed over resources with neanderthals
• The population of AIs may be far larger, and grow more rapidly, than the population of humans, similar to how human populations were likely larger and growing at a faster rate than those of neanderthals

Happy to share some thoughts (and not thereby signalling that I plan not to say more about the object-level):

• Independent investigations are by their nature somewhat weird
  • You get someone coming in with less context, which makes it harder for them to discover things (relative to someone with more context)
  • But they also get to dodge group think or similar issues
• There are, as I see it, two different purposes of independent investigations:
  1. actually gaining insight into the situation
  2. being able to credibly signal that the conclusions are fair/independent/untainted by bias or groupthink
• There's a spectrum of different notions of "independent" which could be at play here:
  • Independent = largely but not completely unconnected with the FTX cluster
  • Independent = unconnected with the FTX cluster, but still in the EA sphere
  • Independent = unconnected with EA
  • The greater the independence, the higher the costs of the investigation, but if it's done well, the more robust the benefits
• Whether it's worth having an independent investigation, and of what kind, depends on:
  • The relative costs of different types of investigation
  • How much people might reasonably learn
  • How much pain there is from distrust that might be helpfully dispelled by an independent investigation
  • What risks, if any, are thereby created? (ammunition for media hit-pieces? chance of sparking vexatious lawsuits?)

In this case:

• Given the existence of the EV-commissioned investigation by Mintz (at significant expense), it seems somewhat weird to me that EV didn't publish more of a summary of the findings
  • I think there are lots of reasons they might not have wanted to publish the full investigation, and feel relatively sympathetic to their not having done that
  • I can imagine there are various risks-of-exposure from publishing even a summary, and they may have been advised by professionals whose job it is to monitor and guard against exposure (lawyers and/or PR folks) to play it safe
  • Nevertheless my guess is that if I were privy to the considerations, I would have thought that the better path involved sharing rather more with the EA community
• At this point I don't think it's likely to be worth another fully-independent investigation, as from a law firm
  • They're very expensive
  • Some of the most interesting questions will ultimately be judgement calls, which means that in order to derive value from it you have to have high trust in the judgement of the people performing the investigation
  • Some of the trust it would facilitate doesn't seem threatened (e.g. there doesn't seem to be any concern that there was a huge cover-up or anything)
• I do think it might well be worth an investigation by someone (or some few) in EA, but not connected to FTX
  • Partially because there seems to be a good amount of appetite for it from the EA community; partially because I think that's probably at the sweet spot of "people most likely to have useful critical takes about how to do things"
  • The principal challenge IMO is finding someone(s) who will:
    • Have good sensible takes on things
    • Be sufficiently non-consequentialist that their takes can be trusted to be "fair assessments" not "things they think will be most likely to lead to good outcomes"
    • Have minimal (if any) conflicts of interest
      • Ideally no connections to FTX
      • Also not beholden to anyone who might be reasonably criticised by an investigation (or whom outside observers might suspect of having that status)
    • Can credibly signal the above, so that their takes can be trusted by a broader audience
    • Be willing to spend time on it (and motivated to do a good job)
  • I think if there was someone who looked good for this, and it looked like a serious and legitimate attempt at an independent investigation, then it probably wouldn't be too challenging to get people to put in some money to pay for their time, and it wouldn't be too challenging to secure cooperation from enough people-that-they'd-like-to-interview
  • This is then kind of a headhunting task; but who would take responsibility for that?
    • It ideally shouldn't be the folks who have too much in the way of connections with FTX
      • Else the choice of person might be seen as suspect?
    • There's maybe a problem where a lot of the central community infrastructure does have some FTX connections, and other folks don't naturally read this as their responsibility?

I'm not sure I fully understand this framework, and thus I could easily have missed something here, especially in the section about "Takeover-favoring incentives". However, based on my limited understanding, this framework appears to miss the central argument for why I am personally not as worried about AI takeover risk as most EAs seem to be.

Here's a concise summary of my own argument for being less worried about takeover risk:

1. There is a cost to violently taking over the world, in the sense of acquiring power unlawfully or destructively with the aim of controlling everything in the whole world, relative to the alternative of simply gaining power lawfully and peacefully, even for agents that don't share 'our' values.
   1. For example, as a simple alternative to taking over the world, an AI could advocate for the right to own their own labor and then try to accumulate wealth and power lawfully by selling their services to others, which would earn them the ability to purchase a gargantuan number of paperclips without much restraint.
2. The cost of violent takeover is not obviously smaller than the benefits of violent takeover, given the existence of lawful alternatives to violent takeover. This is for two main reasons:
   1. In order to wage a war to take over the world, you generally need to pay costs fighting the war, and there is a strong motive for everyone else to fight back against you if you try, including other AIs who do not want you to take over the world (and this includes any AIs whose goals would be hindered by a violent takeover, not just those who are "aligned with humans"). Empirically, war is very costly and wasteful, and less efficient than compromise, trade, and diplomacy.
   2. Violently taking over the war is very risky, since the attempt could fail, and you could be totally shut down and penalized heavily if you lose. There are many ways that violent takeover plans could fail: your takeover plans could be exposed too early, you could also be caught trying to coordinate the plan with other AIs and other humans, and you could also just lose the war. Ordinary compromise, trade, and diplomacy generally seem like better strategies for agents that have at least some degree of risk-aversion.
3. There isn't likely to be "one AI" that controls everything, nor will there likely be a strong motive for all the silicon-based minds to coordinate as a unified coalition against the biological-based minds, in the sense of acting as a single agentic AI against the biological people. Thus, future wars of world conquest (if they happen at all) will likely be along different lines than AI vs. human. 
   1. For example, you could imagine a coalition of AIs and humans fighting a war against a separate coalition of AIs and humans, with the aim of establishing control over the world. In this war, the "line" here is not drawn cleanly between humans and AIs, but is instead drawn across a different line. As a result, it's difficult to call this an "AI takeover" scenario, rather than merely a really bad war.
4. Nothing about this argument is intended to argue that AIs will be weaker than humans in aggregate, or individually. I am not claiming that AIs will be bad at coordinating or will be less intelligent than humans. I am also not saying that AIs won't be agentic or that they won't have goals or won't be consequentialists, or that they'll have the same values as humans. I'm also not talking about purely ethical constraints: I am referring to practical constraints and costs on the AI's behavior. The argument is purely about the incentives of violently taking over the world vs. the incentives to peacefully cooperate within a lawful regime, between both humans and other AIs.

5. A big counterargument to my argument seems well-summarized by this hypothetical statement (which is not an actual quote, to be clear): "if you live in a world filled with powerful agents that don't fully share your values, those agents will have a convergent instrumental incentive to violently take over the world from you". However, this argument proves too much. 
   
   We already live in a world where, if this statement was true, we would have observed way more violent takeover attempts than what we've actually observed historically.

   For example, I personally don't fully share values with almost all other humans on Earth (both because of my indexical preferences, and my divergent moral views) and yet the rest of the world has not yet violently disempowered me in any way that I can recognize.

What types of requests ? Like feature updates to the forum, or seeking help from other forum users on projects ?

I don't know if you're even implying this, but the causal mechanism for altruism arising in humans doesn't need to hold any moral force over us. Just because kin selection caused us to be altruistic, doesn't mean we need to think "what would kin selection want?" when deciding how to be altruistic in future. We can replace the causal origin with our own moral foundations, and follow those instead.

I'm not sure I fully understand this framework, and thus I could easily have missed something here, especially in the section about "Takeover-favoring incentives". However, based on my limited understanding, this framework appears to miss the central argument for why I am personally not as worried about AI takeover risk as most EAs seem to be.

Here's a concise summary of my own argument for being less worried about takeover risk:

1. There is a cost to violently taking over the world, in the sense of acquiring power unlawfully or destructively with the aim of controlling everything in the whole world, relative to the alternative of simply gaining power lawfully and peacefully, even for agents that don't share 'our' values.
   1. For example, as a simple alternative to taking over the world, an AI could advocate for the right to own their own labor and then try to accumulate wealth and power lawfully by selling their services to others, which would earn them the ability to purchase a gargantuan number of paperclips without much restraint.
2. The cost of violent takeover is not obviously smaller than the benefits of violent takeover, given the existence of lawful alternatives to violent takeover. This is for two main reasons:
   1. In order to wage a war to take over the world, you generally need to pay costs fighting the war, and there is a strong motive for everyone else to fight back against you if you try, including other AIs who do not want you to take over the world (and this includes any AIs whose goals would be hindered by a violent takeover, not just those who are "aligned with humans"). Empirically, war is very costly and wasteful, and less efficient than compromise, trade, and diplomacy.
   2. Violently taking over the war is very risky, since the attempt could fail, and you could be totally shut down and penalized heavily if you lose. There are many ways that violent takeover plans could fail: your takeover plans could be exposed too early, you could also be caught trying to coordinate the plan with other AIs and other humans, and you could also just lose the war. Ordinary compromise, trade, and diplomacy generally seem like better strategies for agents that have at least some degree of risk-aversion.
3. There isn't likely to be "one AI" that controls everything, nor will there likely be a strong motive for all the silicon-based minds to coordinate as a unified coalition against the biological-based minds, in the sense of acting as a single agentic AI against the biological people. Thus, future wars of world conquest (if they happen at all) will likely be along different lines than AI vs. human. 
   1. For example, you could imagine a coalition of AIs and humans fighting a war against a separate coalition of AIs and humans, with the aim of establishing control over the world. In this war, the "line" here is not drawn cleanly between humans and AIs, but is instead drawn across a different line. As a result, it's difficult to call this an "AI takeover" scenario, rather than merely a really bad war.
4. Nothing about this argument is intended to argue that AIs will be weaker than humans in aggregate, or individually. I am not claiming that AIs will be bad at coordinating or will be less intelligent than humans. I am also not saying that AIs won't be agentic or that they won't have goals or won't be consequentialists, or that they'll have the same values as humans. I'm also not talking about purely ethical constraints: I am referring to practical constraints and costs on the AI's behavior. The argument is purely about the incentives of violently taking over the world vs. the incentives to peacefully cooperate within a lawful regime, between both humans and other AIs.

5. A big counterargument to my argument seems well-summarized by this hypothetical statement (which is not an actual quote, to be clear): "if you live in a world filled with powerful agents that don't fully share your values, those agents will have a convergent instrumental incentive to violently take over the world from you". However, this argument proves too much. 
   
   We already live in a world where, if this statement was true, we would have observed way more violent takeover attempts than what we've actually observed historically.

   For example, I personally don't fully share values with almost all other humans on Earth (both because of my indexical preferences, and my divergent moral views) and yet the rest of the world has not yet violently disempowered me in any way that I can recognize.

Thanks for this, Vasco, thought-provoking as always!

I do not have too much time to discuss this, but I want to point out that I am pretty unconvinced by the argument for why indirect effects should be easy to discount by the type of the argument you make. So, to be clear, I am not arguing that the conclusion is necessarily wrong (I think this would require stronger arguments), but rather that the argumentation strategy here does not work.

As far as I understand it your argument for "indirect effects cannot be very large" is something like:

1. Deaths from heat are very overstated and not that significant and they capture the most important direct effect.

2. A claim for strong indirect effects is then implausible because onto that direct effect you need to sequence a bunch of very uncertain additional effects with uncertain signs.

Insofar as is this a correct representation of your argument -- please let me know -- I think it has a couple of problems:

1.

a. Dying from heat stress is a very extreme outcome and people will act in response to climate change much earlier than dying. For example, before people die from heat stress, they might abandon their livelihoods and migrate, maybe in large numbers.

b. More abstractly, the fact that an extreme impact outcome (heat death) is relatively rare is not evidence for low impact in general. Climate change pressures are not like a disease that kills you within days of exposure and otherwise has no consequence.

2.

a. You seem to suggest we are very uncertain about many of the effect signs. I think the basic argument why people concerned about climate change would argue that changes will be negative and that there be compounding risks is because natural and human systems are adapted to specific climate conditions. That doesn't mean they cannot adapt at all, but that does mean that we should expect it is more likely that effects are negative, at least as short-term shocks, than positive for welfare. Insofar as you buy a story where people migrate because of climate impacts, for example, I don't think it is unreasonable to say that increased migration pressures are more likely to increase tensions than to reduce them, etc.

b. I think a lot of the other arguments on the side of "indirect risks are low" you cite are ultimately of the form (i) "indirect effects in other causes are also large" or (ii) "pointing to indirect effects make things inscrutable and unverifiable".  (i) might be true but doesn't matter, I think, for the question of whether warming is net-bad and (ii) is also true, but does nothing by itself on whether those indirect effects are real -- we can live in a world where indirect effects are rhetorically abused and still exist and indeed dominate in certain situations!
 

Vasco, how do your estimates account for model uncertainty? I don't understand how you can put some probability on something being possible (i.e. p(extinction|nuclear war) > 0), but end up with a number like 5.93e-12 (i.e. 1 in ~160 billion). That implies an extremely, extremely high level of confidence. Putting ~any weight on models that give higher probabilities would lead to much higher estimates.

The people directory placed into the forum is such a great idea! I've stumbled on a few defunct or rarely updated EA directories and the working ones e.g. High impact professionals seem very useful. I hope it brings more opportunities to qualified people and builds up friendships and connections :)

I can't recall the paper, but I remember reading a paper in moral psychology that argues that on a psychological level, we think of morality in terms of 'is this person moral', not 'is this act moral'. We are trying to figure out if the person in front of us is trustworthy, loyal, kind, etc.

In the study, participants do say that a human experiencing harm is worse than an animal experiencing harm, but view a person who hits a cat as more immoral than a person who hits their spouse. I think what people are implicitly recoiling at is that the person who hits a cat is more likely to be a psychopath. 

I think this maps pretty well onto the example here, and the outrage of people's reactions. And to clarify, I think this explanation captures WHY people react the way they do in the descriptive sense. I don't think that's how people ought to react. 

I've never heard this idea proposed before so it seems novel and interesting.

As you say in the post, the AI risk movement could gain much more awareness by associating itself with the climate risk advocacy movement which is much larger. Compute is arguably the main driver of AI progress, compute is correlated with energy usage, and energy use generally increases carbon emissions so limiting carbon emissions from AI is an indirect way of limiting the compute dedicated to AI and slowing down the AI capabilities race.

This approach seems viable in the near future until innovations in energy technology (e.g. nuclear fusion) weaken the link between energy production and CO2 emissions, or algorithmic progress reduces the need for massive amounts of compute for AI.

The question is whether this indirect approach would be more effective than or at least complementary to a more direct approach that advocates explicit compute limits and communicates risks from misaligned AI.

Thanks for this, Vasco, thought-provoking as always!

I do not have too much time to discuss this, but I want to point out that I am pretty unconvinced by the argument for why indirect effects should be easy to discount by the type of the argument you make. So, to be clear, I am not arguing that the conclusion is necessarily wrong (I think this would require stronger arguments), but rather that the argumentation strategy here does not work.

As far as I understand it your argument for "indirect effects cannot be very large" is something like:

1. Deaths from heat are very overstated and not that significant and they capture the most important direct effect.

2. A claim for strong indirect effects is then implausible because onto that direct effect you need to sequence a bunch of very uncertain additional effects with uncertain signs.

Insofar as is this a correct representation of your argument -- please let me know -- I think it has a couple of problems:

1.

a. Dying from heat stress is a very extreme outcome and people will act in response to climate change much earlier than dying. For example, before people die from heat stress, they might abandon their livelihoods and migrate, maybe in large numbers.

b. More abstractly, the fact that an extreme impact outcome (heat death) is relatively rare is not evidence for low impact in general. Climate change pressures are not like a disease that kills you within days of exposure and otherwise has no consequence.

2.

a. You seem to suggest we are very uncertain about many of the effect signs. I think the basic argument why people concerned about climate change would argue that changes will be negative and that there be compounding risks is because natural and human systems are adapted to specific climate conditions. That doesn't mean they cannot adapt at all, but that does mean that we should expect it is more likely that effects are negative, at least as short-term shocks, than positive for welfare. Insofar as you buy a story where people migrate because of climate impacts, for example, I don't think it is unreasonable to say that increased migration pressures are more likely to increase tensions than to reduce them, etc.

b. I think a lot of the other arguments on the side of "indirect risks are low" you cite are ultimately of the form (i) "indirect effects in other causes are also large" or (ii) "pointing to indirect effects make things inscrutable and unverifiable".  (i) might be true but doesn't matter, I think, for the question of whether warming is net-bad and (ii) is also true, but does nothing by itself on whether those indirect effects are real -- we can live in a world where indirect effects are rhetorically abused and still exist and indeed dominate in certain situations!
 

EA Jobs, Scarcity and Performance

It seems like:

1. For many people, having an EA job is pretty important. 
2. It’s pretty competitive and many people who want EA jobs will not in fact get them. 

There’s been some discussion related to this on the EA Forum, focusing in particular on jobseekers. I’m also interested in exploring this dynamic with people who are working in EA jobs. 

I expect EA job scarcity not only have an impact on EA jobseekers, but also people who are working in EA jobs. 

Given 1 and 2, it seems like for people working in EA jobs it will be pretty important for them to keep their jobs. If the job market is competitive it may not be obvious that they can get another one. (For people who have got one EA job, it will presumably be easier to get another, but maybe not guaranteed). 

For someone who’s in a position of scarcity about their EA job, I can imagine this meaning they focus primarily on performing well/ being seen to perform well. 

This becomes a problem if what counts as performing well and what is actually good to do comes into conflict. Eg. this might involve things like:

• Agreeing with the organisational strategy or one’s manager more than one endorses
• Focusing on ensuring that they have achieved certain outputs independent of whether that output seems good 

In general I expect that under conditions of scarcity people will be less able to do valuable work (and I mean valuable here as ‘actually good’ as opposed to ‘work that is perceived to be valuable). 

(If I’m right about this, then one potential answer to ‘what is it for EA to thrive’, is: EAs aren’t in a position of scarcity). 

Things I’d be interested to ask people who are working at EA jobs to understand whether this is in fact a thing:

• How concerned are you about your perceived performance?
• If your employer/ manager/ funder/ relevant people said something like: ‘We have full confidence in you, your job is guaranteed and we want you to focus on whatever you think is best’ - would that change what you focus on? How much? 

I'm not sure I fully understand this framework, and thus I could easily have missed something here, especially in the section about "Takeover-favoring incentives". However, based on my limited understanding, this framework appears to miss the central argument for why I am personally not as worried about AI takeover risk as most EAs seem to be.

Here's a concise summary of my own argument for being less worried about takeover risk:

1. There is a cost to violently taking over the world, in the sense of acquiring power unlawfully or destructively with the aim of controlling everything in the whole world, relative to the alternative of simply gaining power lawfully and peacefully, even for agents that don't share 'our' values.
   1. For example, as a simple alternative to taking over the world, an AI could advocate for the right to own their own labor and then try to accumulate wealth and power lawfully by selling their services to others, which would earn them the ability to purchase a gargantuan number of paperclips without much restraint.
2. The cost of violent takeover is not obviously smaller than the benefits of violent takeover, given the existence of lawful alternatives to violent takeover. This is for two main reasons:
   1. In order to wage a war to take over the world, you generally need to pay costs fighting the war, and there is a strong motive for everyone else to fight back against you if you try, including other AIs who do not want you to take over the world (and this includes any AIs whose goals would be hindered by a violent takeover, not just those who are "aligned with humans"). Empirically, war is very costly and wasteful, and less efficient than compromise, trade, and diplomacy.
   2. Violently taking over the war is very risky, since the attempt could fail, and you could be totally shut down and penalized heavily if you lose. There are many ways that violent takeover plans could fail: your takeover plans could be exposed too early, you could also be caught trying to coordinate the plan with other AIs and other humans, and you could also just lose the war. Ordinary compromise, trade, and diplomacy generally seem like better strategies for agents that have at least some degree of risk-aversion.
3. There isn't likely to be "one AI" that controls everything, nor will there likely be a strong motive for all the silicon-based minds to coordinate as a unified coalition against the biological-based minds, in the sense of acting as a single agentic AI against the biological people. Thus, future wars of world conquest (if they happen at all) will likely be along different lines than AI vs. human. 
   1. For example, you could imagine a coalition of AIs and humans fighting a war against a separate coalition of AIs and humans, with the aim of establishing control over the world. In this war, the "line" here is not drawn cleanly between humans and AIs, but is instead drawn across a different line. As a result, it's difficult to call this an "AI takeover" scenario, rather than merely a really bad war.
4. Nothing about this argument is intended to argue that AIs will be weaker than humans in aggregate, or individually. I am not claiming that AIs will be bad at coordinating or will be less intelligent than humans. I am also not saying that AIs won't be agentic or that they won't have goals or won't be consequentialists, or that they'll have the same values as humans. I'm also not talking about purely ethical constraints: I am referring to practical constraints and costs on the AI's behavior. The argument is purely about the incentives of violently taking over the world vs. the incentives to peacefully cooperate within a lawful regime, between both humans and other AIs.

5. A big counterargument to my argument seems well-summarized by this hypothetical statement (which is not an actual quote, to be clear): "if you live in a world filled with powerful agents that don't fully share your values, those agents will have a convergent instrumental incentive to violently take over the world from you". However, this argument proves too much. 
   
   We already live in a world where, if this statement was true, we would have observed way more violent takeover attempts than what we've actually observed historically.

   For example, I personally don't fully share values with almost all other humans on Earth (both because of my indexical preferences, and my divergent moral views) and yet the rest of the world has not yet violently disempowered me in any way that I can recognize.

I disagree with the substance, but I don't understand why it gets downvoted.

I’ll respond to your other points in a separate comment later, but for the sake of clarity I want to give a dedicated response to your summary: 

My take is less, "there was some conspiracy where a few organizations did malicious things," and more, "the EA bureaucracy has some significant weaknesses that were highlighted here." 

I very much agree that "the EA bureaucracy has some significant weaknesses that were highlighted here" is the right framing and takeaway. 

My concern (which I believe is shared by other proponents of an independent investigation) is that these weaknesses have not, and are not on track to be, properly diagnosed and fixed. 

I think plenty of EA leaders made mistakes with respect to FTX, but I don’t think there was any malicious conspiracy (except of course for the FTX/Alameda people who were directly involved in the fraud). For the most part, I think people behaved in line with their incentives (which is generally how we should expect people to act). 

The problem is that we don’t have an understanding of how and why those incentives led to mistakes, and we haven’t changed the community’s incentive structures in a way that will prevent those same sorts of mistakes going forward. And I’m concerned that meaningful parts of EA leadership might be inhibiting that learning process in various ways. I'd feel better about the whole situation if there had been some public communications around specific things that have been to improve the efficacy of the EA bureaucracy, including a clear delineation of what things different parts of that bureaucracy are and are not responsible for. 

The argument for Adequate temporal horizon is somewhat hazier

I read you suggesting we'd be explicit about the time horizons AIs would or should consider, but it seems to me we'd want them to think very flexibly about the value of what can be accomplished over different time horizons. I agree it'd be weird if we baked "over the whole lightcone" into all the goals we had, but I think we'd want smarter-than-us AIs to consider whether the coffee they could get us in 5 minutes and one second was potentially way better than the coffee they could get in five minutes, or they could make much more money in 13 months vs a year. 

Less constrained decision-making seems more desirable here, especially if we can just have the AIs report the projected trade offs to us before they move to execution. We don't know our own utility functions that well and it's something we'd want AIs to help with, right? 

Thanks for the update, Toby. I used to defer to you a lot. I no longer do. After investigating the risks myself in decent depth, I consistently arrived to estimates of the risk of human extinction orders of magnitude lower than your existential risk estimates. For example, I understand you assumed in The Precipice an annual existential risk for:

• Nuclear war of around 5*10^-6 (= 0.5*10^-3/100), which is 843 k (= 5*10^-6/(5.93*10^-12)) times mine.
• Volcanoes of around 5*10^-7 (= 0.5*10^-4/100), which is 14.8 M (= 5*10^-7/(3.38*10^-14)) times mine.

In addition, I think the existential risk linked to the above is lower than their extinction risk. The worst nuclear winter of Xia et. al 2022 involves an injection of soot into the stratosphere of 150 Tg, which is just 1 % of the 15 Pg of the Cretaceous–Paleogene extinction event. Moreover, I think this would only be existential with a chance of 0.0513 % (= e^(-10^9/(132*10^6))), assuming:

• An exponential distribution with a mean of 132 M years (= 66*10^6*2) represents the time between i) human extinction in such catastrophe and ii) the evolution of an intelligent sentient species after such a catastrophe. I supposed this on the basis that:
  • An exponential distribution with a mean of 66 M years describes the time between:
    • 2 consecutive such catastrophes.
    • i) and ii) if there are no such catastrophes.
  • Given the above, i) and ii) are equally likely. So the probability of an intelligent sentient species evolving after human extinction in such a catastrophe is 50 % (= 1/2).
  • Consequently, one should expect the time between i) and ii) to be 2 times (= 1/0.50) as long as that if there were no such catastrophes.
• An intelligent sentient species has 1 billion years to evolve before the Earth becomes habitable.

Consider reaching out to Amber Dawn - she's an EA who helps people write posts professionally:

https://forum.effectivealtruism.org/users/amber
 

https://amber-dawn-ace.com/

With open-source models being released and on ramps to downstream innovation lowering, the safety challenges may not be a single threshold but rather an ongoing, iterative cat-and-mouse game.

Just underscores the importance of people in the policy/safety field thinking far ahead

Reading the examples of negative CCIs (e.g. below) makes me think that one of the most informative kinds of future research would be assessing the frequency all events of this kind across all EAs, and assessing whether they differ across Western and non-Western EAs. Based on my own experience, I would expect both Western and non-Western EAs to experience similar events near-constantly, both within EA and without. So it seems it seems like a core crux is whether they occur more frequently or severely in either group / when different groups interact / in some particular setting rather than another.

When they went in the wrong direction, someone yelled the right direction to them in a way that felt infantilising and demeaning.

When they were inside the afterparty space, no one seemed interested in engaging with them, so they left early.

What’s going on with ‘EA Adjacents’? 

There’s a thing where lots of people will say that they are EA Adjacent rather than EA (funny post related to this). In particular, it seems to me that the closer to the core people are, the less inclined they are to identify themselves with EA. What’s going on here? I don’t know, but it’s an interesting trailhead to me. 

Plausibly there are some aspects of EA, the culture, norms, worldview, individuals, organisations etc. that people disagree with or don’t endorse, and so prefer to not identify as EAs. 

I’m unsure how much to treat this as reflective of a substantive issue vs. a quirk, or reflective of things being actually fine. At least in terms of EA being a ‘beacon for thoughtful, sincere, and selfless’, it seems a little bit worrying to me that some of the core members of the community aren’t willing to describe themselves as EA. 

Perhaps a way of getting to the heart of this is asking people something like: Imagine you’re talking to someone who is thoughtful, sincere and selfless. Would you recommend EA to them? Which parts? How strongly? Would you express any reservations? 

Looping back to the question of ‘What is it for EA to thrive?’, one answer is: It’s the kind of community that EA’s would strongly recommend to a thoughtful, sincere and selfless friend. 

(Maybe this is too strong - people will probably reasonably have disagreements about what aspects of EA are good and aren’t, and if everyone is very positive on EA in this way, this plausibly means that there’s not enough disagreement in the community. )

Hi Peter, 

In Famine, Affluence, and Morality, you put forth a position that it should not matter if we help the child who is a neighbor or the child ten thousand miles away.  Is this a strongly held conclusion or a position you want people to continue to debate?

You mentioned you were fortunate enough that Princeton allows you to teach one semester a year and so you have 8 months to spend with your grandkids.  One could argue there are many more children in Trenton, NJ that would benefit from your mentorship.  This is where I disagree with consequentialism.  I believe we should care a lot about the people close to us and collectively we can make sure everyone is cared for.

Thanks for your hard work!

What types of requests ? Like feature updates to the forum, or seeking help from other forum users on projects ?

Executive summary: This document provides a comprehensive overview of current AI policies and regulations across countries worldwide, organized by region and analyzing key factors like regulation status, policy levers, governance types, and involved actors.

Key points:

1. Regulation approaches vary from restrictive to innovative, with many countries still developing AI-specific laws.
2. Common policy levers focus on data, algorithms, and computing power within the "AI triad".
3. Governance types include organizational (e.g. ethics, privacy), case-specific (e.g. healthcare, finance), and model governance (design, training, deployment).
4. Scale of governance ranges from national to regional (e.g. EU, ASEAN) and international efforts.
5. Many countries are aligning with frameworks like the EU AI Act or African Union guidelines.
6. The document notes it may contain errors or outdated information due to the rapidly evolving nature of AI regulation.

This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.

Executive summary: A framework for analyzing AI power-seeking highlights how classic arguments for AI risk rely heavily on the assumption that AIs will be able to easily take over the world, but relaxing this assumption reveals more complex strategic tradeoffs for AIs considering problematic power-seeking.

Key points:

1. Prerequisites for rational AI takeover include agential capabilities, goal-content structure, and takeover-favoring incentives.
2. Classic AI risk arguments assume AIs will meet agential and goal prerequisites, be extremely capable, and have incentives favoring takeover.
3. If AIs cannot easily take over via many paths, the incentives for power-seeking become more complex and uncertain.
4. Analyzing specific AI motivational structures and tradeoffs is important, not just abstract properties like goal-directedness.
5. Strategic dynamics of earlier, weaker AIs matter for improving safety with later, more powerful systems.
6. Framework helps recast and scrutinize key assumptions in classic AI risk arguments.

This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.

What is it for EA to thrive? 

EA Infrastructure Fund's Plan to Focus on Principles-First EA includes a proposal:

The EA Infrastructure Fund will fund and support projects that build and empower the  community of people trying to identify actions that do the greatest good from a scope-sensitive and impartial welfarist view.

And a rationale (there's more detail in the post):

 

• [...] EA is doing something special. 
• [...]  fighting for EA right now could make it meaningfully more likely to thrive long term.
• [...]  we could make EA much better than it currently is - particularly on the “beacon for thoughtful, sincere, and selfless” front. [...]

Here I’m spending some time thinking about this, in particular:

• What does it mean for EA to thrive? 
• What projects could push EA in the direction of thriving? 

(I work at EAIF. These are my personal views/ thoughts. I’m not speaking on behalf of EAIF here)

Most people view farm animals as serving a purpose, whereas animal cruelty is criticized more when more unnecessary. That's why moral progress is made in fashion, poaching, and animal-fighting sports and why veganism should focus more on food waste and traditions like egg tosses and egg decorating: https://www.vox.com/future-perfect/22890292/food-waste-meat-dairy-eggs-milk-animal-welfare Omnivores can respect farm animal sacrifices more, which is a useful mindset shift