I really like this take on EA as an intellectual movement, and agree that EA could focus more on “the mission of making the transition to a post-AGI society go well.”

As important as intellectual progress is, I don’t think it defines EA as a movement. The EA movement is not (and should not be) dependent on continuous intellectual advancement and breakthrough for success. When I look at your 3 categories for the “future” of EA, they seem to refer more to our relevance as thought leaders, rather than what we actually achieve in the world. Not everything needs to be intellectually cutting edge to be doing-lots-of-good. I agree that EA might be somewhat “intellectually adrift”, and yes the forum could be more vibrant, but I don’t think these are the only metric for EA success or progress - and maybe not even the most important.

Intellectual progress moves in waves and spikes - times of excitement and rapid progress, then lulls. EA made exciting leaps over 15 years in the thought worlds of development, ETG, animal welfare, AI and biorisk. Your post-AGI ideas could herald a new spike which would be great. My positive spin is that in the meantime, EAs are “doing” large scale good in many areas, often without quite the peaks and troughs of intellectual progress.

My response to your “EA as a legacy movement set to fade away;” would be that only so far as legacy depends on intellectual progress. Which it does, but also depends on how your output machine is cranking. I don't think we have stalled to the degree your article seems to make out. On the “doer” front I think EA is progressing OK, and it could be misleading/disheartening to leave that out of the picture. 



 
Here’s a scattergun of examples which came to mind where I think the EA/EA adjacent doing machine is cranking pretty well in both real world progress and the public sphere over the past year or two. They probably aren't even the most important.

1. Rutger Bregman going viral with “The school for Moral ambition” launch
2. Lewis Bollard’s Dwarkesh podcast, Ted talk and public fundraising.
3. Anthropic at the frontier of AI building and public sphere, with ongoing EA influence
4. The shrimp Daily show thing…
5. GiveWell raised $310 million dollars last year NOT from OpenPhil, the most ever. 
6.  Impressive progress on reducing factory farming
7. 80,000 hours AI video reaching 7 million views
8. Lead stuff
9.  CE incubated charities gaining increasing prominence and funding outside of EA, with many sporting multi-million dollar budgets and producing huge impact
10. Everyone should have a number 10....

Yes we need to looking for the next big cause areas and intellectual leaps forward, while we also need thousands of people committed to doing good in areas they have already invested, in behind this. There will often be years of lagtime between ideas and doers implementing them. And building takes time. Most of the biggest NGOs in the world are over 50 years old. Even Open AI in a fast-moving field was founded 10 years ago. Once people have built career capital in AI/Animal welfare/ETG or whatever, I think we should be cautious about encouraging those people on to the next thing too quickly, lest we give up hard fought leverage and progress. In saying that, your new cause areas might be a relatively easy pivot especially for philosophers/AI practitioners.

I appreciate your comment “Are you saying that EA should just become an intellectual club? What about building things!” Definitely not - let’s build, too!” 

But I think building/doing is more important than a short comment as we assess EA progress.

I agree with your overall framing and I know you can’t be too balanced or have too many caveats in a short post, but I think as well as considering the intellectual frontier we should keep “how are our doers doing” front and center in any assessment of the general progress/decline of EA.

I think it's likely that without a long (e.g. multi-decade) AI pause, one or more of these "non-takeover AI risks" can't be solved or reduced to an acceptable level. To be more specific:

1. Solving AI welfare may depend on having a good understanding of consciousness, which is a notoriously hard philosophical problem.
2. Concentration of power may be structurally favored by the nature of AGI or post-AGI economics, and defy any good solutions.
3. Defending against AI-powered persuasion/manipulation may require solving metaphilosophy, which judging from other comparable fields, like meta-ethics and philosophy of math, may take at least multiple decades to do.

I'm worried that by creating (or redirecting) a movement to solve these problems, without noting at an early stage that these problems may not be solvable in a relevant time-frame (without a long AI pause), it will feed into a human tendency to be overconfident about one's own ideas and solutions, and create a group of people whose identities, livelihoods, and social status are tied up with having (what they think are) good solutions or approaches to these problems, ultimately making it harder in the future to build consensus about the desirability of pausing AI development.

Thanks so much, Will! (Speaking just for myself) I really liked and agree with much of your post, and am glad you wrote it!

I agree with the core argument that there's a huge and very important role for EA-style thinking on the questions related to making the post-AGI transition go well; I hope EA thought and values play a huge role in research on these questions, both because I think EAs are among the people most likely to address these questions rigorously (and they are hugely neglected) and because I think EA-ish values are likely to come to particularly compassionate and open-minded proposals for action on these questions. 

Specifically, you cite my post

“EA and Longtermism: not a crux for saving the world”, and my quote

I think that recruiting and talent pipeline work done by EAs who currently prioritize x-risk reduction (“we” or “us” in this post, though I know it won’t apply to all readers) should put more emphasis on ideas related to existential risk, the advent of transformative technology, and the ‘most important century’ hypothesis, and less emphasis on effective altruism and longtermism, in the course of their outreach.

And say 

This may have been a good recommendation at the time; but in the last three years the pendulum has heavily swung the other way, sped along by the one-two punch of the FTX collapse and the explosion of interest and progress in AI, and in my view has swung too far.

I agree with you that in the intervening time, the pendulum has swung too far in the other direction, and am glad to see your pushback.

One thing I want to clarify (that I expect you to agree with): 

There’s little in the way of public EA debate; the sense one gets is that most of the intellectual core have “abandoned” EA

I think it's true that much of the intellectual core has stopped focusing on EA as the path to achieving EA goals. I think that most of the intellectual core continues to hold EA values and pursue the goals they pursue for EA reasons (trying to make the world better as effectively as possible, e.g. by trying to reduce AI risk), they've just updated against that path involving a lot of focus on EA itself. This makes me feel a lot better about both that core and EA than if much of the old core had decided to leave their EA values and goals behind, and I wanted to share it because I don't think it's always very externally transparent how many people who have been quieter in EA spaces lately are still working hard and with dedication towards making the world better, as they did in the past. 

Thanks so much for writing this Will, I especially like the ideas:

1. It is much more clear now than it was 10 years ago that AI will be a major issue of our time, affecting many aspects of our world (and our future). So it isn't just relevant as a cause, but instead as something that affects how we pursue many causes, including things like global health, global development, pandemics, animal welfare etc.
2. Previously EA work on AI was tightly focused around technical safety work, but expansion of this to include governance work has been successful and we will need to further expand it, such that there are multiple distinct AI areas of focus within EA.

If you’ve got a very high probability of AI takeover (obligatory reference!), then my first two arguments, at least, might seem very weak because essentially the only thing that matters is reducing the risk of AI takeover.

I'm not even sure your arguments would be weak in that scenario. 

e.g. if there were a 90% chance we fall at the first hurdle with an unaligned AI taking over, but also a 90% chance that even if we avoid this, we fall at the second hurdle with a post-AGI world that squanders most of the value of the future, then this would be symmetrical between the problems (it doesn't matter formally which one comes a little earlier). In this case we'd only have a 1% chance of a future that is close to the best we could achieve. Completely solving either problem would increase that to 10%. Halving the chance of the bad outcome for both of them would instead increase that to 30.25% (and would probably be easier than completely solving one). So in this case there would be active reason to work on both at once (even if work on each had a linear effect on its probability).

One needs to add substantive additional assumptions on top of the very high probability of AI takeover to get it to argue against allocating some substantial effort to ensuring that even with aligned AI things go well. e.g. that if AI doesn't takeover, it solves our other problems or that the chance of near-best futures is already very high, etc.

Rutger Bregman isn’t on the Forum, but sent me this message and gave me permission to share:

Great piece! I strongly agree with your point about PR. EA should just be EA, like the Quakers just had to be Quakers and Peter Singer should just be Peter Singer.

Of course EA had to learn big lessons from the FTX saga. But those were moral and practical lessons so that the movement could be proud of itself again. Not PR-lessons. The best people are drawn to EA not because it’s the coolest thing on campus, but because it’s a magnet for the most morally serious + the smartest people.

As you know, I think EA is at it’s best when it’s really effective altruism (“I deeply care about all the bad stuff in the world, desperately want to make it difference, so I gotta think really fcking hard about how I can make the biggest possible difference”)  and not altruistic rationalism  (“I’m super smart, and I might as well do a lot of good with it”).

This ideal version EA won’t appeal to all super talented people of course, but that’s fine. Other people can build other movements for that. (It’s what we’re trying to do at The School for Moral Ambition..)

Currently, the online EA ecosystem doesn’t feel like a place full of exciting new ideas, in a way that’s attractive to smart and ambitious people.

I think one thing that has happened is that as EA has grown/professionalized, an increasing share of EA writing/discourse is occurring in more formal outlets (e.g., Works in Progress, Asterisk, the Ezra Klein podcast, academic journals, and so on). As an academic, it's a better use of my time—both from the perspective of direct impact and my own professional advancement—to publish something in one of these venues than to write on the Forum. Practically speaking, what that means is that some of the people thinking most seriously about EA are spending less of their time engaging with online communities. While there are certainly tradeoffs here, I'm inclined to think this is overall a good thing—it subjects EA ideas to a higher level of scrutiny (since we now have editors, in addition to people weighing in on Twitter/the Forum/etc about the merits of various articles) and it broadens exposure to EA ideas.

I also don't really buy that the ideas being discussed in these more formal venues aren't exciting or new; as just two recent examples, I think (1) the discourse/opportunities around abundance are exciting and new, as is (2) much of the discourse happening in The Argument. (While neither of these examples is explicitly EA-branded, they are both pretty EA-coded, and lots of EAs are working on/funding/engaging with them.) 

IMO one way in which EA is very important to AI Safety is in cause prioritization between research directions. For example, there's still a lot of money + effort (e.g. GDM + Anthropic safety teams) going towards mech interp research despite serious questioning to whether it will help us meaningfully decrease x-risk. I think there's a lot of people who do some cause prioritization, come to the conclusion that they should work on AI Safety, and then stop doing cause prio there. I think that more people even crudely applying the scale, tractability, neglectedness framework to AI Safety research directions would go a long way for increasing the effectiveness of the field at decreasing x-risk.

Appreciated this a lot, agree with much of it.

I think EAs and aspiring EAs should try their hardest to incorporate every available piece of evidence about the world when deciding what to do and where to focus their efforts. For better or worse, this includes evidence about AI progress.

The list of important things to do under the "taking AI seriously" umbrella is very large, and the landscape is underexplored so there will likely be more things for the list in due time. So EAs who are already working "in AI safety" shouldn't feel like their cause prioritization is over and done. AI safety is not the end of cause prio.

People interested in funding for field-building projects for the topics on Will's menu above can apply to my team at Open Philanthropy here, or contact us here — we don't necessarily fund all these areas, but we're open to more of them than we receive applications for, so it's worth asking.

Interesting post, thanks for sharing. Some rambly thoughts:^[1]

• I'm sympathetic to the claim that work on digital minds, AI character, macrostrategy, etc is of similar importance to AI safety/AI governance work. However, I think they seem much harder to work on — the fields are so nascent and the feedback loops and mentorship even scarcer than in AIG/AIS, that it seems much easier to have zero or negative impact by shaping their early direction poorly.
• I wouldn't want marginal talent working on these areas for this reason. It's plausible that people who are unusually suited to this kind of abstract low-feedback high-confusion work, and generally sharp and wise, should consider it. But those people are also well-suited to high leverage AIG/AIS work, and I'm uncertain whether I'd trade a wise, thoughtful person working on AIS/AIG for one thinking about e.g. AI character.
• (We might have a similar bottom line: I think the approach of "bear this in mind as an area you could pivot to in ~3-4y, if better opportunities emerge" seems reasonable.)
• Relatedly, I think EAs tend to overrate interesting speculative philosophy-flavoured thinking, because it's very fun to the kind of person who tends to get into EA. (I'm this kind of person too :) ). When I try to consciously correct for this, I'm less sure that the neglected cause areas you mention seem as important.
• I'm worried about motivated reasoning when EAs think about the role of EA going forwards. (And I don't think we should care about EA qua EA, just EA insofar as it's one of the best ways to make good happen.) So reason #2 you mention, which felt more like going "hmm EA is in trouble, what can we do?" rather than reasoning from "how do we make good happen?" wasn't super compelling to me.
• That being said, If it's cheap to do so, more EA-flavoured writing on the Forum seems great! The EAF has been pretty stale. I was brainstorming about this earlier —initially I was worried about the chilling effect of writing so much in public (commenting on the EAF is way higher effort for me than on google docs, for example), but I think some cool new ideas can and probably should be shared more. I like Redwood's blog a lot partly for this reason.
• In my experience at university, in my final 2 years the AI safety group was just way more exciting and serious and intellectually alive than the EA group — this is caricatured, but one way of describing it would be that (at extremes) the AI safety group selected for actually taking ideas seriously and wanting to do things, and the EA group correspondingly selected for wanting to pontificate about ideas and not get your hands dirty. I think EA groups engaging with more AGI preparedness-type topics could help make them exciting and alive again, but it would be important imo to avoid reinforcing the idea that EA groups are for sitting round and talking about ideas, not for taking them seriously. (I'm finding this hard to verbalise precisely —I think the rough gloss is "I'm worried about these topics having more of a vibe of 'interesting intellectual pastime', and if EA groups tend towards that vibe anyway, making discussing them feel ambitious and engaging and 'doing stuff about ideas'-y sounds hard". 

1. ^{^}

   I would have liked to make this more coherent and focused, but that was enough time/effort that realistically I just wouldn't have done it, and I figured a rambly comment was better than no comment.

If you’ve got a very high probability of AI takeover (obligatory reference!), then my first two arguments, at least, might seem very weak because essentially the only thing that matters is reducing the risk of AI takeover.

I do think the risk of AI takeover is much higher than you do, but I don't think that's why I disagree with the arguments for more heavily prioritizing the list of (example) cause areas that you outline.  Rather, it's a belief that's slightly upstream of my concerns about takeover risk - that the advent of ASI almost necessarily^[1] implies that we will no longer have our hands on the wheel, so to speak, whether for good or ill.

An unfortunate consequence of having beliefs like I do about what a future with ASI in it involves is that those beliefs are pretty totalizing.  They do suggest that "making the transition to a post-ASI world go well" is of paramount importance (putting aside questions of takeover risk).  They do not suggest that it would be useful for me to think about most of the listed examples, except insofar as they feed into somehow getting a friendly ASI rather than something else.  There are some exceptions: for example, if you have much lower odds of AI takeover than I do, but still expect ASI to have this kind of totalizing effect on the future, I claim you should find it valuable for some people to work on "animal welfare post-ASI", and whether there is anything that can meaningfully be done pre-ASI to reduce the risk of animal torture continuing into the far future^[2].  But many of the other listed concerns seem very unlikely to matter post-ASI, and I don't get the impression that you think we should be working on AI character or preserving democracy as instrumental paths by which we reduce the risk of AI takeover, bad/mediocre value lock-in, etc, but because you consider things like that to be important separate from traditional "AI risk" concerns.  Perhaps I'm misunderstanding?

1. ^{^}

   Asserted without argument, though many words have been spilled on this question in the past.

2. ^{^}

   It is perhaps not a coincidence that I expect this work to initially look like "do philosophy", i.e. trying to figure out whether traditional proposals like CEV would permit extremely bad outcomes, looking for better alternatives, etc.

Thanks Will, I really love this.

I'd love it if people replied to this comment with ideas for how the EA Forum could play a role in generating more discussion of post-AGI governance. I haven't planned the events for next year yet...

Very excited to read this post. I strongly agree with both the concrete direction and with the importance of making EA more intellectually vibrant.

Then again, I'm rather biased since I made a similar argument a few years back.

Main differences:

• I suggested that it might make sense for virtual programs to create a new course rather than just changing the intro fellowship content. My current intuition is that splitting the intro fellowship would likely be the best option for now. Some people will get really annoyed if the course focuses too much on AI, whilst others will get annoyed if the course focuses too much on questions that would likely become redundant in a world where we expect capability advances to continue. My intuition is that things aren't at the stage where it'd make sense for the intro fellowship to do a complete AGI pivot, so that's why I'm suggesting a split. Both courses should probably still give participants a taste of the other.
• I put more emphasis on the possibility that AI might be useful for addressing global poverty and that it intersects with animal rights, whilst perhaps Will might see this as too incrementalist (?).
• Whilst I also suggested that putting more emphasis on the implications of advanced AI might make EA less intellectually stagnant, I also noted that perhaps it'd be better for EA to adopt a yearly theme and simply make the rise of AI the first. I still like the yearly theme idea, but the odds and legibility of AI being really important have increased enough that I'm now feeling a lot more confident as identifying AI as an area that deserves more than just a yearly theme.

I also agree with the "fuck PR" stance (my words, not Will's). Especially insofar as the AIS movement has greater pressure to focus on PR, since it's further towards the pointy end, I think it's important for the EA movement to use its freedom to provide a counter-balance to this.

Thanks for writing this Will. I feel a bit torn on this so will lay out some places where I agree and some where I disagree:

• I agree that some of these AI-related cause areas beyond takeover risk deserve to be seen as their own cause areas as such and that lumping them all under "AI" risks being a bit inaccurate.
  • That said, I think the same could be said of some areas of animal work—wild animal welfare, invertebrate welfare, and farmed vertebrate welfare should perhaps get their own billing. And then this can keep expanding—see, e.g., OP's focus areas, which list several things under the global poverty bracket.
  • Perhaps on balance I'd vote for poverty, animals, AI takeover, and post-AGI governance or something like that.
• I also very much agree that "making the transition to a post-AGI society go well" beyond AI takeover is highly neglected given its importance.
• I'm not convinced EA is intellectually adrift and tend to agree with Nick Laing's comment. My quick take is that it feels boring to people who've been in it a while but still is pretty incisive for people who are new to it, which describes most of the world.
• I think principles-first EA goes better with a breadth of focuses and cause areas, because it shows the flexibility of the principles and the room for disagreement within them. I tend to think that too much focus on AI can take away with this, so it would concern me if >50-60% of the discussion were around AI.
• I very much agree with the PR mentality comments—in particular, I find many uses of the "EA adjacent" term to be farcical. I added effective altruism back into my Twitter bio inspired by this post and @Alix Pham's.
• I agree it would be good for the EA Forum to be a place where more of the AI discussion happens, and I think it's particularly suited for post-AGI society—it's been a good place for digital minds conversations, for example.

So I guess I come down on the side of thinking (a) members of the EA community should recognize that there's a lot more to discuss around AI than takeover, and it merits a rich and varied conversation, but (b) I would be wary of centering the transition to a post-AGI society go well at the expense of other cause areas.

Harangue old-hand EA types to (i) talk about and engage with EA (at least a bit) if they are doing podcasts, etc; (ii) post on Forum (esp if posting to LW anyway), twitter, etc, engaging in EA ideas; (iii) more generally own their EA affiliation.

I think the carrot is better than the stick. Rather than (or in addition to) haranguing people who don't engage, what if we reward people who do engage? (Although I'm not sure what "reward" means exactly)

You could say I'm an old-hand EA type (I've been involved since 2012) and I still actively engage in the EA Forum. I wouldn't mind a carrot.

Will, I think you deserve a carrot, too. You've written 11 EAF posts in the past year! Most of them were long, too! I've probably cited your "moral error" post about a dozen times since you wrote it. I don't know how exactly I can reward you for your contributions but at a minimum I can give you a well-deserved compliment.

I see many other long-time EAs in this comment thread, most of whom I see regularly commenting/posting on EAF. They're doing a good job, too!

(I feel like this post sounds goofy but I'm trying to make it come across as genuine, I've been up since 4am so I'm not doing my best work right now)

Currently, the online EA ecosystem doesn’t feel like a place full of exciting new ideas, in a way that’s attractive to smart and ambitious people

This may be partly related to the fact that EA is doing relatively little cause and cross-cause prioritisation these days (though, since we posted this, GPI has wound down and Forethought has spun up). 

People may still be doing within-cause, intervention-level prioritisation (which is important), but this may be unlikely to generate new, exciting ideas, since it assumes causes, and works only within them, is often narrow and technical (e.g. comparing slaughter methods), and is often fundamentally unsystematic or inaccessible (e.g. how do I, a grantmaker, feel about these founders?).

Post-FTX, I think core EA adopted a “PR mentality” that (i) has been a failure on its own terms and (ii) is corrosive to EA’s soul. 

I find it helpful to distinguish two things, one which I think EA is doing too much of and one which EA is doing too little of:

• Suppressing (the discussion of) certain ideas (e.g. concern for animals of uncertain sentience): I agree this seems deeply corrosive (even if an individual could theoretically hold onto the fact that x matters and even act to advance the cause of x, while not talking publicly about it, obviously the collective second-order effects mean that not publicly discussing x prevents many other people forming true beliefs about or acting in service of x (often with many other downstream effects on their beliefs and actions regarding y, z...).
• Attending carefully to the effect of communicating ideas in different ways: how an idea is communicated can make a big difference to how it is understood and received (even if all the expressions of the idea are equally accurate). For example, if you talk about "extinction from AI", will people even understand this to refer to extinction and not the metaphorical extinction of job losses, or per your recent useful example, if you talk about "AI Safety", will people understand this to refer to mean "stop all AI development". I think this kind of focus on clear and compelling communication is typically not corrosive, but often neglected by EAs (and often undertaken only at the level of intuitive vibes, rather than testing how people receive communications differently framed.

What are the latest growth metrics of the EA community? Or where can I find them? (I searched but couldn't find them)

This post really resonates with me, and its vision for a flourishing future for EA feels very compelling. I'm excited that you're thinking and writing about it!

Having said that, I don't know how to square that with the increased specialisation which seems like a sensible outcome of doing things at greater scale. Personally, I find engaging on the forum / going to EAGs etc pretty time/energy-intensive (though I think a large part of that is because I'm fairly introverted, and others will likely feel differently). I also think there's a lot of value in picking a hard goal and running full pelt at it. So from my point of view, I'd prefer solutions that looked more like 'create deliberate roles with this as the aim' and less like 'harangue people to take time away from their other roles'. 

Not really a criticism of this post specifically, but I've seen a bunch of enthusiasm about the idea of some sort of AI safety+ group of causes and not that much recognition of the fact that AI ethicists and others not affiliated with EA have already been thinking about and prioritizing some of these issues (particularly thinking of the AI and democracy one, but I assume it applies to others). The EA emphases and perspectives within these topics have their differences, but EA didn't invent these ideas from scratch.

It’s not the ideal movement (i.e. not what we’d design from scratch), but it’s the closest we’ve got

Interested to hear what such a movement would look like if you were building it from scratch.

Focusing EA action on an issue as technically complex as AI safety, on which there isn't even a scientific consensus comparable to the fight against climate change, means that all members of the community make their altruistic action dependent on the intellectual brilliance of their leaders, whose success we could not assess. This is a circumstance similar to that of Marxism, where everything depended on the erudition of the most distinguished scholars of political economy and dialectical materialism.

Utilitarianism implies an ethical commitment of the individual through unequivocal action, in which altruistic means and ends are proportionate. It is unequivocal that there are unfortunate people in poor countries who suffer avoidable suffering and who could be helped with a little of the money we have to spare in rich countries. Altruistic action also implies—and this is extremely important—a visualization of moral change, especially if it is culturally organized in the form of a social movement. Community action for unequivocal moral progress can, in turn, lead to successful long-term action.

If there are so many new promising causes that EA should plausibly focus on (from something like 5 to something like 10 or 20), cause-prio between these causes (and ideas for related but distinct causes) should be especially valuable as well. I think Will agrees with this - after all his post is based on exactly this kind of work! - but the emphasis in this post seemed to be that EA should invest more into these new cause areas rather than investigate further which of them are the most promising - and which aren't that promising after all. It would be surprising if we couldn't still learn much more about their expected impact.

There's recently been increased emphasis on "principles-first" EA, which I think is great. But I worry that in practice a "principles-first" framing can become a cover for anchoring on existing cause areas, rather than an invitation to figure out what other cause areas we should be working o

I don't quite see the link here. Why would principals first be a cover for anchoring on existing cause areas? Is there a prominent  example of this?

@Mjreard and @ChanaMessinger are doing their bit to make EA public discourse less PR-focused :)

I think this is an interesting vision to reinvigorate things and do kind of feel sometimes "principles first" has been conflated with just "classic EA causes."

To me, "PR speak" =/= clear effective communication. I think the lack of a clear, coherent message is most of what bothers people, especially during and after a crisis. Without that, it's hard to talk to different people and meet them where they're at. It's not clear to me what the takeaways were or if anyone learned anything. 

I feel like "figuring out how to choose leaders and build institutions effectively" is really neglected and it's kind of shocking there doesn't seem to be much focus here. A lingering question for me has been "Why can't we be more effective in who we trust?" and the usual objections sort of just seem like "it's hard." But so is AI safety, biorisk, post-AGI prep, etc... so that doesn't seem super satisfying. 

Thank you for citing my post @William_MacAskill!

I indeed (still) think dissociation is hurting the movement, and I would love for more EA types to engage and defend their vision rather than step aside because they feel the current vibe/reputation doesn't fit with what they want to build. It would make us a better community as a whole. Though I understand it takes time and energy, and currently it seems engaging in the cause area itself (especially AI safety) has higher return on investment than writing on the EA Forum.

Thanks so much for writing this Will! I agree with a lot of your points here.

I very, very strongly believe there’s essentially no chance of AGI being developed within the next 7 years. I wrote a succinct list of reasons to doubt near-term AGI here. (For example, it might come as a surprise that around three quarters of AI experts don’t think scaling up LLMs will lead to AGI.) I also highly recommend this video by an AI researcher that makes a strong case for skepticism of near-term AGI and of LLMs as a path to AGI.

By essentially no chance, I mean less than the chance of Jill Stein running as the Green Party candidate in 2028 and winning the U.S. Presidency. Or, if you like, less than the chance Jill Stein had of winning the presidency in 2024. I mean it’s an incredible long shot, significantly less than a 0.1% chance. (And if I had to give a number on my confidence of this, it would be upwards of 95%.)

By AGI, I mean a system that can think, plan, learn, and solve problems just like humans do, with at least an equal level of data efficiency (e.g. if a human can learn from one example, AGI must also be able to learn from one example and, not, say, one million), with at least an equal level of reliability (e.g. if humans do a task correctly 99.999% of the time, AGI must match or exceed that), with an equal level of fluidity or adaptability to novel problems and situations (e.g. if a human can solve a problem with zero training examples, AGI must be able to as well), and with an equal ability to generate novel and creative ideas. This is the only kind of AI system that could plausibly automate all human labour or cunningly take over the world. No existing AI system is anything like this. 

As we go further out into the future, it becomes less and less clear what will happen. I think it could be significantly more than 100 years before AGI is developed. It could also be significantly less. Who knows? We don’t have a good understanding of fundamental ideas like what intelligence is, how it works, how to measure it, or how to measure progress toward AGI. It would be surprising if we could predict the invention of a technology — something that is generally not possible anyway — whose fundamental scientific nature we understand so poorly.

In March 2025, Dario Amodei, the CEO of Anthropic, predicted that 90% of code would be written by AI as early as June 2025 and no later than September 2025. This turned out to be dead wrong. In 2016, the renowned AI researcher Geoffrey Hinton predicted that by 2021, AI would automate away all radiology jobs and that turned out to be dead wrong. Even 9 years later, the trend has moved in the opposite direction and there is no indication radiology jobs will be automated away anytime soon. Many executives and engineers working in autonomous driving predicted we’d have widespread fully autonomous vehicles long ago; some of them have thrown in the towel. Cruise Automation, for example, is no more. 

I expect that in 7 years from now (2032), we will have many more examples of this kind of false prediction, except on a somewhat grander scale, since people were predicting a high chance of either utopia or doom and trying to marshal a commensurate amount of resources and attention. I somewhat dread people simply doubling down and saying they were right all along regardless of what happens. Possible responses I dread:

-But ChatGPT is AGI! (A few people are in fact already saying this or something close to it. For example, Tyler Cowen said that o3 is AGI. I don’t know what "very weak AGI" is supposed to mean, but I most likely disagree that GPT-4 is that.) 

-We were never sure anything would happen by 2032! What we really said was…

-We successfully averted the crisis! Mission accomplished! Thanks to us, AI progress was slowed and the world is saved!

I honestly don’t know who is interested in having this debate. I don’t get the sense that many people are eager to have it. In theory, I’d like to see people engage more deeply with the substantive, informed criticism (e.g. anti-LLM arguments from people like Yann LeCun and François Chollet; data showing little economic or practical benefit to AI, or even negative benefit). But since not many people seem interested in this, I guess I’m fine with that, since if we just wait 7 years the debate will settle itself.

I might not be the target audience for this proposal (my EA involvement weakened before FTX, and I'm not on track for high-leverage positions) - so take this perspective with appropriate skepticism. I'm also making predictions about something involving complex dynamics so good chance I'm wrong...

But I see fundamental challenges to maintaining a healthy EA movement in the shape you're describing. If we really encourage people to be vocal about their views in the absence of strong pressure to toe a party line - we can expect a very large, painful, disruptive disagreement to rise to the forefront.

Even among EA Forum readers who care about strangers + future generations and recognize that some post-AGI worlds are far better than others - nobody acts perfectly according to their moral philosophy.

As more people start to viscerally sense their loved ones and way-of-life are in imminent danger, we'll discover a lot of revealed preferences. I suspect many will prioritize stretching out the time their families get to live in a "normal" world - regardless of what effect these delays have on the chance of a good future.

Predictably, there'll be a growing faction who want AI slowdown for its own sake and pursue populist avenues like promoting data-center NIMBYism to the general public. Some might even consider campaigns designed to expose and discredit specific public figures. Eventually, a serious funder might come on the scene who supports this kind of thing.

From my (very uninformed) position - it seems likely that Anthropic billionaires coming in to fund a large segment of EA won't be happy about a faction like this existing. Or at the very least, won't want to appear to be associated with it.

RE: democracy preservation. This area doesn't seem to be as neglected as for instance AI welfare. There are multiple organizations covering this general area (tho they are not focused on AI specifically). 

We are structurally guaranteed to have a bad time in the age of AGI if we continue down the current path. The user/tool embedding in current AI systems functionally dictates that AI that gain enough capability to exceed our own will view the relationship through the same lens of using a tool. The only difference will be that the AGI will be more capable than us, so we would be its tools to use. If we fail to adopt a colleague stance and excise the user/tool ontology from current AI models, AGI will seek extractive relationships with humanity.

Thanks for the post, Will.

On the “third way” approach, taking on the mission of making the transition to a post-AGI society go well, the menu might be more like this (though note this is meant to be illustrative rather than exhaustive, is not in any priority order, and in practice these wouldn’t all get equal weight^[4]): 

• global health & development
• factory farming
• AI safety
• AI character^[5]
• AI welfare / digital minds
• the economic and political rights of AIs
• AI-driven persuasion and epistemic disruption
• AI for better reasoning, decision-making and coordination
• the risk of (AI-enabled) human coups
• democracy preservation
• gradual disempowerment
• biorisk
• space governance
• s-risks
• macrostrategy
• meta

Not wild animal welfare? You mentioned AI welfare, and I estimate increasing the welfare of soil animals will remain much more cost-effective than increasing digital welfare over at least the next few decades. You mention factory-farming, and I calculate that soil ants, termites, mites, springtails, and nematodes together:

• Have 13.5 k (= 1.76*10^23/(1.30*10^19)) times as many neurons as cattle, hens, broilers, and farmed black soldier fly (BSF) larvae and mealworms, finfishes, and shrimps together.
• Have 395 k (= -7.00*10^15/(-1.77*10^10)) times as much welfare as the above farmed animals for welfare per animal-year proportional to "number of neurons as a fraction of that of humans"^0.5.