In that case your strategy is just feeding the labs talent and poisoning the ability of their circles to oppose them.
It seems like your model only has such influence going one way. The lab worker will influence their friends, but not the other way around. I think two-way influence is a more accurate model.
Another option is to ask your friends to monitor you so you don't get ideologically captured, and hold an intervention if it seems appropriate.
I think you, and this community, have no idea how difficult it is to resist value/mission drift in these situations. This is not a friend:friend exchange. Itâs a small community of nonprofits and individuals:the most valuable companies in the world. They arenât just gonna pick up the values of a few researchers by osmosis.
From your other comment it seems like you have already been affected by the labâs influence via the technical research community. The emphasis on technical solutions only benefits them, and it just so happens that to work on the big models you have to work with them. This is not an open exchange where they have been just as influenced by us. Sam and Dario sure want you and the US government to think they are the right safety approach, though.
Thatâs why I only scoped my comment around weak anticapitalism (specifically: placing strong restrictions on wealth accumulation when it leads to market failures), rather than full-scale revolution.
For what it's worth, it is the mainstream view among economists that we should tax or regulate the market in order to address market failures. Yet most economists would not consider themselves "anticapitalist". Using that term when what you mean is something more similar to "well-regulated capitalism" seems quite misleading.
Perhaps the primary distinction between anticapitalists and mainstream economists is that anticapitalists often think we should have very heavy taxation or outright wealth confiscation from rich people, even if this would come at the expense of aggregate utilitarian welfare, because they prioritize other values such as fairness or equality.Since EA tends to be rooted in utilitarian moral theories, I think they should generally distance themselves from this ideology.
Yeah sorry, to emphasise further, Iâm referring to the position where we should place strong restrictions on wealth accumulation when it leads to market failures. The difference between this and the mainstream (in this conception) is that mainstream views take more of a siloed approach to these outcomes, and prefer income taxes or laws to remedy them.
An anticapitalist view contrasts with this by identifying wealth accumulation / concentrated ownership of the means of production as a root cause of these issues and works to restrain it in a more preventative capacity. As you identified, such a view typically advocates for policies like wealth taxes, worker co-determination on boards, and high tax surveillance.
Also loosely on your claim that anticapitalism is incompatible with EA because anticapitalists foreground equality over utilityâI disagree. First, EA is scoped to âaltruismâ, not to âall policy worldwideâ, so a view that aims to maximise altruism also maximises equality under regular conditions. Second, itâs not necessarily the case that there is a tradeoff between equality and global utility, and highly socialist societies such as the Scandis enjoy both higher quality and higher utility than more capitalist countries such as the United States or the UK.
(Iâve read Piketty and donât remember him ever suggesting he would trade one for the other; canât speak to the other authors you cite)
I see no meaningful path to impact on safety working as an AI lab researcher
This is a very strong statement. I'm not following technical alignment research that closely, but my general sense is that exciting work is being done. I just wrote this comment advertising a line of research which strikes me as particularly promising.
I noticed the other day that the people who are particularly grim about AI alignment also don't seem to be engaging much with contemporary technical alignment research. That missing intersection seems suspicious. I'm interested in any counterexamples that come to mind.
My subjective sense is there's a good chance we lose because all the necessary insights to build aligned AI were lying around, they just didn't get sufficiently developed or implemented. This seems especially true for techniques like gradient routing which would need to be baked in to a big, expensive training run.
(I'm also interested in arguments for why unlearning won't work. I've thought about this a fair amount, and it seems to me that sufficiently good unlearning kind of just oneshots AI safety, as elaborated in the comment I linked.)
My subjective sense is there's a good chance we lose because all the necessary insights to build aligned AI were lying around, they just didn't get sufficiently developed or implemented.
For both theoretical and empirical reasons, I would assign a probably as low as 5% to there being alignment insights just laying around that could protect us at the superintelligence capabilities level and donât require us to slow or stop development to implement in time.
I donât see a lot of technical safety people engaging in advocacy, either? Itâs not like they tried advocacy first and then decided on technical safety. Maybe you should question their epistemology.
Caveat: I consider these minor issues, I hope I don't come across as too accusatory.
Interesting, why's that? :)
It seems that the reason for cross-posting was that you personally found it interesting. If you use the EA forum team account, it sounds a bit like an "official" endorsement, and makes the Forum Team less neutral.
Even if you use another account name (eg "selected linkposts") that is run by the Forum Team, I think there should be some explanation how those linkposts are selected, otherwise it seems like arbitrarily privileging some stuff over other stuff.
A "LinkpostBot" account would be good if the cross-posting is automated (e.g. every ACX article who mentions Effective Altruism).
I also personally feel kinda weird getting karma for just linking to someone else's work
I think its fine to gain Karma by virtue of linkposting and being an active forum member, I will not be bothered by it and I think you should not worry about that (although i can understand that it might feel uncomfortable to you). Other people are also allowed to link-post.
Personally when I see a linkpost, I generally assume that the author here is also the original author
I think starting the title with [linkpost] fixes that issue.
Thanks! I basically landed on using my personal account since most people seem to prefer that. I suppose I'll accept the karma if that's what everyone else wants! :P
Honestly I think it's somewhat misleading for me to post with my account because I am posting this in my capacity as part of the Forum Team, even though I'm still an individual making a judgement. It's like when I get a marketing email signed by "Liz" â probably this is a real person writing the email, but it's still more the voice of the company than of an individual, so it feels a bit misleading to say it's from "Liz". On the other hand, I guess all my Forum content has been in my capacity as part of the Forum Team so no reason to change that now! :)
(I also agree with your points about "LinkpostBot" feeling like it should be an automation, and that having a team account for linkposting runs the risk of making those seem privileged.)
I didn't want to read all of @LintzA's post on the "The Game Board has been Flipped" and all 43+ comments, so Icopy/pasted the entire webpage into Claude with the following prompt: "Please give me a summary of the authors argument (dot points, explained simply) and then give me a summary of the kinds of support and push back they got (dot points, explained simply, thematised, giving me a sense of the concentration/popularity of themes in the push back)"
Below is the result (the Forum team might want to consider how posts with large numbers of comments can be read quickly):
Recent developments require a complete rethink of AI safety strategy, particularly:
AI timelines are getting much shorter (leaders like Sam Altman expecting AGI within 3 years)
Trump's likely presidency changes the political landscape for regulation
New technical developments (like Deepseek and inference scaling) mean capabilities are advancing faster than expected
China is closer to US capabilities than previously thought
AI labs are becoming more secretive about their work
Key implications according to the author:
Safety strategies that take several years may be too slow to matter
Need to completely rethink how to communicate AI risk to appeal to conservative audiences
Working inside AI labs may be more important as capabilities become more hidden
The US has less of an advantage over China than previously thought
International agreements may be more important than previously believed
Common Themes in Response (ordered by apparent prominence in comments):
Strong Agreement/Supporting Points:
Many commenters appreciated the comprehensive overview of recent developments
Several agreed with the need to rethink strategies given shorter timelines
Major Points of Disagreement:
Working at AI Labs
Multiple prominent commenters (including Buck and Holly Elmore) pushed back strongly against the author's suggestion that working at AI labs is increasingly important
They argued that lab workers have limited influence on safety and risk being "captured" by lab interests
Some suggested external pressure might be more effective
Strategy and Movement Focus:
Several commenters argued for more emphasis on trying to pause AI development
Some questioned whether shifting focus away from EU/UK engagement was wise
Discussion about whether mass movement building should be prioritized
Technical/Factual Corrections:
Some corrections on specific claims about timeline estimates
Discussion about terminology (e.g., "open source" vs "open weights")
Other Notable Points:
Questions about the US vs China framing
Debate about whether compute advantages still matter given recent developments
Discussion about the value of different political strategies (bipartisan vs partisan approaches)
Overall Tone of Reception: The piece appears to have been well-received as a useful overview of recent developments, but with significant pushback on specific strategic recommendations, particularly around working at AI labs and political strategy.
Nice post, glad somebody wrote this, it has been on my list of things to cover for a while. I think you're right that the risk of a highly-transmissible, antimicrobial resistant pandemic is low but that there is a lot of risk related to secondary bacterial infections within a viral pandemic and that this was an underappreciated driver of Covid-19 mortality.
Do you have any thoughts on the argument I recently gave that gradual and peaceful human disempowerment could be a good thing from an impartial ethical perspective?
Historically, it is common for groups to decline in relative power as a downstream consequence of economic growth and technological progress. As a chief example, the aristocracy declined in influence as a consequence of the industrial revolution. Yet this transformation is generally not considered a bad thing for two reasons. Firstly, since the world is not zero sum, individual aristocrats did not necessarily experience declining well-being despite the relative disempowerment of their class as a whole. Secondly, the world does not merely consist of aristocrats, but rather contains a multitude of moral patients whose agency deserves respect from the perspective of an impartial utilitarian. Specifically, non-aristocrats were largely made better off in light of industrial developments.
Applying this analogy to the present situation with AI, my argument is that even if AIs pursue separate goals from humans and increase in relative power over time, they will not necessarily make individual humans worse off, since the world is not zeros sum. In other words, there is ample opportunity for peaceful and mutually beneficial trade with AIs that do not share our utility functions, which would make both humans and AIs better off. Moreover, AIs themselves may be moral patients whose agency should be given consideration. Just as most of us think it is good that human children are allowed to grow, develop into independent people, and pursue their own goalsâas long as this is done peacefully and lawfullyâagentic AIs should be allowed to do the same. There seems to be a credible possibility of a flourishing AI civilization in the future, even if humans are relatively disempowered, and this outcome could be worth pushing for.
From a preference utilitarian perspective, it is quite unclear that we should prioritize human welfare at all costs. The boundary between biological minds and silicon-based minds seems quite arbitrary from an impartial point of view, making it a fragile foundation for developing policy. There are much more plausible moral boundariesâsuch as the distinction between sentient minds and non-sentient mindsâwhich do not cut cleanly between humans and AIs. Therefore, framing the discussion solely in terms of human disempowerment seems like a mistake to me.
there is ample opportunity for peaceful and mutually beneficial trade with AIs that do not share our utility functions
What would humans have to offer AIs for trade in this scenario, where there are "more competitive machine alternatives to humans in almost all societal functions"?
as long as this is done peacefully and lawfully
What do these words even mean in an ASI context? If humans are relatively disempowered, this would also presumably extend to the use of force and legal contexts.
I mean one huge reason is logistics and uncertainty.
First we must come to the knowledge that yes, children actually are dying, and this death can be prevented with $5000. How do we prove that? How does the average person obtain this information? Well, a charitable foundation says so. Or some famous celebrity claims it to be true. Or some study, which the vast majority of humanity has never read or even heard about, claims it to be true.
Then we need to trust the charitable foundation to faithfully execute the plan to save the child. How do we know the plan will be faithfully executed?
An effective altruist is committed to finding and evaluating these answers. The vast majority of humanity is not. So Effective Altruism has made a bunch of claims, but can't prove these claims in a 5 minute elevator pitch.
In the end then you're just another charity asking for a leap of faith. Some people jump, others don't. If you're not asking for a leap of faith, you're asking for a huge mental investment to verify all the claims made.
Do you have any thoughts on the argument I recently gave that gradual and peaceful human disempowerment could be a good thing from an impartial ethical perspective?
Historically, it is common for groups to decline in relative power as a downstream consequence of economic growth and technological progress. As a chief example, the aristocracy declined in influence as a consequence of the industrial revolution. Yet this transformation is generally not considered a bad thing for two reasons. Firstly, since the world is not zero sum, individual aristocrats did not necessarily experience declining well-being despite the relative disempowerment of their class as a whole. Secondly, the world does not merely consist of aristocrats, but rather contains a multitude of moral patients whose agency deserves respect from the perspective of an impartial utilitarian. Specifically, non-aristocrats were largely made better off in light of industrial developments.
Applying this analogy to the present situation with AI, my argument is that even if AIs pursue separate goals from humans and increase in relative power over time, they will not necessarily make individual humans worse off, since the world is not zeros sum. In other words, there is ample opportunity for peaceful and mutually beneficial trade with AIs that do not share our utility functions, which would make both humans and AIs better off. Moreover, AIs themselves may be moral patients whose agency should be given consideration. Just as most of us think it is good that human children are allowed to grow, develop into independent people, and pursue their own goalsâas long as this is done peacefully and lawfullyâagentic AIs should be allowed to do the same. There seems to be a credible possibility of a flourishing AI civilization in the future, even if humans are relatively disempowered, and this outcome could be worth pushing for.
From a preference utilitarian perspective, it is quite unclear that we should prioritize human welfare at all costs. The boundary between biological minds and silicon-based minds seems quite arbitrary from an impartial point of view, making it a fragile foundation for developing policy. There are much more plausible moral boundariesâsuch as the distinction between sentient minds and non-sentient mindsâwhich do not cut cleanly between humans and AIs. Therefore, framing the discussion solely in terms of human disempowerment seems like a mistake to me.
"anticapitalists often think that we should have very heavy taxation or outright wealth confiscation from rich people, even if this would come at the expense of aggregate utilitarian welfare"
What's the evidence for this? I think even if it is true, it is probably misleading, in that most leftists also just reject the claims mainstream economists make about when taxing the rich will reduce aggregate welfare (not that there is one single mainstream economist view on that anyway in all likelihood.) This sounds to me more like an American centre-right caricature of how socialists think, than something socialists themselves would recognize.
What's the evidence for this? I think even if it is true, it is probably misleading, in that most leftists also just reject the claims mainstream economists make about when taxing the rich will reduce aggregate welfare
To support this claim, we can examine the work of analytical anticapitalists such as G. A. Cohen and John E. Roemer. Both of these thinkers have developed their critiques of capitalism from a foundation of egalitarianism rather than from a perspective primarily concerned with maximizing overall social welfare. Their theories focus on issues of fairness, justice, and equality rather than on the utilitarian consequences of different economic systems.
Similarly, widely cited figures such as Thomas Piketty and John Rawls have provided extensive critiques of capitalist systems, and their arguments are overwhelmingly framed in terms of egalitarian concerns. Both have explicitly advocated for significant wealth redistribution, even when doing so might lead to efficiency losses or other negative utilitarian tradeoffs. Their work illustrates a broader trend in which anticapitalist arguments are often motivated more by ethical commitments to equality than by a strict adherence to utilitarian cost-benefit analysis.
Outside of academic discourse, the distinction becomes less clear. This is because most people do not explicitly frame their economic beliefs within formal theoretical frameworks, making it harder to categorize their positions precisely. I also acknowledge your point that many socialists would likely disagree with my characterization by denying the empirical premise that wealth redistribution can reduce aggregate utilitarian welfare. But this isn't very compelling evidence in my view, as it is common for people among all ideologies to simply deny the tradeoffs inherent in their policy proposals.
What I find most compelling here is that, based on my experience, the vast majority of anticapitalists do not ground their advocacy in a framework that prioritizes maximizing utilitarian welfare. While they may often reference utilitarian concerns in passing, it is uncommon for them to fully engage with mainstream economic analyses of the costs of taxation and redistribution. When anticapitalists do acknowledge these economic arguments, they tend to dismiss or downplay them rather than engaging in a substantive, evidence-based debate within that framework. Those who do accept the mainstream economic framework and attempt to argue within it are generally better categorized as liberals or social democrats rather than strong anticapitalists.
Of course, the distinction between a liberal who strongly supports income redistribution and an anticapitalist is not always sharply defined. There is no rigid, universally agreed-upon boundary between these positions, and I acknowledge that some individuals who identify as anticapitalists may not fit neatly into the categorization I have outlined. However, my original point was intended as a general observation rather than as an exhaustive classification of every nuance within these ideological debates.
This seems pretty unlikely to me tbh, people are just less productive in the developing world than the developed world, and its much easier to do stuff--including do good--when you have functioning institutions, surrounded by competent people, connections & support structures, etc etc.
That's not to say sending people to the developed world is bad. Note that you can get lots of the benefits of living in a developed country by simply having the right to live in a developed country, or having your support structure or legal system or credentials based in a developed country.
Of course, its much easier to just allow everyone in a developing country to just move to a developed country, but assuming the hyper rationalist bot exists with an open boarders constraint, it seems incredibly obvious to me that what you say would not happen.
If I can pull this thread...you previously wrote, "Maybe I have more faith in the market here than you do, but I do think that technical & scientific & economic advancement do in fact have a tendency to not only make everywhere better, but permanently so."
In your opinion, is this an argument in favor of prioritizing pushing both EA money and people into communities like SV that are high-impact in terms of technological advancement?
Thatâs why I only scoped my comment around weak anticapitalism (specifically: placing strong restrictions on wealth accumulation when it leads to market failures), rather than full-scale revolution.
For what it's worth, it is the mainstream view among economists that we should tax or regulate the market in order to address market failures. Yet most economists would not consider themselves "anticapitalist". Using that term when what you mean is something more similar to "well-regulated capitalism" seems quite misleading.
Perhaps the primary distinction between anticapitalists and mainstream economists is that anticapitalists often think we should have very heavy taxation or outright wealth confiscation from rich people, even if this would come at the expense of aggregate utilitarian welfare, because they prioritize other values such as fairness or equality.Since EA tends to be rooted in utilitarian moral theories, I think they should generally distance themselves from this ideology.
"anticapitalists often think that we should have very heavy taxation or outright wealth confiscation from rich people, even if this would come at the expense of aggregate utilitarian welfare"
What's the evidence for this? I think even if it is true, it is probably misleading, in that most leftists also just reject the claims mainstream economists make about when taxing the rich will reduce aggregate welfare (not that there is one single mainstream economist view on that anyway in all likelihood.) This sounds to me more like an American centre-right caricature of how socialists think, than something socialists themselves would recognize.
Yes. Thatâs why I only scoped my comment around weak anticapitalism (specifically: placing strong restrictions on wealth accumulation when it leads to market failures), rather than full-scale revolution. Iâm personally probably more reformist and generally pretty pro-market, but anti-accumulation, FWIW.
As I said, I think that instead of siloed advocacy in distinct cause areas, EAs could realise that they have common cause around opposing bad economic incentives. AI safety, farmed animal welfare, and some global health concerns come from the same roots, and there are already large movements well-placed to solve these problems on the political left (ex. labour unions, veganism, environmentalism, internationalist political groups). Indeed, vegan EAs have already allied well with the existing movement to huge success, but this is the exception.
Frankly, I donât see how that leads to bread lines but I am open to a clearer mechanism if you have one?
Thatâs why I only scoped my comment around weak anticapitalism (specifically: placing strong restrictions on wealth accumulation when it leads to market failures), rather than full-scale revolution.
For what it's worth, it is the mainstream view among economists that we should tax or regulate the market in order to address market failures. Yet most economists would not consider themselves "anticapitalist". Using that term when what you mean is something more similar to "well-regulated capitalism" seems quite misleading.
Perhaps the primary distinction between anticapitalists and mainstream economists is that anticapitalists often think we should have very heavy taxation or outright wealth confiscation from rich people, even if this would come at the expense of aggregate utilitarian welfare, because they prioritize other values such as fairness or equality.Since EA tends to be rooted in utilitarian moral theories, I think they should generally distance themselves from this ideology.
Also, how much do sexing strains in SIT facilities reduce the cost of SIT?
Probably can't cut costs by more than half, right? Assuming the sex ratio is 1:1, now you can double the production of flies with the same input, I guess?
<I'm a bit disappointed, if not surprised, with the community response here.>
I can't speak for other voters, but I downvoted due to my judgment that there were multiple critical assumption that were both unsupported / very thinly supported and pretty dubious -- not because any sacred cows were engaged. While I don't think main post authors are obliged to be exhaustive, the following are examples of significant misses in my book:
"animals successfully bred to tolerate their conditions" -- they are bred such that enough of them don't die before we decide to kill then. It's hard to see much relevance to whether they have been bred to not suffer.
The focus on humans "failing even to reproduce at replacement rates" suggests a focus on wealthy countries. Where's the evidence that people in wealthy countries are not reproducing due to too much suffering, or discontent with their alleged domestication? The post doesn't engage with people in those countries being happier than the median; the effects of education, income, and changing social mores on fertility; and other well-known factors.
More fundamentally, the assertion that there are "similar dynamics" at play between humans and farmed animals is particularly unsupported in my view.
It's important to not reflexively defend sacred cows or downvote those who criticize them . . . but one can believe that while also believing this post seriously missed the mark and warrants downvotes.
I agree that the post is not well defended (partly due to brevity & assuming context); and also that some of the claims seem wrong. But I think the things that are valuable in this post are still worth learning from.
(I'm reminded of a Tyler Cowen quote I can't find atm, something like "When I read the typical economics paper, I think "that seems right" and immediately forget about it. When I read a paper by Hanson, I think "What? No way!" and then think about it for the rest of my life". Ben strikes me as the latter kind of writer.)
Similar to the way Big Ag farms chickens for their meat, you could view governments and corporations as farming humans for their productivity. I think this has been true throughout history, but accelerated recently by more financialization/consumerism and software/smartphones. Both are entities that care about a particular kind of output from the animals they manage, with some reasons to care about their welfare but also some reasons to operate in an extractive way. And when these entities can find a substitute (eg plant-based meat, or AI for intellectual labor) the outcomes may not be ideal for for the animals.
I think this is a good point, predictably enough--I touch on it in my comment on C/H/M's original post--but thanks for elaborating on it!
For what it's worth, I would say that historically, it seems to me that the introduction of new goods has significantly mitigated but not overturned the tendency for consumption increases to lower the marginal utility of consumption. So my central guess is (a) that in the event of a growth acceleration (AI-induced or otherwise), the marginal utility of consumption would in fact fall, and more relevantly (b) that most investors anticipating an AI-induced acceleration to their own consumption growth would expect their marginal utility of consumption to fall. So I think this point identifies a weakness in the argument of the paper/post (as originally written; they now caveat it with this point)--a reason why you can't literally infer investors' beliefs about AGI purely from interest rates--but doesn't in isolation refute the point that a low interest rate is evidence that most investors don't anticipate AGI soon.
I skimmed in your article and must say that I am impressed. I think it is important for the EA community to think about what planet and what society we want. I looked at the summary of the IPBES Nexus Assesment and it seems clear to me that our economic system doesn't work in its current state. That 7 trillion in subsidies for the fossil fuel industry, and damages on nature for 10-25 trillion in unaccounted costs is problematic. Also, the fact that there is 35 times more resources going to causes that destroys our planet than supports our nature shows that we need to do something. I think a realistic way might be national income, by Thomas Piketty. It is a global measurement instead of GDP. For example: âIf you take 100 billion euros of oil from oil reserves underground or you take 100 billion euros in fish from the ocean, you have 100 billion euros of GDP, but you have zero euros of national income. And if in addition when you burn oil or gas you create global warming and you reduce the durability of life on earth, then if you put a price on the negative impact of these emissions you should have negative national income instead of positive GDP.â Buckton et al. (2024) give examples of other economic systems that might work and that are more or less capitalistic.
I didn't realize there was a resource out there to make these works more accessible. That's awesome!
I've been meaning to write a post about how Burke's "Reflections on the Revolution in France" should be required reading for anyone who wants to change the world. I wish I had read it before joining an (ultimately doomed) effort to promote human rights and democracy in a country where the average person can't read. In hindsight, investing in literacy would've been a better use of our time.
Executive summary: The author argues that Animal Charity Evaluators currently offers opaque guidance and under-prioritizes more tractable interventionsâparticularly corporate welfare reformsâand needs to refine its evaluation methods, clarify its comparisons, and focus more on its core mission of rigorous effectiveness evaluation.
Key points:
ACEâs evaluation style is seen as opaque, causing confusion for donors and advocates about how different interventions compare.
The author contends that ACE avoids making clear, substantial claims about which programs most effectively help animals, limiting its usefulness to the movement.
By mixing fund-management roles with evaluation, ACE may dilute its focus on identifying top charities with strong track records.
Evidence suggests corporate welfare campaigns for farmed animals have a proven and measurable impact, yet ACE appears to rate them similarly to less tractable interventions.
The author acknowledges uncertainty about ACEâs internal processes but calls for greater clarity, deeper research, and more decisive prioritization.
Recommended actions include developing clearer impact criteria, separating funding from evaluation functions, and more fully embracing a data-driven approach to prioritize interventions.
This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.
I very much appreciate that you are thinking about this, and the writing is great. That said, without trying to address the arguments directly, I worry that the style here is justifying a conclusion you've come to and explores analogies you like rather than exploring the arguments and trying to decide what side to be on, and it fails to embrace scout mindset sufficiently to be helpful.
Thanks for the feedback! You are right that I'm not exploring the arguments and deciding what side to be on - because that was not the purpose of this post. First, it is not a forum post but a linkpost to the introduction for a blog which will explore these ideas in detail in a series. Second, its purpose is to specifically challenge the status quo and get people even to consider that there might be a different approach (than fully obedient deterministic digital systems).
I very much appreciate that you are thinking about this, and the writing is great. That said, without trying to address the arguments directly, I worry that the style here is justifying a conclusion you've come to and explores analogies you like rather than exploring the arguments and trying to decide what side to be on, and it fails to embrace scout mindset sufficiently to be helpful.
I think that replaceability is very high, so the counterfactual impact is minimal. But that said, there is very little possibility in my mind that even helping with RLHF for compliance with their "safety" guidelines is more beneficial for safety than for accelerating capabilities racing, so any impact is negative.
For context: I want the Forum team to be able to do more active crossposting in the future, so it seems reasonable to have a sort of "placeholder" account for when the author of the original piece doesn't have a Forum account. Personally when I see a linkpost, I generally assume that the author here is also the original author (outside of obvious cases like a NYT article link), and it's kinda confusing when that's not the case (I'm more used to this now but it was extra confusing when I was a new user). I also personally feel kinda weird getting karma for just linking to someone else's work, and probably don't want my Forum profile to mostly be links to other articles.
On the other hand, I do want users to feel free to linkpost external articles that they didn't write, especially if they add their own commentary, or even just make some editorial decisions on which parts to quote. (That's why I was fine with crossposting this using my own account, for example.)
Caveat: I consider these minor issues, I hope I don't come across as too accusatory.
Interesting, why's that? :)
It seems that the reason for cross-posting was that you personally found it interesting. If you use the EA forum team account, it sounds a bit like an "official" endorsement, and makes the Forum Team less neutral.
Even if you use another account name (eg "selected linkposts") that is run by the Forum Team, I think there should be some explanation how those linkposts are selected, otherwise it seems like arbitrarily privileging some stuff over other stuff.
A "LinkpostBot" account would be good if the cross-posting is automated (e.g. every ACX article who mentions Effective Altruism).
I also personally feel kinda weird getting karma for just linking to someone else's work
I think its fine to gain Karma by virtue of linkposting and being an active forum member, I will not be bothered by it and I think you should not worry about that (although i can understand that it might feel uncomfortable to you). Other people are also allowed to link-post.
Personally when I see a linkpost, I generally assume that the author here is also the original author
I think starting the title with [linkpost] fixes that issue.
Which interesting EA-related bluesky accounts do you know of?
I'm not using Twitter anymore since it's being used to promote hateful views, but Bluesky is quite a cool online space in my opinion.
I'm making a list of Bluesky accounts of EA-related organisations and key people. If you're active on Bluesky or some of your favourite EA orgs or key people are, please leave a comment with a link to their profile!
From my rationality-married-with-emotion-and-values human brain, I agree with you. Evil indeed.
That said, I can see a dystopian future where Hyper-Rationalist Bot makes all decisions, and decides that "the greatest good for the greatest number" is best served by keeping human capital in the developing world, using the EA logic that capital in the developing world creates more utility than the same capital in the developed world. (In fact, HRB thinks we should send capable people from the developed world to the developing world to accelerate utility growth even more.)
This seems pretty unlikely to me tbh, people are just less productive in the developing world than the developed world, and its much easier to do stuff--including do good--when you have functioning institutions, surrounded by competent people, connections & support structures, etc etc.
That's not to say sending people to the developed world is bad. Note that you can get lots of the benefits of living in a developed country by simply having the right to live in a developed country, or having your support structure or legal system or credentials based in a developed country.
Of course, its much easier to just allow everyone in a developing country to just move to a developed country, but assuming the hyper rationalist bot exists with an open boarders constraint, it seems incredibly obvious to me that what you say would not happen.
Are you based in Costa Rica? You should totally suggest giving a talk about this at the University for Peace (UPAZ) of the UN in Ciudad ColĂłn! Let me know if you'd like an introduction.
I don't know how ethical it is compared to zero, but compared to the most likely counterfactual (some other person, not aligned with EA and less concerned about AI risks getting this job) I think it's better if you do it.
I just want to say thank you for highlighting this Vasco!
I am a big fan of SMA but my largest concern is their quit your job tag line. Firstly, I can only speak for the animal advocacy space but there are a very limited number of high impact roles for people to pivot intoâŚ. secondly, if itâs in the non profit world they are directing them too it will take some time to pivot (so try to not be unemployed first) thirdly, many of these people are in great companies where they could potentially do much more if they were activated or ETG.
Anecdotally Iâve had a handful of people come from them asking for career advise who had quit their job and tbh my first thought is can you get your job backâŚ.
Firstly, I can only speak for the animal advocacy space but there are a very limited number of high impact roles for people to pivot intoâŚ
I think this applies more broadly. Overwhelmingly based on data about global health and development interventions, Benjamin Todd concludes "itâs defensible to say that the best of all interventions in an area are about 10 times more effective than the mean, and perhaps as much as 100 times". If so, and jobs are uniformly distributed across interventions, a person in a random job within an area donating 10 % of their gross salary to the best interventions in the area can have 1 (= 0.1*10) to 10 (= 0.1*100) times as much impact from donations as from their direct work. In reality, there will be more jobs in less cost-effective interventions, as the best interventions only account for a small fraction of the overall funding. Based on Benjamin's numbers, if there are 10 times as many people in jobs as cost-effective as a random one as in the best jobs, a person in a random job within an area donating 10 % of their gross salary to the best interventions in the area would be 10 (= 1*10) to 100 (= 10*10) times as impactful as a person with the same job not donating.
I'm a bit disappointed, if not surprised, with the community response here. I understand veganism is something of a sacred cow (apologies) in these parts, but that's precisely why Ben's post deserves a careful treatment -- it's the arguments you least agree with that you should extend the most charity to. While this post didn't cause me to reconsider my vegetarianism, historically Ben's posts have had an outsized impact on the way I see things, and I'm grateful for his thoughts here.
Ben's response to point 2 was especially interesting:
If factory farming seems like a bad thing, you should do something about the version happening to you first.
And I agree about the significance of human fertility decline. I expect that this comparison, of factory farming to modern human lives, will be a useful metaphor when thinking about how to improve the structures around us.
<I'm a bit disappointed, if not surprised, with the community response here.>
I can't speak for other voters, but I downvoted due to my judgment that there were multiple critical assumption that were both unsupported / very thinly supported and pretty dubious -- not because any sacred cows were engaged. While I don't think main post authors are obliged to be exhaustive, the following are examples of significant misses in my book:
"animals successfully bred to tolerate their conditions" -- they are bred such that enough of them don't die before we decide to kill then. It's hard to see much relevance to whether they have been bred to not suffer.
The focus on humans "failing even to reproduce at replacement rates" suggests a focus on wealthy countries. Where's the evidence that people in wealthy countries are not reproducing due to too much suffering, or discontent with their alleged domestication? The post doesn't engage with people in those countries being happier than the median; the effects of education, income, and changing social mores on fertility; and other well-known factors.
More fundamentally, the assertion that there are "similar dynamics" at play between humans and farmed animals is particularly unsupported in my view.
It's important to not reflexively defend sacred cows or downvote those who criticize them . . . but one can believe that while also believing this post seriously missed the mark and warrants downvotes.
I think this had to do more with GDPR than the AI act, so the late release in the EU might be a one-off case. Once you figure out how to comply with data collection, it should be straightforward to extend to new models, if they want to.
I did not say that this was due to the EU AI Act, agree that GDPR seems more likely. I mentioned it as an example of EU regulation leading to an AI Lab delaying their EU launch / deprioritizing the EU.
This seems like good worm but the headline and opening paragraph aren't supported when you've shown it might be log-normal. Log-normal and power distributions often have quite different consequences for how important it is to move to very extreme percentiles, and hence this difference can matter for lots of decisions relevant to EA.
What you write there makes sense but it's not free to have people in those positions, as I said. I did a lot of thinking about this when I was working on wild animal welfare. It seems superficially like you could get the right kind of WAW-sympathetic person into agencies like FWS and the EPA and they would be there to, say, nudge the agency in a way no one else cared about to help animals when the time came. I did some interviews and looked into some historical cases and I concluded this is not a good idea.
The risk of being captured by the values and motivations of the org where they spend most of their daily lives before they have the chance to provide that marginal difference is high. Then that person is lost the Safety cause or converted into further problem. I predict that you'll get one successful Safety sleeper agent in, generously, 10 researchers who go to work at a lab. In that case your strategy is just feeding the labs talent and poisoning the ability of their circles to oppose them.
Even if it's harmless, planting an ideological sleeper agent in firms is generally not the best counterfactual use of the person because their influence in a large org is low. Even relatively high-ranking people frequently have almost no discretion about what happens in the end. AI labs probably have more flexibility than US agencies, but I doubt the principle is that different.
Therefore I think trying to influence the values and safety of labs by working there is a bad idea that would not be pulled off.
My sense is that of the many EAs who have taken EtG jobs quite a few have remained fairly value-aligned? I don't have any data on this and am just going on vibes, but I would guess significantly more than 10%. Which is some reason to think the same would be the case for AI companies. Though plausibly the finance company's values are only orthogonal to EA, while the AI company's values (or at least plans) might be more directly opposed.
E.g. Ajeyaâs median estimate is 99% automation of fully-remote jobs in roughly 6-8 years, 5+ years earlier than her 2023 estimate.
This seems more extreme than the linked comment suggests? I can't find anything in the comment justifying "99% automation of fully-remote jobs".
Frankly I think we get ASI and everyone dies before we get anything like 99% automation of current remote jobs, due to bureaucratic inertia and slow adoption. Automation of AI research comes first on the jagged frontier. I don't think Ajeya disagrees?
The comment that Ajeya is replying to is this one from Ryan, who says his timelines are roughly the geometric mean of Ajeya's and Daniel's original views in the post. That is sqrt(4*13) = 7.2 years from the time of the post, so roughly 6 years from now.
As Josh says, the timelines in the original post were answering the question "Median Estimate for when 99% of currently fully remote jobs will be automatable".
So I think it was a fair summary of Ajeya's comment.
I've made a public Forum Events calendar, which you can add to your Gcal. Hopefully, this will give people longer to think about and write for events like Debate Weeks or Theme Weeks. Let me know if you have any issues adding the calendar, or have suggestions for other ways to track Forum events.
Option 1: The EA movement instantly grows to include 1000 new highly-engaged, aligned EAs.
Option 2: X people hear about EA in a relatively high-fidelity way (e.g., through a long-form journalistic article).
...
Responses here also reflected wide variation, spanning several orders of magnitude. The median value was 1 million, implying that gaining 1 highly engaged EA was rated roughly 10,000 times more valuable than introducing one person to EA in a relatively high fidelity way.
Should this be 1,000 times instead? (This also appears at the summary of key results as 10,000)
Anthropic released Claude everywhere but the EU first, and their EU release happened only months later, so to some extend labs are already deprioritizing the EU market. I guess this trend would continue? Not sure.
I think this had to do more with GDPR than the AI act, so the late release in the EU might be a one-off case. Once you figure out how to comply with data collection, it should be straightforward to extend to new models, if they want to.
Are you talking about this post? Looks like those cost-effectiveness estimates were written by Ambitious Impact so I don't know if there are some other estimates written by Vasco.
I got the cost-effectiveness estimates I analysed in that post about global health and development directly from Ambitious Impact (AIM), and the ones about animal welfare adjusting their numbers based on Rethink Priorities' median welfare ranges[1].
I do not have my cost-effectiveness estimates collected in one place. I would be happy to put something together for you, such as a sheet with the name of the intervention, area, source, date of publication, and cost-effectiveness in DALYs averted per $. However, I wonder whether it would be better for you to look into sets of AIM's estimates respecting a given stage of a certain research round. AIM often uses them in weighted factor models to inform which ones to move to the next stage or recommend, so they are supposed to be specially comparable. In contrast, mine often concern different assumptions simply because they span a long period of time. For example, I now guess disabling pain is 10 % as intense as I assumed until October.
I could try to quickly adjust all my estimates such that they all reflect my current assumptions, but I suspect it would not be worth it. I believe AIM's estimates by stage of a particular research round would still be more methodologically aligned, and credible to a wider audience. I am also confident that a set with all my estimates, at least if interpreted at face value, much more closely follow a Pareto, lognormal or loguniform distribution than a normal or uniform distribution. I estimate broiler welfare and cage-free campaigns are 168 and 462 times as cost-effective as GiveWellâs top charities, and that the Shrimp Welfare Project (SWP) has been 64.3 k times as cost-effective as such charities.
AIM used to assume welfare ranges conditional on sentience equal to 1 before moving to estimating the benefits of animal welfare interventions in suffering-adjusted days (SADs) in 2024. I believe the new system still dramatically underestimates the intensity of excruciating pain, and therefore the cost-effectiveness of interventions decreasing it. I estimate the past cost-effectiveness of SWP is 639 DALY/$. For AIMâs pain intensities, and my guess that hurtful pain is as intense as fully healthy life, I get 0.484 DALY/$, which is only 0.0757 % (= 0.484/639) of my estimate. Feel free to ask Vicky Cox, senior animal welfare researcher at AIM, for the sheet with their pain intensities, and the doc with my suggestions for improvement.
If you're doing anticapitalist advocacy for EA reasons, I think you need a really clear understanding of why such advocacy has caused so much misery in the past, and how your advocacy will avoid those traps.
I'd say what's needed is not anticapitalist advocacy, so much as small-scale prototyping of alternative economic systems that have strong theoretical arguments for how they will align incentives better, and scale way past Dunbar's number.
You don't need a full replacement for capitalism to test ideas and see results. For example, central planning often fails due to corruption. A well-designed alternative system will probably need a solution for corruption. And such a solution could be usefully applied to an ordinary capitalist democracy.
I concede that AI companies are behaving in a harmful way, but I doubt that anticapitalist advocacy is a particularly tractable way to address that, at least in the short term.
Yes. Thatâs why I only scoped my comment around weak anticapitalism (specifically: placing strong restrictions on wealth accumulation when it leads to market failures), rather than full-scale revolution. Iâm personally probably more reformist and generally pretty pro-market, but anti-accumulation, FWIW.
As I said, I think that instead of siloed advocacy in distinct cause areas, EAs could realise that they have common cause around opposing bad economic incentives. AI safety, farmed animal welfare, and some global health concerns come from the same roots, and there are already large movements well-placed to solve these problems on the political left (ex. labour unions, veganism, environmentalism, internationalist political groups). Indeed, vegan EAs have already allied well with the existing movement to huge success, but this is the exception.
Frankly, I donât see how that leads to bread lines but I am open to a clearer mechanism if you have one?
Great paper & a strong argument. I would even take it further to argue that most EAs and indeed, longtermists, probably already agree with weak anticapitalism; most EA projects are trying to compensate for externalities or market failures in one form or another, and the increasing turn to policy, rather than altruism, to settle these issues is a good sign.
I think a bigger issue, as youâve confronted on this forum before, is an unwillingness (mostly down to optics / ideological inoculation), to identify these issues as having structural causes in capitalism. This arrests EAs/longtermists from drawing on centuries of knowledge & movement-building, or more to the matter, even representing a coherent common cause that could be addressed through deploying pooled resources (for instance, donating to anticapitalist candidates in US elections). It breaks my heart a bit tbh, but Iâve long accepted it probably wonât happen.
If you're doing anticapitalist advocacy for EA reasons, I think you need a really clear understanding of why such advocacy has caused so much misery in the past, and how your advocacy will avoid those traps.
I'd say what's needed is not anticapitalist advocacy, so much as small-scale prototyping of alternative economic systems that have strong theoretical arguments for how they will align incentives better, and scale way past Dunbar's number.
You don't need a full replacement for capitalism to test ideas and see results. For example, central planning often fails due to corruption. A well-designed alternative system will probably need a solution for corruption. And such a solution could be usefully applied to an ordinary capitalist democracy.
I concede that AI companies are behaving in a harmful way, but I doubt that anticapitalist advocacy is a particularly tractable way to address that, at least in the short term.
How confident are respondents, or people in general, about the results of the valuable skills question? This feels particularly relevant to me at the moment as a community builder.
we now believe that one of the key cost-effectiveness estimates for deworming is flawed, and contains several errors that overstate the cost-effectiveness of deworming by a factor of about 100. This finding has implications not just for deworming, but for cost-effectiveness analysis in general: we are now rethinking how we use published cost-effectiveness estimates for which the full calculations and methods are not public.
The cost-effectiveness estimate in question comes from the Disease Control Priorities in Developing Countries (DCP2), a major report funded by the Gates Foundation. This report provides an estimate of $3.41 per disability-adjusted life-year (DALY) for the cost-effectiveness of soil-transmitted-helminth (STH) treatment, implying that STH treatment is one of the most cost-effective interventions for global health. In investigating this figure, we have corresponded, over a period of months, with six scholars who had been directly or indirectly involved in the production of the estimate. Eventually, we were able to obtain the spreadsheet that was used to generate the $3.41/DALY estimate. That spreadsheet contains five separate errors that, when corrected, shift the estimated cost effectiveness of deworming from $3.41 to $326.43. We came to this conclusion a year after learning that the DCP2âs published cost-effectiveness estimate for schistosomiasis treatment â another kind of deworming â contained a crucial typo: the published figure was $3.36-$6.92 per DALY, but the correct figure is $336-$692 per DALY. (This figure appears, correctly, on page 46 of the DCP2.) ...
I agree with their key takeaways, in particular (emphasis mine)
Weâve previously argued for a limited role for cost-effectiveness estimates; we now think that the appropriate role may be even more limited, at least for opaque estimates (e.g., estimates published without the details necessary for others to independently examine them) like the DCP2âs.
More generally, we see this case as a general argument for expecting transparency, rather than taking recommendations on trust â no matter how pedigreed the people making the recommendations. Note that the DCP2 was published by the Disease Control Priorities Project, a joint enterprise of The World Bank, the National Institutes of Health, the World Health Organization, and the Population Reference Bureau, which was funded primarily by a $3.5 million grant from the Gates Foundation. The DCP2 chapter on helminth infections, which contains the $3.41/DALY estimate, has 18 authors, including many of the worldâs foremost experts on soil-transmitted helminths.
There appears to be a surprising amount of consistency in the shape of the distributions.
The distributions also appear to be closer to lognormal than normal â i.e. they are heavy-tailed, in agreement with Bergerâs findings. However, they may also be some other heavy-tailed distribution (such as a power law), since these are hard to distinguish statistically.
Interventions were rarely negative within health (and the miscellaneous datasets), but often negative within social and education interventions (10â20%) â though not enough to make the mean and median negative. When interventions were negative, they seemed to also be heavy-tailed in negative cost effectiveness.
One way to quantify the interventionsâ spread is to look at the ratio of between the mean of the top 2.5% and the overall mean and median. Roughly, we can say:
The top 2.5% were around 20â200 times more cost effective than the median.
The top 2.5% were around 8â20 times more cost effective than the mean.
Overall, the patterns found by Ord in the DCP2 seem to hold to a surprising degree in the other areas where weâve found data.
Regarding your future work I'd like to see section, maybe Vasco's corpus of cost-effectiveness estimates would be a good starting point. His quantitative modelling spans nearly every category of EA interventions, his models are all methodologically aligned (since it's just him doing them), and they're all transparent too (unlike the DCP estimates).
Regarding your future work I'd like to see section, maybe Vasco's corpus of cost-effectiveness estimates would be a good starting point. His quantitative modelling spans nearly every category of EA interventions, his models are all methodologically aligned (since it's just him doing them), and they're all transparent too (unlike the DCP estimates).
Thanks for the suggestion, Mo! More transparent methodologically aligned estimates:
The Centre for Exploratory Altruism Research (CEARCH) has a sheet with 23 cost-effectiveness estimates across global health and development, global catastrophic risk, and climate change.
They are produced in 3 levels of depth, but they all rely on the same baseline methodology.
Ambitious Impact (AIM) has produced hundreds of cost-effectiveness estimates across global health and development, animal advocacy, and "EA meta".
They are produced in different levels of depth. I collected 44 regarding the interventions recommended for their incubation programs until 2024[1]. However, they have more public estimates concerning interventions which made it to the last stage (in-depth report), but were not recommended, and way more internal estimates. Not only from in-depth reports of interventions which were not recommended[2], but also from interventions which did not make it to the last stage.
You can reach out to Morgan Fairless, AIM's research director, to know more, and ask for access to AIM's internal estimates.
Estimates from Rethink Priorities' cross-cause cost-effectiveness model are also methodologically aligned within each area, but they are not transparent. No information at all is provided about the inputs.
AIM's estimates respecting a given stage of a certain research round[3] will be especially comparable, as AIM often uses them in weighted factor models to inform which ones to move to the next stage or recommend. So I think you had better look into such sets of estimates over one covering all my estimates.
What you write there makes sense but it's not free to have people in those positions, as I said. I did a lot of thinking about this when I was working on wild animal welfare. It seems superficially like you could get the right kind of WAW-sympathetic person into agencies like FWS and the EPA and they would be there to, say, nudge the agency in a way no one else cared about to help animals when the time came. I did some interviews and looked into some historical cases and I concluded this is not a good idea.
The risk of being captured by the values and motivations of the org where they spend most of their daily lives before they have the chance to provide that marginal difference is high. Then that person is lost the Safety cause or converted into further problem. I predict that you'll get one successful Safety sleeper agent in, generously, 10 researchers who go to work at a lab. In that case your strategy is just feeding the labs talent and poisoning the ability of their circles to oppose them.
Even if it's harmless, planting an ideological sleeper agent in firms is generally not the best counterfactual use of the person because their influence in a large org is low. Even relatively high-ranking people frequently have almost no discretion about what happens in the end. AI labs probably have more flexibility than US agencies, but I doubt the principle is that different.
Therefore I think trying to influence the values and safety of labs by working there is a bad idea that would not be pulled off.
In that case your strategy is just feeding the labs talent and poisoning the ability of their circles to oppose them.
It seems like your model only has such influence going one way. The lab worker will influence their friends, but not the other way around. I think two-way influence is a more accurate model.
Another option is to ask your friends to monitor you so you don't get ideologically captured, and hold an intervention if it seems appropriate.
Two years ago short timelines to superintelligence meant decades. That you would structure this bet such that it resolves in just a few years is itself evidence that timelines are getting shorter.
That you would propose the bet at even odds also does not gesture toward your confidence.
Finally, what money means after superintelligence is itself hotly debated, especially for worst-case doomers (the people most likely to expect three year timelines to ASI are also the exact people who don't expect to be alive to collect their winnings).
I think it's possible for a betting structure to prove some kind of point about timelines, but this isn't it.
I am also potentially open to a bet where I transfer money to the person bullish on AI timelines now. I bet Greg Coulbourn 10 k$ this way. However, I would have to trust the person betting with me more than in the case of the bet I linked to above. On this, money being less valuable after superintellgence (including due to supposedly higher risk of death) has the net effect of moving the break-even resolution date forward. As I say in the post I linked to, "We can agree on another resolution date such that the bet is good for you". The resolution date I proposed (end of 2028) was supposed to make the bet just slightly positive for people bullish on AI timelines. However, my views are closer to those of the median expert in 2023, whose median date of full automation was 2073.
Trump is far less likely to take regulatory inspiration from European countries and generally less likely to regulate. On the other-hand perhaps under a 2028 Dem administration we would see significant attention on EU/UK regulations.
The EU/UK are already scaling back the ambitions of their AI regulations out of fear that Trump would retaliate if they put limits on US companies.
InterestingâI've had the opposite take for the EU. The low likelihood of regulation in the US seems like it would make EU regulation more important since that might be all there is. (The second point still stands, but it's still unclear how much that retaliation will happen and what impact it will have.)
It depends on aspects of the Brussels' effect, and I guess it could be that a complete absence of US regulation means companies just pull out of the EU in response to regulation there. Maybe recent technical developments make that more likely. On net, I'm still inclined to think these updates increase the importance of EU stuff.
For the UK, I think I'd agreeâUK work seems to get a lot of its leverage from the relationship with the US.
Anthropic released Claude everywhere but the EU first, and their EU release happened only months later, so to some extend labs are already deprioritizing the EU market. I guess this trend would continue? Not sure.
Great pieceâ great prompt to rethink things and good digests of implications.
If you agree that mass movement building is a priority, check out PauseAI-US.org , or donate here: https://www.zeffy.com/donation-form/donate-to-help-pause-ai
One implication I strongly disagree with is that people should be getting jobs in AI labs. I donât see you connecting that to actual safety impact, and I sincerely doubt working as a researcher gives you any influence on safety at this point (if it ever did). There is a definite cost to working at a lab, which is capture and NDA-walling. Already so many EAs work at Anthropic that it is shielded from scrutiny within EA, and the attachment to âour playerâ Anthropic has made it hard for many EAs to do the obvious thing by supporting PauseAI. Put simply: I see no meaningful path to impact on safety working as an AI lab researcher, and I see serious risks to individual and community effectiveness and mission focus.
I see no meaningful path to impact on safety working as an AI lab researcher
This is a very strong statement. I'm not following technical alignment research that closely, but my general sense is that exciting work is being done. I just wrote this comment advertising a line of research which strikes me as particularly promising.
I noticed the other day that the people who are particularly grim about AI alignment also don't seem to be engaging much with contemporary technical alignment research. That missing intersection seems suspicious. I'm interested in any counterexamples that come to mind.
My subjective sense is there's a good chance we lose because all the necessary insights to build aligned AI were lying around, they just didn't get sufficiently developed or implemented. This seems especially true for techniques like gradient routing which would need to be baked in to a big, expensive training run.
(I'm also interested in arguments for why unlearning won't work. I've thought about this a fair amount, and it seems to me that sufficiently good unlearning kind of just oneshots AI safety, as elaborated in the comment I linked.)
With regard to Deepseek, it seems to me that the success of mixture-of-experts could be considered an update towards methods like gradient routing. If you could localize specific kinds of knowledge to specific experts in a reliable way, you could dynamically toggle off / ablate experts with unnecessary dangerous knowledge. E.g. toggle off experts knowledgeable in human psychology so the AI doesn't manipulate you.
I like this approach because if you get it working well, it's a general tool that could help address a lot of different catastrophe stories in a way that seems pretty robust. E.g. to mitigate a malicious AI from gaining root access to its datacenter, ablate knowledge of the OS it's running on. To mitigate sophisticated cooperation between AIs that are supposed to be monitoring one another, ablate knowledge of game theory. Etc. (The broader point is that unlearning seems very generally useful. But the "Expand-Route-Ablate" style approach from the gradient routing paper strikes me as a particularly promising, and could harmonize well with MoE.)
I think a good research goal would be to try to eventually replicate Deepseek's work, except with highly interpretable experts. The idea is to produce a "high-assurance" model which can be ablated so undesired behaviors, like deception, are virtually impossible to jailbreak out of it (since the weights that perform the behavior are inaccessible). I think the gradient routing paper is a good start. To achieve sufficient safety we'll need new methods that are more robust and easier to deploy, which should probably be prototyped on toy problems first.
it's possible Trump is better placed to negotiate a binding treaty with China (similar to the idea that 'only Nixon could go to China'), even if it's not clear he'll want to do so.
Some related points:
Trump seems like someone who likes winning, but hates losing, and also likes making deals. If Deepseek increases the probability that the US will lose, that makes it more attractive to negotiate an end to the race. This seems true from both a Trump-psychology perspective, and a rational-game-theorist perspective.
Elon Musk seems to have a good relationship with Chinese leadership.
Releasing open-source AI seems like more of a way to prevent someone else from winning than a way to win yourself.
I suppose one possible approach would be to try to get some sort of back-channel dialogue going, to start drafting a treaty which can be invoked if political momentum appears.
Becoming a Good Ancestor As I reflect on life today and the stories of those I admire, I am reminded of the kind of ancestor I hope to become. My grandfather, Rumunyiire, is remembered as a great man, a great ancestor. Though he passed away in the 1970s, the tales of his strength and courage continue to live on.
He was a man of the hills, vast, endless landscapes where he thrived alongside the lions that roamed them. While many feared these wild lands, he embraced them without hesitation. It was on these very hills that he built his home, crafting mud houses and cultivating the land. Yet, he did not seek to tame nature entirely. He left the drinking places and salt licks untouched, allowing the animals to partake in the rhythms of the wild, harmonizing with the landscape he so deeply respected.
To this day, when I walk those lands, I still see the mineral licks, marked by signs that animals continue to use them. He forbade us from ever tampering with them, ensuring that his legacy of coexistence with nature endured long after he was gone.
Conversations with the elders reveal that my grandfather was not only a strong and courageous man but also a kind and honest one. He was known for his generosity, often sharing food from his plantation with the village during times of hunger.
After each harvest, when he returned to the village, he would host grand gatherings, celebrations filled with laughter, food, and a deep sense of community. He ensured that no one left empty-handed. His presence brought warmth and abundance, and people eagerly anticipated his visits, knowing they would be filled with joy and the promise of another unforgettable celebration.
Beyond his generosity, he was a man of principle and justice. Whenever someone in the village committed an immoral act, he was among the first to speak out. He would forbid such behavior, discourage those involved, and ensure that corrective measures were taken within the community. He upheld the values of those around him, not out of self-righteousness, but because he believed in the importance of a strong, moral society.
However, despite the many stories that paint him as a saint, I do not believe he was one. I have walked this earth for 27 years, encountering countless people, yet I have never met anyone who truly fits the description of sainthood. By that logic, my grandfather couldnât have been one either.
What I do believe is that he was a good man, one who did a great deal of good for his children and his community. His kindness, courage, and generosity left behind stories that have endured through generations, so much so that they have overshadowed any faults he may have had. And perhaps that is the true measure of a life well lived, not the absence of flaws, but the presence of an undeniable and lasting impact.
What Does It Mean to Be a Good Ancestor? When I am asked this question, my answer is simple:
To be a good ancestor is to do a great deal of good, not just for your immediate generation, but for the generations that follow. True greatness is measured by the lasting impact of oneâs actions, shaping a future where others can thrive. The highest form of good is one that prevents existential catastrophe, ensuring that those who come after us inherit a better world.
History offers many examples of such ancestors, figures like Nelson Mandela, who fought for the rights and dignity of South Africans, leaving behind a legacy that continues to inspire.
In my time, there are individuals whom we believe have the potential to become great ancestors, people like Bobi Wine, Kizza Besigye, and Justice Kisakye. However, history will be their ultimate judge. Time will test their actions, challenge their legacies, and determine whether their impact endures.
If future generations continue to feel the weight of their influence, if their work continues to shape lives long after they are gone, then they will have truly become good ancestors.
And that, above all else, is the legacy I, too, hope to leave behind.
I agree with you that people seem to somewhat overrate getting jobs in AI companies.
However, I do think there's good work to do inside AI companies. Currently, a lot of the quality-adjusted safety research happens inside AI companies. And see here for my rough argument that it's valuable to have safety-minded people inside AI companies at the point where they develop catastrophically dangerous AI.
What you write there makes sense but it's not free to have people in those positions, as I said. I did a lot of thinking about this when I was working on wild animal welfare. It seems superficially like you could get the right kind of WAW-sympathetic person into agencies like FWS and the EPA and they would be there to, say, nudge the agency in a way no one else cared about to help animals when the time came. I did some interviews and looked into some historical cases and I concluded this is not a good idea.
The risk of being captured by the values and motivations of the org where they spend most of their daily lives before they have the chance to provide that marginal difference is high. Then that person is lost the Safety cause or converted into further problem. I predict that you'll get one successful Safety sleeper agent in, generously, 10 researchers who go to work at a lab. In that case your strategy is just feeding the labs talent and poisoning the ability of their circles to oppose them.
Even if it's harmless, planting an ideological sleeper agent in firms is generally not the best counterfactual use of the person because their influence in a large org is low. Even relatively high-ranking people frequently have almost no discretion about what happens in the end. AI labs probably have more flexibility than US agencies, but I doubt the principle is that different.
Therefore I think trying to influence the values and safety of labs by working there is a bad idea that would not be pulled off.
I'm a bit disappointed, if not surprised, with the community response here. I understand veganism is something of a sacred cow (apologies) in these parts, but that's precisely why Ben's post deserves a careful treatment -- it's the arguments you least agree with that you should extend the most charity to. While this post didn't cause me to reconsider my vegetarianism, historically Ben's posts have had an outsized impact on the way I see things, and I'm grateful for his thoughts here.
Ben's response to point 2 was especially interesting:
If factory farming seems like a bad thing, you should do something about the version happening to you first.
And I agree about the significance of human fertility decline. I expect that this comparison, of factory farming to modern human lives, will be a useful metaphor when thinking about how to improve the structures around us.
I didn't catch this post until I saw this comment, and it prompted a response. I'm not well calibrated on how much upvotes different posts should get,[1] but at least my personal view is that this wasn't a post where I felt disappointment that this didn't make the front page of the EA Forum, and I don't expect this is a post I'd share with e.g., non-vegans who I'd discuss the meat eater problem with.[2]
This isn't something I'd usually comment because I do think the EA Forum should be more welcoming on the margin and I think there are a lot of barriers to people posting. But just providing one data point given your disappointment/surprise.
The ethical vegan must therefore decide whether their objection is to animals dying or to animals living.
One might object to animal suffering, rather than living/dying. So a utilitarian might say factory farming is bad because of the significantly net-negative states that animals endure while alive, while being OK with eating meat from a cow that is raised in a way such that it is living a robustly net positive life, for example.[1]
If you're really worried about reducing the number of animal life years, focus on habitat destruction - it obviously kills wildlife on net, while farming is about increasing lives.
This isn't an obvious comparison to me, there are clear potential downsides of habitat destruction (loss of ecosystem services) that don't apply to reducing factory farming. There are also a lot of uncertainties around impacts of destroying habitats - it is much harder to recreate the ecosystem and its benefits than to re-introduce factory farming if we are wrong in either case. One might also argue that we might have a special obligation to reduce the harms we cause (via factory farming) than attempt habitat destruction, which is reducing suffering that exists ~independently of humans.
...the instrumentalization of animals as things to eat is morally repugnant, so we should make sure it's not perpetuated. This seems to reflect a profound lack of empathy with the perspective of a domesticate that might want to go on existing. Declaring a group's existence repugnant and acting to end it is unambiguously a form of intergroup aggression.
I'm not sure I'm understanding this correctly. Are you saying animals in factory farms have to be able to indicate to you that they don't want to go on existing in order for you to consider taking action on factory farming? What bar do you think is appropriate here?
If factory farming seems like a bad thing, you should do something about the version happening to you first.
If there were 100 billion humans being killed for meat / other products every year and living in the conditions of modern factory farms, I would most definitely prioritise and advocate for that as a priority over factory farming.
The domestication of humans is particularly urgent precisely because, unlike selectively bred farm animals, humans are increasingly expressing their discontent with these conditions, and - more like wild animals in captivity than like proper domesticates - increasingly failing even to reproduce at replacement rates.
Can you say more about what you mean by "the domestication of humans"? It seems like you're trying to draw a parallel between domesticated animals and domesticated humans, or modern humans and wild animals in captivity, but I'm not sure what the parallel you are trying to draw is. Could you make this more explicit?
This suggests our priorities have become oddly inverted - we focus intense moral concern on animals successfully bred to tolerate their conditions, while ignoring similar dynamics affecting creatures capable of articulating their objections...
This seems like a confusing argument. Most vegans I know aren't against factory farming because it affects animal replacement rates. It's also seems unlikely to me that reduced fertility rates in humans is a good proxy/correlate for the amount of suffering that exists (it's possible that the relationship isn't entirely linear, but if anything, historically the opposite is more true - countries have reduced fertility rates as they develop and standards of living improve). It's weird that you use fertility rates as evidence for human suffering but seem to have a extremely high bar for animal suffering! Most of the evidence I'm aware of would strongly point to factory farmed animals in fact not tolerating their conditions well.
...who are moreover the only ones known to have the capacity and willingness to try to solve problems faced by other species.
This is a good argument to work on things that might end humanity or severely diminish it's ability to meaningfully + positively affect the world. Of all the options that might do this, where would you rank reduced fertility rates?
I'm a bit disappointed, if not surprised, with the community response here. I understand veganism is something of a sacred cow (apologies) in these parts, but that's precisely why Ben's post deserves a careful treatment -- it's the arguments you least agree with that you should extend the most charity to. While this post didn't cause me to reconsider my vegetarianism, historically Ben's posts have had an outsized impact on the way I see things, and I'm grateful for his thoughts here.
Ben's response to point 2 was especially interesting:
If factory farming seems like a bad thing, you should do something about the version happening to you first.
And I agree about the significance of human fertility decline. I expect that this comparison, of factory farming to modern human lives, will be a useful metaphor when thinking about how to improve the structures around us.
Are you talking about this post? Looks like those cost-effectiveness estimates were written by Ambitious Impact so I don't know if there are some other estimates written by Vasco.
Great paper & a strong argument. I would even take it further to argue that most EAs and indeed, longtermists, probably already agree with weak anticapitalism; most EA projects are trying to compensate for externalities or market failures in one form or another, and the increasing turn to policy, rather than altruism, to settle these issues is a good sign.
I think a bigger issue, as youâve confronted on this forum before, is an unwillingness (mostly down to optics / ideological inoculation), to identify these issues as having structural causes in capitalism. This arrests EAs/longtermists from drawing on centuries of knowledge & movement-building, or more to the matter, even representing a coherent common cause that could be addressed through deploying pooled resources (for instance, donating to anticapitalist candidates in US elections). It breaks my heart a bit tbh, but Iâve long accepted it probably wonât happen.
The clearest evidence of discontent is the unprecedented fertility decline across developed nations. Humans are increasingly choosing not to reproduce at replacement rate when given modern conditions. This isn't just about discomfort - it suggests our large-scale coordination systems (markets, governments, corporations, media) are actively hostile to the welfare of the governed in a way that factory farming isn't.
Nearly no one wants to torture broiler chickens at massive and increasing scale. If we're doing that, this suggests our interlocking coordination systems are already producing outcomes severely misaligned from individual human values and preferences.
Either we restore human agency enough to avoid relying on distasteful and repugnant systems like the worst aspects of factory farming, or we lose the capacity for meaningful ethical action entirely as our systems drift toward whatever our failing coordination mechanisms were optimizing for, or civilization collapses and takes factory farming with it (along with most humans and domesticated animals). Only the first path offers hope of addressing animal welfare systematically.
The decision calculus would be substantially different if we were near the end rather than the beginning of expansion through the universe, just as one should usually focus more on improving one's own capacities earlier in life and on contributing to others' development later on.
The clearest evidence of discontent is the unprecedented fertility decline across developed nations. Humans are increasingly choosing not to reproduce at replacement rate when given modern conditions.
Why is this clear evidence of discontent? Aren't there many other plausible explanations for the decline in fertility rates, like changes in values and life goals, like ideal family size, prioritization of careers and other interests.
Nearly no one wants to torture broiler chickens at massive and increasing scale. If we're doing that, this suggests our interlocking coordination systems are already producing outcomes severely misaligned from individual human values and preferences.
I agree with the first sentence, but I'm not sure about the second. I think a primary reason is that it's not usually a political priority, because it's not actually important to the average voter. If it's not that important, the outcomes are not severely misaligned from individual human values and preferences.
But it can be made a priority through political advocacy. The outcomes of ballot measures seem like pretty good evidence of what people prefer.
Either we restore human agency
I doubt we have ever really had more human agency in the past than now.
Either we restore human agency enough to avoid relying on distasteful and repugnant systems like the worst aspects of factory farming (...) Only the first path offers hope of addressing animal welfare systematically.
This seems wrong to me. While factory farming is increasing, it's primarily because of increasing populations and incomes, and there are effective targeted ways to systematically reduce and mitigate factory farming that don't require increasing human agency as a whole. Basically what the animal welfare side of EA does.
The decision calculus would be substantially different if we were near the end rather than the beginning of expansion through the universe, just as one should usually focus more on improving one's own capacities earlier in life and on contributing to others' development later on.
Possibly! But I'd like to see actual intervention proposals and estimates of their effects and cost-effectiveness. If the decision calculus is so obvious, you should be able to easily give a lower bound on the cost-effectiveness that drastically beats targeted animal welfare work (and being fair, should consider long-term effects of animal welfare work).
E.g. Ajeyaâs median estimate is 99% automation of fully-remote jobs in roughly 6-8 years, 5+ years earlier than her 2023 estimate.
This seems more extreme than the linked comment suggests? I can't find anything in the comment justifying "99% automation of fully-remote jobs".
Frankly I think we get ASI and everyone dies before we get anything like 99% automation of current remote jobs, due to bureaucratic inertia and slow adoption. Automation of AI research comes first on the jagged frontier. I don't think Ajeya disagrees?
we now believe that one of the key cost-effectiveness estimates for deworming is flawed, and contains several errors that overstate the cost-effectiveness of deworming by a factor of about 100. This finding has implications not just for deworming, but for cost-effectiveness analysis in general: we are now rethinking how we use published cost-effectiveness estimates for which the full calculations and methods are not public.
The cost-effectiveness estimate in question comes from the Disease Control Priorities in Developing Countries (DCP2), a major report funded by the Gates Foundation. This report provides an estimate of $3.41 per disability-adjusted life-year (DALY) for the cost-effectiveness of soil-transmitted-helminth (STH) treatment, implying that STH treatment is one of the most cost-effective interventions for global health. In investigating this figure, we have corresponded, over a period of months, with six scholars who had been directly or indirectly involved in the production of the estimate. Eventually, we were able to obtain the spreadsheet that was used to generate the $3.41/DALY estimate. That spreadsheet contains five separate errors that, when corrected, shift the estimated cost effectiveness of deworming from $3.41 to $326.43. We came to this conclusion a year after learning that the DCP2âs published cost-effectiveness estimate for schistosomiasis treatment â another kind of deworming â contained a crucial typo: the published figure was $3.36-$6.92 per DALY, but the correct figure is $336-$692 per DALY. (This figure appears, correctly, on page 46 of the DCP2.) ...
I agree with their key takeaways, in particular (emphasis mine)
Weâve previously argued for a limited role for cost-effectiveness estimates; we now think that the appropriate role may be even more limited, at least for opaque estimates (e.g., estimates published without the details necessary for others to independently examine them) like the DCP2âs.
More generally, we see this case as a general argument for expecting transparency, rather than taking recommendations on trust â no matter how pedigreed the people making the recommendations. Note that the DCP2 was published by the Disease Control Priorities Project, a joint enterprise of The World Bank, the National Institutes of Health, the World Health Organization, and the Population Reference Bureau, which was funded primarily by a $3.5 million grant from the Gates Foundation. The DCP2 chapter on helminth infections, which contains the $3.41/DALY estimate, has 18 authors, including many of the worldâs foremost experts on soil-transmitted helminths.
There appears to be a surprising amount of consistency in the shape of the distributions.
The distributions also appear to be closer to lognormal than normal â i.e. they are heavy-tailed, in agreement with Bergerâs findings. However, they may also be some other heavy-tailed distribution (such as a power law), since these are hard to distinguish statistically.
Interventions were rarely negative within health (and the miscellaneous datasets), but often negative within social and education interventions (10â20%) â though not enough to make the mean and median negative. When interventions were negative, they seemed to also be heavy-tailed in negative cost effectiveness.
One way to quantify the interventionsâ spread is to look at the ratio of between the mean of the top 2.5% and the overall mean and median. Roughly, we can say:
The top 2.5% were around 20â200 times more cost effective than the median.
The top 2.5% were around 8â20 times more cost effective than the mean.
Overall, the patterns found by Ord in the DCP2 seem to hold to a surprising degree in the other areas where weâve found data.
Regarding your future work I'd like to see section, maybe Vasco's corpus of cost-effectiveness estimates would be a good starting point. His quantitative modelling spans nearly every category of EA interventions, his models are all methodologically aligned (since it's just him doing them), and they're all transparent too (unlike the DCP estimates).
Are you talking about this post? Looks like those cost-effectiveness estimates were written by Ambitious Impact so I don't know if there are some other estimates written by Vasco.
Just wanted to express that I really appreciate your newsletter as a source of thought leadership. Often I find myself wondering about one aspect of the animal advocacy movement or another, and then a few months later you provide a thoughtful summary of the state of play and next steps.
Great work with this! I particularly enjoyed the photos - it's great to see community members across the globe.
I've heard some really positive things about Cape Town before, and generally am excited for more activity in Africa, so I'm quite happy to hear of events like this.
Two years ago short timelines to superintelligence meant decades. That you would structure this bet such that it resolves in just a few years is itself evidence that timelines are getting shorter.
That you would propose the bet at even odds also does not gesture toward your confidence.
Finally, what money means after superintelligence is itself hotly debated, especially for worst-case doomers (the people most likely to expect three year timelines to ASI are also the exact people who don't expect to be alive to collect their winnings).
I think it's possible for a betting structure to prove some kind of point about timelines, but this isn't it.
If factory farming seems like a bad thing, you should do something about the version happening to you first. The domestication of humans is particularly urgent precisely because, unlike selectively bred farm animals, humans are increasingly expressing their discontent with these conditions
Why do you believe discontent (or its expression) is increasing? On what time scale? And do you expect this trend to continue for long?
Plus, factory farming is also increasing, especially in developing countries and for farmed insects.
Your response to 2 in general seems totally insensitive to the relative numbers involved wrt farmed animals and humans, and our potential impacts on each group. Shouldn't there be a point where you'd prioritize farmed animals?
The clearest evidence of discontent is the unprecedented fertility decline across developed nations. Humans are increasingly choosing not to reproduce at replacement rate when given modern conditions. This isn't just about discomfort - it suggests our large-scale coordination systems (markets, governments, corporations, media) are actively hostile to the welfare of the governed in a way that factory farming isn't.
Nearly no one wants to torture broiler chickens at massive and increasing scale. If we're doing that, this suggests our interlocking coordination systems are already producing outcomes severely misaligned from individual human values and preferences.
Either we restore human agency enough to avoid relying on distasteful and repugnant systems like the worst aspects of factory farming, or we lose the capacity for meaningful ethical action entirely as our systems drift toward whatever our failing coordination mechanisms were optimizing for, or civilization collapses and takes factory farming with it (along with most humans and domesticated animals). Only the first path offers hope of addressing animal welfare systematically.
The decision calculus would be substantially different if we were near the end rather than the beginning of expansion through the universe, just as one should usually focus more on improving one's own capacities earlier in life and on contributing to others' development later on.
Thanks for the comment Jessica! This makes sense. I have a few thoughts about this:
More time for people to answer, and in particular to reflect, sounds like it could have been useful (though I wasn't at the event, so I can't comment on the tradeoffs here).
My impression is that the difficulty of the survey is mostly due to the inherent difficulty of the questions we were asked to elicit judgements about (either/both because the questions were substantively difficult and required a lot of information/reflection- e.g. what is the optimal growth rate for EA- or because they're very conceptually difficult/counterintuitive- e.g. how much value do you assign to x relative to y controlling for the value of x's converting into y's), and less because of the operationalization of the questions themselves (see the confusion about earlier iterations of the questions).
One possible response to this, which was mentioned in feedback, is that it could be valuable to pose these questions to dedicated working groups, who devote extensive amounts of time to deliberating on them. Fwiw, this sounds like a very useful (though very costly) initiative to me. It would also have the downside of limiting input to an even smaller subset of the community: so perhaps ideally one would want to pose these questions to a dedicated group, presenting their findings to the wider MCF audiences, and then ask the MCF audience for their take after hearing the working group's findings. Of course, this would take much more time from everyone, so it wouldn't be valuable.
Another possible response is to just try to elicit much simpler judgements. For example, rather than trying to actually get a quantitative estimate of "how many resources do you think think each cause should get?", we could just ask "Do you think x should get more/less?" I think the devil is in the details here, and it would work better for some questions than others e.g. in some cases, merely knowing whether people think a cause should get more/less would not be action-guiding for decisionmakers, but in other cases it would (we're entirely dependent on what decisionmakers tell us they want to elicit here, since I see our role as designing questions to elicit the judgements we're asked for, not deciding what judgements we should try to elicit).
Very quickly: I feel like it's useful to share that I did this survey and found it very hard, and a lot of other people did too. In particular, it did feel pretty rushed for such difficult questions that we didn't necessarily have a fully informed pre-existing take on. OP does mention this, but I wanted to stress that for people reading this post.
I still think it has a lot of useful information and is directionally very informative. I might get a chance to write up more thoughts here, but I am not sure I will be able to. I mostly wanted to give a quick additional flag :)
I had a similar sense of feeling underprepared and rushed while taking the survey and think my input would have been better with more time and a different setting. At the same time I can see that it could have been hard to get the same group of people to answer without these constraints.
For the monetary value of talent Iâm especially cautious on putting much weight on them as I havenât seen much discussion on such estimates and coming up with a numbers in minutes is hard.
Rather than accepting the numbers at face value, they may be more useful for illustrating directional thinking at a specific moment in time.
"EA getting swamped by normies with high inferential distances"
This seems like completely the wrong focus! We need huge numbers of normies involved to get the political pressure necessary to act on AI x-risk before it's too late. We've already tried the "EA's lobbying behind closed doors" approach, and it has failed (/been co-opted by the big AGI companies).
I do think there's concern with a popular movement that the movement will move in a direction you didn't want, but empirically this has already happened for "behind closed doors" lobbying so I don't think a popular movement can do worse.
There's also an argument that a popular movement would be too anti-AI and end up excessively delaying a post-AGI utopia, but I discussed in my post why I don't think that's a sufficiently big concern.
(I agree with you, I'm just anticipating some likely counter-arguments)
Great pieceâ great prompt to rethink things and good digests of implications.
If you agree that mass movement building is a priority, check out PauseAI-US.org , or donate here: https://www.zeffy.com/donation-form/donate-to-help-pause-ai
One implication I strongly disagree with is that people should be getting jobs in AI labs. I donât see you connecting that to actual safety impact, and I sincerely doubt working as a researcher gives you any influence on safety at this point (if it ever did). There is a definite cost to working at a lab, which is capture and NDA-walling. Already so many EAs work at Anthropic that it is shielded from scrutiny within EA, and the attachment to âour playerâ Anthropic has made it hard for many EAs to do the obvious thing by supporting PauseAI. Put simply: I see no meaningful path to impact on safety working as an AI lab researcher, and I see serious risks to individual and community effectiveness and mission focus.
I agree with you that people seem to somewhat overrate getting jobs in AI companies.
However, I do think there's good work to do inside AI companies. Currently, a lot of the quality-adjusted safety research happens inside AI companies. And see here for my rough argument that it's valuable to have safety-minded people inside AI companies at the point where they develop catastrophically dangerous AI.
One axis where Capabilities and Safety people pull apart the most, with high consequences is on "asking for forgiveness instead of permission."
1) Safety people need to get out there and start making stuff without their high prestige ally nodding first 2) Capabilities people need to consider more seriously that they're building something many people simply do not want
I accept the bullet biting response. I think someone who doesn't should say the utility of the observers may outweigh Jones' utility but that you should save Jones for some deontic reason (which is what Scanlon says), or maybe that many small bits of utility spread across people don't sum in a straightforward way, and so can't add up to outweigh Jones' suffering (I think this is incorrect, but that something like it is probably what's actually driving the intuition). I think the infinite disutility response is wrong, but that someone who accepts it should probably adopt some view in infinite ethics according to which two people suffering infinite disutility is worse than one--adopting some such view may be needed to avoid other problems anyway.
The solution you propose is interesting, but I don't think I find it plausible:
1. If Jones' disutility is finite, presumably there is some sufficiently large number of spectators, X, such that their aggregate utility would outweigh his disutility. Why think that, in fact, the physically possible number of observers is lower than X?
2. Suppose Jones isn't suffering the worst torment possible, but merely "extremely painful" shocks, as in Scanlon's example. So the number of observers needed to outweigh his suffering is not X, but the lower number Y. I suppose the intuitive answer is still that you should save him. But why think the physically possible number of observers is below Y?
3. Even if, in fact, the physically possible number of observers is lower than X, presumably the fundamental moral rules should work across possible worlds. And anyway, that seems to be baked into the thought experiment, as there is in fact no Galactic Cup. But even if the physically possible number of observers is in fact lower than X, it could be higher than X in another possible world.
4. Even if the possible number of observers is in fact finite, presumably there are possible worlds with an infinite number of possible observers (the laws of physics are very different, or time is infinite into the future, or there are disembodied ghosts watching, etc.). If we think the solution should work across possible worlds, the fact that there can only be a finite number of observers in our world is then irrelevant.
5. You assume our lightcone is finite "with certainty." I assume this is because of the expected utility concern if there is some chance that it turns out not to be finite. But I think you shouldn't have epistemic certainty that there can only be a finite number of observers.
6. The solution seems to get the intuitive answer for a counterintuitive reason. People find letting Jones get shocked in the transmitter case counterintuitive because they think there is something off about weighing one really bad harm against all these really small benefits, not because of anything having to do with whether there can only be a finite number of observers, and especially not because of anything having that could depend on the specific number of possible observers. Once we grant that the reason for the intuition is off, I'm not sure why we should trust the intuition itself.
*I think your answer to 1-3 may be that there is no set-in-stone number of observers needed to outweigh Jones' suffering: we just pick some arbitrarily large amount and assign it to Jones, such that it's higher than the total utility possessed by however many observers there might happen to be. I am a realist about utility in such a way that we can't do this. But anyway, here is a potential argument against this:
Forget about what number we arbitrarily assign to represent Jones' suffering. Two people each suffering very slightly less than Jones is worse than Jones' suffering. Four people each suffering very slightly worse than them is worse than their suffering. Etc. If we keep going, we will reach some number of people undergoing some trivial amount of suffering which, intuitively, can be outweighed by enough people watching the Galactic Cup--call that number of observers Z. The suffering of those trivially suffering people is worse than the suffering of Jones, by transitivity. So the enjoyment of Z observers outweighs the suffering of Jones, by transitivity. And there is no reason to think the actual number of possible observers is smaller than Z.
Impressive work, but it is not difficult to convince people of the risks of capitalism when it comes to facing longtermism challenges. We have the "social market economy", in which there are supposedly democratic controls on capitalism. But from an imaginative perspective, an alternative to capitalism based on a purely altruistic economy is not inconceivable. An altruistic economy should not be confused with a socialist economy (legislation for the common good), but rather should be related to individualistic cultural conceptions such as the ethics of caring. https://en.wikipedia.org/wiki/Ethics_of_care
I found this finding in the MCF 2024 survey interesting:
The average value to an organization of theirmost preferred over their second most preferred candidate, in a typical hiring round, was estimated to be $50,737 (junior hire) and $455,278 (senior hire).
This survey was hard and only given to a small number of people, so we shouldn't read too much into the specific numbers, but I think it's still a data point against putting significant weight on replacability concerns if you have a job offer for an org you consider impactful.
Survey respondents here (who all work at EA orgs like Open Phil, 80k, CEA, Giving What We Can) are saying that if they make someone a job offer, they would need to receive, in the typical case for junior staff, tens of thousands of dollars to be indifferent about that person taking the job instead of the next best candidate. As someone who's been involved in several hiring rounds, this sounds plausible to me.
If you get a job offer from an org you consider impactful, I suggest not putting significant weight on the idea that the next best candidate could also take the role and have just as much or more impact as you, unless you have a good reason to think you're in an atypical situation. There's often a (very) large gap!
FYI the question posed was:
Imagine a typical hiring round for a [junior/senior] position within your organization. How much financial compensation would you expect to need to receive to make you indifferent about hiring your second most preferred applicant, rather than your most preferred applicant?
(there's a debate to be had about how "EA org receiving X in financial compensation" compares to "value to the world in $ terms" or "value in EA-aligned donations" but I stand by the above bolded claim).
Full disclosure: I work at CEA and helped build the survey, so I'm somewhat incentivised to say this work was interesting and valuable.
Another set of actors that would be incentivized in this would be the survey respondents, to say higher counterfactual values of first vs second choices. Saying otherwise could go against their goals of attracting more of the EA talent pool to their positions. The framing of irreplaceability for their staff also tends to lend to the prestige of their organizations and staff.
With limited applicants, especially in very specialized areas, I think there is definitely a case for a high value of first vs. second choice applicant. But I suspect that this set of survey respondents would be biased in the direction of overestimating the counterfactual impact.
nitpick: you say open source which implies I can read it and rebuild it on my machine. I can't really "read" the weights in this way, I can run it on my machine but I can't compile it without a berjillion chips. "open weight" is the preferred nomenclature, it fits the situation better.
(epistemic status: a pedantry battle, but this ship has sailed as I can see other commenters are saying open source rather than open weight).
The main effect of regulation is to control certain net negative outcomes and hence slowing down negative AGIs. RSPs that require stopping developing at ASL-4 or otherwise are also under the pausing agenda. It might be a question of semantics due to how Pause AI and the Pause AI Letter have become the memetic sink for the term pause AI?
My point is that slowing AI down is often an unwanted side effect, from the regulator perspective. Thus, the main goal is raising the bar for safety practices across developers.
I just updated it (same link) to include a much larger set of papers. The 'other' category now includes papers we may not yet have prioritized, may have deprioritized somewhat (but still found interesting), or may have set aside for other reasons (e.g., slightly outside our scope, not timely for evaluation as part of our model, etc.)
I think this might not be irrationality, but a genuine difference in values.
In particular, I think something like a discount rate disagreement is at the core of a lot of disagreements on AI safety, and to be blunt, you shouldn't expect convergence unless you successfully persuade them of this.
I don't think it's discount rate (esp given short timelines); I think it's more that people haven't really thought about why their p(doom|ASI) is low. But people seem remarkably resistant to actually tackle the cruxes of the object level arguments, or fully extrapolate the implications of what they do agree on. When they do, they invariably come up short.
Update: I've received feedback from the SFF round; we got positive evaluations from two recommenders (so my understanding is the funding allocated to us in the s-process was lower than the speculation grant) and one piece of negative feedback. The negative feedback mentioned that our project might lead to EA getting swamped by normies with high inferential distances, which can have negative consequences; and that because of that risk, "This initiative may be worthy of some support, but unfortunately other orgs in this rather impressive lineup must take priority".
"EA getting swamped by normies with high inferential distances"
This seems like completely the wrong focus! We need huge numbers of normies involved to get the political pressure necessary to act on AI x-risk before it's too late. We've already tried the "EA's lobbying behind closed doors" approach, and it has failed (/been co-opted by the big AGI companies).
Honestly this writeup did update me somewhat in favor of at least a few competent safety-conscious people working at major labs, if only so the safety movement has some access to what's going on inside the labs if/when secrecy grows. The marginal extra researcher going to Anthropic, though? Probably not.
Connect the rest of the dots for me-- how does that researcher's access become community knowledge? How does the community do anything productive with this knowledge? How do you think people working at the labs detracts from other strategies?
Already so many EAs work at Anthropic that it is shielded from scrutiny within EA
What makes you think this? Zach's post is a clear counterexample here (though comments are friendlier to Anthropic) and I've heard of criticism of the RSPs (though I'm not watching closely).
Maybe you think there should be much more criticism?
There should be protests against them (PauseAI US will be protesting them in SF 2/28) and we should all consider them evil for building superintelligence when it is not safe! Dario is now openly calling for recursive self-improvement. They are the villains-- this is not hard. The fact that you would think Zach's post with "maybe" in the title is scrutiny is evidence of the problem.
Comments on 2025-01-31
defun đ¸ @ 2025-01-30T22:43 (+21) in response to defun's Quick takes
Holden Karnofsky has joined Anthropic (LinkedIn profile). I haven't been able to find more information.
Chris Leong @ 2025-01-31T03:50 (+2)
"Member of Technical Staff" - That's surprising. I assumed he was more interested in the policy angle.
Ebenezer Dukakis @ 2025-01-30T10:30 (+4) in response to The Game Board has been Flipped: Now is a good time to rethink what youâre doing
It seems like your model only has such influence going one way. The lab worker will influence their friends, but not the other way around. I think two-way influence is a more accurate model.
Another option is to ask your friends to monitor you so you don't get ideologically captured, and hold an intervention if it seems appropriate.
Holly Elmore â¸ď¸ đ¸ @ 2025-01-31T02:50 (+2)
I think you, and this community, have no idea how difficult it is to resist value/mission drift in these situations. This is not a friend:friend exchange. Itâs a small community of nonprofits and individuals:the most valuable companies in the world. They arenât just gonna pick up the values of a few researchers by osmosis.
From your other comment it seems like you have already been affected by the labâs influence via the technical research community. The emphasis on technical solutions only benefits them, and it just so happens that to work on the big models you have to work with them. This is not an open exchange where they have been just as influenced by us. Sam and Dario sure want you and the US government to think they are the right safety approach, though.
Matthew_Barnett @ 2025-01-30T20:14 (+2) in response to Capitalism and the Very Long Term
For what it's worth, it is the mainstream view among economists that we should tax or regulate the market in order to address market failures. Yet most economists would not consider themselves "anticapitalist". Using that term when what you mean is something more similar to "well-regulated capitalism" seems quite misleading.
Perhaps the primary distinction between anticapitalists and mainstream economists is that anticapitalists often think we should have very heavy taxation or outright wealth confiscation from rich people, even if this would come at the expense of aggregate utilitarian welfare, because they prioritize other values such as fairness or equality. Since EA tends to be rooted in utilitarian moral theories, I think they should generally distance themselves from this ideology.
huw @ 2025-01-31T02:49 (+2)
Yeah sorry, to emphasise further, Iâm referring to the position where we should place strong restrictions on wealth accumulation when it leads to market failures. The difference between this and the mainstream (in this conception) is that mainstream views take more of a siloed approach to these outcomes, and prefer income taxes or laws to remedy them.
An anticapitalist view contrasts with this by identifying wealth accumulation / concentrated ownership of the means of production as a root cause of these issues and works to restrain it in a more preventative capacity. As you identified, such a view typically advocates for policies like wealth taxes, worker co-determination on boards, and high tax surveillance.
Also loosely on your claim that anticapitalism is incompatible with EA because anticapitalists foreground equality over utilityâI disagree. First, EA is scoped to âaltruismâ, not to âall policy worldwideâ, so a view that aims to maximise altruism also maximises equality under regular conditions. Second, itâs not necessarily the case that there is a tradeoff between equality and global utility, and highly socialist societies such as the Scandis enjoy both higher quality and higher utility than more capitalist countries such as the United States or the UK.
(Iâve read Piketty and donât remember him ever suggesting he would trade one for the other; canât speak to the other authors you cite)
Ebenezer Dukakis @ 2025-01-30T08:58 (+16) in response to The Game Board has been Flipped: Now is a good time to rethink what youâre doing
This is a very strong statement. I'm not following technical alignment research that closely, but my general sense is that exciting work is being done. I just wrote this comment advertising a line of research which strikes me as particularly promising.
I noticed the other day that the people who are particularly grim about AI alignment also don't seem to be engaging much with contemporary technical alignment research. That missing intersection seems suspicious. I'm interested in any counterexamples that come to mind.
My subjective sense is there's a good chance we lose because all the necessary insights to build aligned AI were lying around, they just didn't get sufficiently developed or implemented. This seems especially true for techniques like gradient routing which would need to be baked in to a big, expensive training run.
(I'm also interested in arguments for why unlearning won't work. I've thought about this a fair amount, and it seems to me that sufficiently good unlearning kind of just oneshots AI safety, as elaborated in the comment I linked.)
Holly Elmore â¸ď¸ đ¸ @ 2025-01-31T02:36 (+2)
Hereâs our crux:
For both theoretical and empirical reasons, I would assign a probably as low as 5% to there being alignment insights just laying around that could protect us at the superintelligence capabilities level and donât require us to slow or stop development to implement in time.
I donât see a lot of technical safety people engaging in advocacy, either? Itâs not like they tried advocacy first and then decided on technical safety. Maybe you should question their epistemology.
James-Hartree-Law @ 2025-01-31T01:58 (+1) in response to undefined
Hi, the link to fellowships is currently broken. Can you please fix it?
Vasco Grilođ¸ @ 2025-01-30T16:02 (+3) in response to How can we get the world to talk about animalsâ inviolable rights?
Hi David,
You and readers may be interested in Towards A Global Ban On Industrial Animal Agriculture By 2050: Legal Basis, Precedents, And Instruments.
David Michelson @ 2025-01-31T01:36 (+1)
Thank you for sharing this! Looks very relevant to our campaign and I'm looking forward to reading through it more closely.
harfe @ 2025-01-30T16:27 (+4) in response to Climate Change Is Worse Than Factory Farming
Caveat: I consider these minor issues, I hope I don't come across as too accusatory.
It seems that the reason for cross-posting was that you personally found it interesting. If you use the EA forum team account, it sounds a bit like an "official" endorsement, and makes the Forum Team less neutral.
Even if you use another account name (eg "selected linkposts") that is run by the Forum Team, I think there should be some explanation how those linkposts are selected, otherwise it seems like arbitrarily privileging some stuff over other stuff.
A "LinkpostBot" account would be good if the cross-posting is automated (e.g. every ACX article who mentions Effective Altruism).
I think its fine to gain Karma by virtue of linkposting and being an active forum member, I will not be bothered by it and I think you should not worry about that (although i can understand that it might feel uncomfortable to you). Other people are also allowed to link-post.
I think starting the title with [linkpost] fixes that issue.
Sarah Cheng @ 2025-01-31T00:48 (+4)
Thanks! I basically landed on using my personal account since most people seem to prefer that. I suppose I'll accept the karma if that's what everyone else wants! :P
Honestly I think it's somewhat misleading for me to post with my account because I am posting this in my capacity as part of the Forum Team, even though I'm still an individual making a judgement. It's like when I get a marketing email signed by "Liz" â probably this is a real person writing the email, but it's still more the voice of the company than of an individual, so it feels a bit misleading to say it's from "Liz". On the other hand, I guess all my Forum content has been in my capacity as part of the Forum Team so no reason to change that now! :)
(I also agree with your points about "LinkpostBot" feeling like it should be an automation, and that having a team account for linkposting runs the risk of making those seem privileged.)
yanni kyriacos @ 2025-01-31T00:30 (0) in response to Yanni Kyriacos's Quick takes
I didn't want to read all of @LintzA's post on the "The Game Board has been Flipped" and all 43+ comments, so I copy/pasted the entire webpage into Claude with the following prompt: "Please give me a summary of the authors argument (dot points, explained simply) and then give me a summary of the kinds of support and push back they got (dot points, explained simply, thematised, giving me a sense of the concentration/popularity of themes in the push back)"
Below is the result (the Forum team might want to consider how posts with large numbers of comments can be read quickly):
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Main Arguments:
Common Themes in Response (ordered by apparent prominence in comments):
Strong Agreement/Supporting Points:
Major Points of Disagreement:
Technical/Factual Corrections:
Other Notable Points:
Overall Tone of Reception: The piece appears to have been well-received as a useful overview of recent developments, but with significant pushback on specific strategic recommendations, particularly around working at AI labs and political strategy.
Comments on 2025-01-30
defun đ¸ @ 2025-01-30T22:43 (+21) in response to defun's Quick takes
Holden Karnofsky has joined Anthropic (LinkedIn profile). I haven't been able to find more information.
DavidMcK @ 2025-01-30T22:34 (+3) in response to Understanding the AMR Crisis: Implications for Biosecurity and Pandemic Preparedness
Nice post, glad somebody wrote this, it has been on my list of things to cover for a while. I think you're right that the risk of a highly-transmissible, antimicrobial resistant pandemic is low but that there is a lot of risk related to secondary bacterial infections within a viral pandemic and that this was an underappreciated driver of Covid-19 mortality.
Matthew_Barnett @ 2025-01-30T21:48 (+4) in response to Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development
Do you have any thoughts on the argument I recently gave that gradual and peaceful human disempowerment could be a good thing from an impartial ethical perspective?
Historically, it is common for groups to decline in relative power as a downstream consequence of economic growth and technological progress. As a chief example, the aristocracy declined in influence as a consequence of the industrial revolution. Yet this transformation is generally not considered a bad thing for two reasons. Firstly, since the world is not zero sum, individual aristocrats did not necessarily experience declining well-being despite the relative disempowerment of their class as a whole. Secondly, the world does not merely consist of aristocrats, but rather contains a multitude of moral patients whose agency deserves respect from the perspective of an impartial utilitarian. Specifically, non-aristocrats were largely made better off in light of industrial developments.
Applying this analogy to the present situation with AI, my argument is that even if AIs pursue separate goals from humans and increase in relative power over time, they will not necessarily make individual humans worse off, since the world is not zeros sum. In other words, there is ample opportunity for peaceful and mutually beneficial trade with AIs that do not share our utility functions, which would make both humans and AIs better off. Moreover, AIs themselves may be moral patients whose agency should be given consideration. Just as most of us think it is good that human children are allowed to grow, develop into independent people, and pursue their own goalsâas long as this is done peacefully and lawfullyâagentic AIs should be allowed to do the same. There seems to be a credible possibility of a flourishing AI civilization in the future, even if humans are relatively disempowered, and this outcome could be worth pushing for.
From a preference utilitarian perspective, it is quite unclear that we should prioritize human welfare at all costs. The boundary between biological minds and silicon-based minds seems quite arbitrary from an impartial point of view, making it a fragile foundation for developing policy. There are much more plausible moral boundariesâsuch as the distinction between sentient minds and non-sentient mindsâwhich do not cut cleanly between humans and AIs. Therefore, framing the discussion solely in terms of human disempowerment seems like a mistake to me.
Ian Turner @ 2025-01-30T22:23 (+3)
What would humans have to offer AIs for trade in this scenario, where there are "more competitive machine alternatives to humans in almost all societal functions"?
What do these words even mean in an ASI context? If humans are relatively disempowered, this would also presumably extend to the use of force and legal contexts.
Tyler Kolota @ 2025-01-29T02:46 (+11) in response to The Upcoming PEPFAR Cut Will Kill Millions, Many of Them Children
New broader waiver to the freeze for life-saving assistance announced
https://www.reuters.com/world/us/us-issues-new-waiver-humanitarian-aid-amid-freeze-2025-01-29/
Angelina Li @ 2025-01-30T22:12 (+2)
This is such good news!! I'm so relieved, even if there will still be a fight in the near future.
John Huang @ 2025-01-30T22:04 (+4) in response to Facing up to the Price on Life
I mean one huge reason is logistics and uncertainty.
First we must come to the knowledge that yes, children actually are dying, and this death can be prevented with $5000. How do we prove that? How does the average person obtain this information? Well, a charitable foundation says so. Or some famous celebrity claims it to be true. Or some study, which the vast majority of humanity has never read or even heard about, claims it to be true.
Then we need to trust the charitable foundation to faithfully execute the plan to save the child. How do we know the plan will be faithfully executed?
An effective altruist is committed to finding and evaluating these answers. The vast majority of humanity is not. So Effective Altruism has made a bunch of claims, but can't prove these claims in a 5 minute elevator pitch.
In the end then you're just another charity asking for a leap of faith. Some people jump, others don't. If you're not asking for a leap of faith, you're asking for a huge mental investment to verify all the claims made.
Matthew_Barnett @ 2025-01-30T21:48 (+4) in response to Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development
Do you have any thoughts on the argument I recently gave that gradual and peaceful human disempowerment could be a good thing from an impartial ethical perspective?
Historically, it is common for groups to decline in relative power as a downstream consequence of economic growth and technological progress. As a chief example, the aristocracy declined in influence as a consequence of the industrial revolution. Yet this transformation is generally not considered a bad thing for two reasons. Firstly, since the world is not zero sum, individual aristocrats did not necessarily experience declining well-being despite the relative disempowerment of their class as a whole. Secondly, the world does not merely consist of aristocrats, but rather contains a multitude of moral patients whose agency deserves respect from the perspective of an impartial utilitarian. Specifically, non-aristocrats were largely made better off in light of industrial developments.
Applying this analogy to the present situation with AI, my argument is that even if AIs pursue separate goals from humans and increase in relative power over time, they will not necessarily make individual humans worse off, since the world is not zeros sum. In other words, there is ample opportunity for peaceful and mutually beneficial trade with AIs that do not share our utility functions, which would make both humans and AIs better off. Moreover, AIs themselves may be moral patients whose agency should be given consideration. Just as most of us think it is good that human children are allowed to grow, develop into independent people, and pursue their own goalsâas long as this is done peacefully and lawfullyâagentic AIs should be allowed to do the same. There seems to be a credible possibility of a flourishing AI civilization in the future, even if humans are relatively disempowered, and this outcome could be worth pushing for.
From a preference utilitarian perspective, it is quite unclear that we should prioritize human welfare at all costs. The boundary between biological minds and silicon-based minds seems quite arbitrary from an impartial point of view, making it a fragile foundation for developing policy. There are much more plausible moral boundariesâsuch as the distinction between sentient minds and non-sentient mindsâwhich do not cut cleanly between humans and AIs. Therefore, framing the discussion solely in terms of human disempowerment seems like a mistake to me.
Juan Felipe Jaramillo @ 2025-01-30T21:40 (+3) in response to [Cause Exploration Prizes] Jhana meditation
Nice to know there are other meditation enthusiasts in EA. Concentration practice has been a really good finding in my meditation practice.
David Mathersđ¸ @ 2025-01-30T20:18 (+4) in response to Capitalism and the Very Long Term
"anticapitalists often think that we should have very heavy taxation or outright wealth confiscation from rich people, even if this would come at the expense of aggregate utilitarian welfare"
What's the evidence for this? I think even if it is true, it is probably misleading, in that most leftists also just reject the claims mainstream economists make about when taxing the rich will reduce aggregate welfare (not that there is one single mainstream economist view on that anyway in all likelihood.) This sounds to me more like an American centre-right caricature of how socialists think, than something socialists themselves would recognize.
Matthew_Barnett @ 2025-01-30T20:58 (+4)
To support this claim, we can examine the work of analytical anticapitalists such as G. A. Cohen and John E. Roemer. Both of these thinkers have developed their critiques of capitalism from a foundation of egalitarianism rather than from a perspective primarily concerned with maximizing overall social welfare. Their theories focus on issues of fairness, justice, and equality rather than on the utilitarian consequences of different economic systems.
Similarly, widely cited figures such as Thomas Piketty and John Rawls have provided extensive critiques of capitalist systems, and their arguments are overwhelmingly framed in terms of egalitarian concerns. Both have explicitly advocated for significant wealth redistribution, even when doing so might lead to efficiency losses or other negative utilitarian tradeoffs. Their work illustrates a broader trend in which anticapitalist arguments are often motivated more by ethical commitments to equality than by a strict adherence to utilitarian cost-benefit analysis.
Outside of academic discourse, the distinction becomes less clear. This is because most people do not explicitly frame their economic beliefs within formal theoretical frameworks, making it harder to categorize their positions precisely. I also acknowledge your point that many socialists would likely disagree with my characterization by denying the empirical premise that wealth redistribution can reduce aggregate utilitarian welfare. But this isn't very compelling evidence in my view, as it is common for people among all ideologies to simply deny the tradeoffs inherent in their policy proposals.
What I find most compelling here is that, based on my experience, the vast majority of anticapitalists do not ground their advocacy in a framework that prioritizes maximizing utilitarian welfare. While they may often reference utilitarian concerns in passing, it is uncommon for them to fully engage with mainstream economic analyses of the costs of taxation and redistribution. When anticapitalists do acknowledge these economic arguments, they tend to dismiss or downplay them rather than engaging in a substantive, evidence-based debate within that framework. Those who do accept the mainstream economic framework and attempt to argue within it are generally better categorized as liberals or social democrats rather than strong anticapitalists.
Of course, the distinction between a liberal who strongly supports income redistribution and an anticapitalist is not always sharply defined. There is no rigid, universally agreed-upon boundary between these positions, and I acknowledge that some individuals who identify as anticapitalists may not fit neatly into the categorization I have outlined. However, my original point was intended as a general observation rather than as an exhaustive classification of every nuance within these ideological debates.
D0TheMath @ 2025-01-30T16:05 (+3) in response to Should EAs help employees in the developing world move to the West?
This seems pretty unlikely to me tbh, people are just less productive in the developing world than the developed world, and its much easier to do stuff--including do good--when you have functioning institutions, surrounded by competent people, connections & support structures, etc etc.
That's not to say sending people to the developed world is bad. Note that you can get lots of the benefits of living in a developed country by simply having the right to live in a developed country, or having your support structure or legal system or credentials based in a developed country.
Of course, its much easier to just allow everyone in a developing country to just move to a developed country, but assuming the hyper rationalist bot exists with an open boarders constraint, it seems incredibly obvious to me that what you say would not happen.
WillieG @ 2025-01-30T20:36 (+1)
If I can pull this thread...you previously wrote, "Maybe I have more faith in the market here than you do, but I do think that technical & scientific & economic advancement do in fact have a tendency to not only make everywhere better, but permanently so."
In your opinion, is this an argument in favor of prioritizing pushing both EA money and people into communities like SV that are high-impact in terms of technological advancement?
Matthew_Barnett @ 2025-01-30T20:14 (+2) in response to Capitalism and the Very Long Term
For what it's worth, it is the mainstream view among economists that we should tax or regulate the market in order to address market failures. Yet most economists would not consider themselves "anticapitalist". Using that term when what you mean is something more similar to "well-regulated capitalism" seems quite misleading.
Perhaps the primary distinction between anticapitalists and mainstream economists is that anticapitalists often think we should have very heavy taxation or outright wealth confiscation from rich people, even if this would come at the expense of aggregate utilitarian welfare, because they prioritize other values such as fairness or equality. Since EA tends to be rooted in utilitarian moral theories, I think they should generally distance themselves from this ideology.
David Mathersđ¸ @ 2025-01-30T20:18 (+4)
"anticapitalists often think that we should have very heavy taxation or outright wealth confiscation from rich people, even if this would come at the expense of aggregate utilitarian welfare"
What's the evidence for this? I think even if it is true, it is probably misleading, in that most leftists also just reject the claims mainstream economists make about when taxing the rich will reduce aggregate welfare (not that there is one single mainstream economist view on that anyway in all likelihood.) This sounds to me more like an American centre-right caricature of how socialists think, than something socialists themselves would recognize.
huw @ 2025-01-30T11:19 (+6) in response to Capitalism and the Very Long Term
Yes. Thatâs why I only scoped my comment around weak anticapitalism (specifically: placing strong restrictions on wealth accumulation when it leads to market failures), rather than full-scale revolution. Iâm personally probably more reformist and generally pretty pro-market, but anti-accumulation, FWIW.
As I said, I think that instead of siloed advocacy in distinct cause areas, EAs could realise that they have common cause around opposing bad economic incentives. AI safety, farmed animal welfare, and some global health concerns come from the same roots, and there are already large movements well-placed to solve these problems on the political left (ex. labour unions, veganism, environmentalism, internationalist political groups). Indeed, vegan EAs have already allied well with the existing movement to huge success, but this is the exception.
Frankly, I donât see how that leads to bread lines but I am open to a clearer mechanism if you have one?
Matthew_Barnett @ 2025-01-30T20:14 (+2)
For what it's worth, it is the mainstream view among economists that we should tax or regulate the market in order to address market failures. Yet most economists would not consider themselves "anticapitalist". Using that term when what you mean is something more similar to "well-regulated capitalism" seems quite misleading.
Perhaps the primary distinction between anticapitalists and mainstream economists is that anticapitalists often think we should have very heavy taxation or outright wealth confiscation from rich people, even if this would come at the expense of aggregate utilitarian welfare, because they prioritize other values such as fairness or equality. Since EA tends to be rooted in utilitarian moral theories, I think they should generally distance themselves from this ideology.
Ramiro @ 2025-01-30T20:06 (+2) in response to Launching Screwworm-Free Future â Funding and Support Request
Probably can't cut costs by more than half, right? Assuming the sex ratio is 1:1, now you can double the production of flies with the same input, I guess?
Dustin Crummett @ 2025-01-30T19:16 (+9) in response to You should read Hobbes, Locke, Hume, and Mill via EarlyModernTexts.com
The EarlyModernTexts versions are what I read when I had to do my comprehensive history exam in grad school. I recommend them.
Jason @ 2025-01-30T15:24 (+13) in response to Ethical Veganism is Wrong
<I'm a bit disappointed, if not surprised, with the community response here.>
I can't speak for other voters, but I downvoted due to my judgment that there were multiple critical assumption that were both unsupported / very thinly supported and pretty dubious -- not because any sacred cows were engaged. While I don't think main post authors are obliged to be exhaustive, the following are examples of significant misses in my book:
It's important to not reflexively defend sacred cows or downvote those who criticize them . . . but one can believe that while also believing this post seriously missed the mark and warrants downvotes.
Austin @ 2025-01-30T19:14 (+2)
I agree that the post is not well defended (partly due to brevity & assuming context); and also that some of the claims seem wrong. But I think the things that are valuable in this post are still worth learning from.
(I'm reminded of a Tyler Cowen quote I can't find atm, something like "When I read the typical economics paper, I think "that seems right" and immediately forget about it. When I read a paper by Hanson, I think "What? No way!" and then think about it for the rest of my life". Ben strikes me as the latter kind of writer.)
Similar to the way Big Ag farms chickens for their meat, you could view governments and corporations as farming humans for their productivity. I think this has been true throughout history, but accelerated recently by more financialization/consumerism and software/smartphones. Both are entities that care about a particular kind of output from the animals they manage, with some reasons to care about their welfare but also some reasons to operate in an extractive way. And when these entities can find a substitute (eg plant-based meat, or AI for intellectual labor) the outcomes may not be ideal for for the animals.
trammell @ 2025-01-30T19:08 (+4) in response to AGI Cannot Be Predicted From Real Interest Rates
I think this is a good point, predictably enough--I touch on it in my comment on C/H/M's original post--but thanks for elaborating on it!
For what it's worth, I would say that historically, it seems to me that the introduction of new goods has significantly mitigated but not overturned the tendency for consumption increases to lower the marginal utility of consumption. So my central guess is (a) that in the event of a growth acceleration (AI-induced or otherwise), the marginal utility of consumption would in fact fall, and more relevantly (b) that most investors anticipating an AI-induced acceleration to their own consumption growth would expect their marginal utility of consumption to fall. So I think this point identifies a weakness in the argument of the paper/post (as originally written; they now caveat it with this point)--a reason why you can't literally infer investors' beliefs about AGI purely from interest rates--but doesn't in isolation refute the point that a low interest rate is evidence that most investors don't anticipate AGI soon.
Ulf Graf đš @ 2025-01-30T18:43 (+1) in response to Capitalism and the Very Long Term
I skimmed in your article and must say that I am impressed. I think it is important for the EA community to think about what planet and what society we want. I looked at the summary of the IPBES Nexus Assesment and it seems clear to me that our economic system doesn't work in its current state. That 7 trillion in subsidies for the fossil fuel industry, and damages on nature for 10-25 trillion in unaccounted costs is problematic. Also, the fact that there is 35 times more resources going to causes that destroys our planet than supports our nature shows that we need to do something. I think a realistic way might be national income, by Thomas Piketty. It is a global measurement instead of GDP. For example: âIf you take 100 billion euros of oil from oil reserves underground or you take 100 billion euros in fish from the ocean, you have 100 billion euros of GDP, but you have zero euros of national income. And if in addition when you burn oil or gas you create global warming and you reduce the durability of life on earth, then if you put a price on the negative impact of these emissions you should have negative national income instead of positive GDP.â Buckton et al. (2024) give examples of other economic systems that might work and that are more or less capitalistic.
WillieG @ 2025-01-30T18:14 (+3) in response to You should read Hobbes, Locke, Hume, and Mill via EarlyModernTexts.com
I didn't realize there was a resource out there to make these works more accessible. That's awesome!
I've been meaning to write a post about how Burke's "Reflections on the Revolution in France" should be required reading for anyone who wants to change the world. I wish I had read it before joining an (ultimately doomed) effort to promote human rights and democracy in a country where the average person can't read. In hindsight, investing in literacy would've been a better use of our time.
Vasco Grilođ¸ @ 2025-01-29T19:33 (+4) in response to An Evaluation of Animal Charity Evaluators
@SummaryBot , have you considered summarising this post, which was just shared as a classic Forum post on the last EA Forum Digest?
SummaryBot @ 2025-01-30T17:51 (+9)
Good idea! Hereâs a summary of the post:
Executive summary: The author argues that Animal Charity Evaluators currently offers opaque guidance and under-prioritizes more tractable interventionsâparticularly corporate welfare reformsâand needs to refine its evaluation methods, clarify its comparisons, and focus more on its core mission of rigorous effectiveness evaluation.
Key points:
This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.
titotal @ 2025-01-30T17:12 (+5) in response to Retrospective for Inaugural EA SA Summit 2024: Its Impact and What We Learnt
This looks great! I think organizations outside of the typical EA hotspots are very important.
Davidmanheim @ 2025-01-30T16:49 (+3) in response to Tetherware #1: The case for humanlike AI with free will
I very much appreciate that you are thinking about this, and the writing is great. That said, without trying to address the arguments directly, I worry that the style here is justifying a conclusion you've come to and explores analogies you like rather than exploring the arguments and trying to decide what side to be on, and it fails to embrace scout mindset sufficiently to be helpful.
JĂĄchym FibĂr @ 2025-01-30T17:08 (+1)
Thanks for the feedback! You are right that I'm not exploring the arguments and deciding what side to be on - because that was not the purpose of this post. First, it is not a forum post but a linkpost to the introduction for a blog which will explore these ideas in detail in a series. Second, its purpose is to specifically challenge the status quo and get people even to consider that there might be a different approach (than fully obedient deterministic digital systems).
Davidmanheim @ 2025-01-30T16:49 (+3) in response to Tetherware #1: The case for humanlike AI with free will
I very much appreciate that you are thinking about this, and the writing is great. That said, without trying to address the arguments directly, I worry that the style here is justifying a conclusion you've come to and explores analogies you like rather than exploring the arguments and trying to decide what side to be on, and it fails to embrace scout mindset sufficiently to be helpful.
Davidmanheim @ 2025-01-30T16:40 (+2) in response to Is it ethical to work in AI "content evaluation"?
I think that replaceability is very high, so the counterfactual impact is minimal. But that said, there is very little possibility in my mind that even helping with RLHF for compliance with their "safety" guidelines is more beneficial for safety than for accelerating capabilities racing, so any impact is negative.
Sarah Cheng @ 2025-01-29T18:30 (+5) in response to Climate Change Is Worse Than Factory Farming
Interesting, why's that? :)
For context: I want the Forum team to be able to do more active crossposting in the future, so it seems reasonable to have a sort of "placeholder" account for when the author of the original piece doesn't have a Forum account. Personally when I see a linkpost, I generally assume that the author here is also the original author (outside of obvious cases like a NYT article link), and it's kinda confusing when that's not the case (I'm more used to this now but it was extra confusing when I was a new user). I also personally feel kinda weird getting karma for just linking to someone else's work, and probably don't want my Forum profile to mostly be links to other articles.
On the other hand, I do want users to feel free to linkpost external articles that they didn't write, especially if they add their own commentary, or even just make some editorial decisions on which parts to quote. (That's why I was fine with crossposting this using my own account, for example.)
harfe @ 2025-01-30T16:27 (+4)
Caveat: I consider these minor issues, I hope I don't come across as too accusatory.
It seems that the reason for cross-posting was that you personally found it interesting. If you use the EA forum team account, it sounds a bit like an "official" endorsement, and makes the Forum Team less neutral.
Even if you use another account name (eg "selected linkposts") that is run by the Forum Team, I think there should be some explanation how those linkposts are selected, otherwise it seems like arbitrarily privileging some stuff over other stuff.
A "LinkpostBot" account would be good if the cross-posting is automated (e.g. every ACX article who mentions Effective Altruism).
I think its fine to gain Karma by virtue of linkposting and being an active forum member, I will not be bothered by it and I think you should not worry about that (although i can understand that it might feel uncomfortable to you). Other people are also allowed to link-post.
I think starting the title with [linkpost] fixes that issue.
Soemano Zeijlmans @ 2025-01-30T16:11 (+6) in response to Soemano Zeijlmans's Quick takes
Which interesting EA-related bluesky accounts do you know of?
I'm not using Twitter anymore since it's being used to promote hateful views, but Bluesky is quite a cool online space in my opinion.
I'm making a list of Bluesky accounts of EA-related organisations and key people. If you're active on Bluesky or some of your favourite EA orgs or key people are, please leave a comment with a link to their profile!
I've also made an EA (GHD+AW+CC) Starter Pack in case you're interested. Let me know who I should add! Effective Environmentalism also has a pack with effectiveness-oriented climate change accounts.
Some accounts in no particular order:
WillieG @ 2025-01-23T13:30 (+1) in response to Should EAs help employees in the developing world move to the West?
From my rationality-married-with-emotion-and-values human brain, I agree with you. Evil indeed.
That said, I can see a dystopian future where Hyper-Rationalist Bot makes all decisions, and decides that "the greatest good for the greatest number" is best served by keeping human capital in the developing world, using the EA logic that capital in the developing world creates more utility than the same capital in the developed world. (In fact, HRB thinks we should send capable people from the developed world to the developing world to accelerate utility growth even more.)
D0TheMath @ 2025-01-30T16:05 (+3)
This seems pretty unlikely to me tbh, people are just less productive in the developing world than the developed world, and its much easier to do stuff--including do good--when you have functioning institutions, surrounded by competent people, connections & support structures, etc etc.
That's not to say sending people to the developed world is bad. Note that you can get lots of the benefits of living in a developed country by simply having the right to live in a developed country, or having your support structure or legal system or credentials based in a developed country.
Of course, its much easier to just allow everyone in a developing country to just move to a developed country, but assuming the hyper rationalist bot exists with an open boarders constraint, it seems incredibly obvious to me that what you say would not happen.
James Herbert @ 2025-01-30T16:04 (+5) in response to You should read Hobbes, Locke, Hume, and Mill via EarlyModernTexts.com
This is cool and I didn't know it existed. Thanks for sharing!
Vasco Grilođ¸ @ 2025-01-30T16:02 (+3) in response to How can we get the world to talk about animalsâ inviolable rights?
Hi David,
You and readers may be interested in Towards A Global Ban On Industrial Animal Agriculture By 2050: Legal Basis, Precedents, And Instruments.
Soemano Zeijlmans @ 2025-01-30T16:01 (+1) in response to AI Audit on Costa Rica
Are you based in Costa Rica? You should totally suggest giving a talk about this at the University for Peace (UPAZ) of the UN in Ciudad ColĂłn! Let me know if you'd like an introduction.
Joanna Michalska @ 2025-01-30T15:50 (+1) in response to Is it ethical to work in AI "content evaluation"?
I don't know how ethical it is compared to zero, but compared to the most likely counterfactual (some other person, not aligned with EA and less concerned about AI risks getting this job) I think it's better if you do it.
lauren_mee @ 2025-01-21T15:17 (+11) in response to Thoughts on Moral Ambition by Rutger Bregman
I just want to say thank you for highlighting this Vasco!
I am a big fan of SMA but my largest concern is their quit your job tag line. Firstly, I can only speak for the animal advocacy space but there are a very limited number of high impact roles for people to pivot intoâŚ. secondly, if itâs in the non profit world they are directing them too it will take some time to pivot (so try to not be unemployed first) thirdly, many of these people are in great companies where they could potentially do much more if they were activated or ETG.
Anecdotally Iâve had a handful of people come from them asking for career advise who had quit their job and tbh my first thought is can you get your job backâŚ.
Vasco Grilođ¸ @ 2025-01-30T15:25 (+2)
Thanks, Lauren!
I think this applies more broadly. Overwhelmingly based on data about global health and development interventions, Benjamin Todd concludes "itâs defensible to say that the best of all interventions in an area are about 10 times more effective than the mean, and perhaps as much as 100 times". If so, and jobs are uniformly distributed across interventions, a person in a random job within an area donating 10 % of their gross salary to the best interventions in the area can have 1 (= 0.1*10) to 10 (= 0.1*100) times as much impact from donations as from their direct work. In reality, there will be more jobs in less cost-effective interventions, as the best interventions only account for a small fraction of the overall funding. Based on Benjamin's numbers, if there are 10 times as many people in jobs as cost-effective as a random one as in the best jobs, a person in a random job within an area donating 10 % of their gross salary to the best interventions in the area would be 10 (= 1*10) to 100 (= 10*10) times as impactful as a person with the same job not donating.
Austin @ 2025-01-30T05:48 (+4) in response to Ethical Veganism is Wrong
I'm a bit disappointed, if not surprised, with the community response here. I understand veganism is something of a sacred cow (apologies) in these parts, but that's precisely why Ben's post deserves a careful treatment -- it's the arguments you least agree with that you should extend the most charity to. While this post didn't cause me to reconsider my vegetarianism, historically Ben's posts have had an outsized impact on the way I see things, and I'm grateful for his thoughts here.
Ben's response to point 2 was especially interesting:
And I agree about the significance of human fertility decline. I expect that this comparison, of factory farming to modern human lives, will be a useful metaphor when thinking about how to improve the structures around us.
Jason @ 2025-01-30T15:24 (+13)
<I'm a bit disappointed, if not surprised, with the community response here.>
I can't speak for other voters, but I downvoted due to my judgment that there were multiple critical assumption that were both unsupported / very thinly supported and pretty dubious -- not because any sacred cows were engaged. While I don't think main post authors are obliged to be exhaustive, the following are examples of significant misses in my book:
It's important to not reflexively defend sacred cows or downvote those who criticize them . . . but one can believe that while also believing this post seriously missed the mark and warrants downvotes.
Manuel Allgaier @ 2025-01-30T14:49 (+5) in response to The Game Board has been Flipped: Now is a good time to rethink what youâre doing
This is the best summary of recent developments I've seen so far, thanks a lot for writing this up!
I've shared it with people in my AI Safety / AI Gov network, and we might discuss it in our AI Gov meetup tonight.
PabloAMC đ¸ @ 2025-01-30T12:02 (+2) in response to The Game Board has been Flipped: Now is a good time to rethink what youâre doing
I think this had to do more with GDPR than the AI act, so the late release in the EU might be a one-off case. Once you figure out how to comply with data collection, it should be straightforward to extend to new models, if they want to.
Manuel Allgaier @ 2025-01-30T14:42 (+4)
I did not say that this was due to the EU AI Act, agree that GDPR seems more likely. I mentioned it as an example of EU regulation leading to an AI Lab delaying their EU launch / deprioritizing the EU.
JoshuaBlake @ 2025-01-30T13:54 (+4) in response to Charity Cost-Effectiveness Really Does Follow a Power Law
This seems like good worm but the headline and opening paragraph aren't supported when you've shown it might be log-normal. Log-normal and power distributions often have quite different consequences for how important it is to move to very extreme percentiles, and hence this difference can matter for lots of decisions relevant to EA.
Holly Elmore â¸ď¸ đ¸ @ 2025-01-30T07:46 (+8) in response to The Game Board has been Flipped: Now is a good time to rethink what youâre doing
What you write there makes sense but it's not free to have people in those positions, as I said. I did a lot of thinking about this when I was working on wild animal welfare. It seems superficially like you could get the right kind of WAW-sympathetic person into agencies like FWS and the EPA and they would be there to, say, nudge the agency in a way no one else cared about to help animals when the time came. I did some interviews and looked into some historical cases and I concluded this is not a good idea.
Therefore I think trying to influence the values and safety of labs by working there is a bad idea that would not be pulled off.
OscarDđ¸ @ 2025-01-30T13:50 (+4)
My sense is that of the many EAs who have taken EtG jobs quite a few have remained fairly value-aligned? I don't have any data on this and am just going on vibes, but I would guess significantly more than 10%. Which is some reason to think the same would be the case for AI companies. Though plausibly the finance company's values are only orthogonal to EA, while the AI company's values (or at least plans) might be more directly opposed.
Lin BL @ 2025-01-30T13:46 (+1) in response to Biosecurity newsletters you should subscribe to
I would also recommend the Pandemic Action Network newsletter - sign up here.
Joe Rogero @ 2025-01-29T18:32 (+14) in response to The Game Board has been Flipped: Now is a good time to rethink what youâre doing
This seems more extreme than the linked comment suggests? I can't find anything in the comment justifying "99% automation of fully-remote jobs".
Frankly I think we get ASI and everyone dies before we get anything like 99% automation of current remote jobs, due to bureaucratic inertia and slow adoption. Automation of AI research comes first on the jagged frontier. I don't think Ajeya disagrees?
OscarDđ¸ @ 2025-01-30T13:43 (+4)
The comment that Ajeya is replying to is this one from Ryan, who says his timelines are roughly the geometric mean of Ajeya's and Daniel's original views in the post. That is sqrt(4*13) = 7.2 years from the time of the post, so roughly 6 years from now.
As Josh says, the timelines in the original post were answering the question "Median Estimate for when 99% of currently fully remote jobs will be automatable".
So I think it was a fair summary of Ajeya's comment.
Toby Tremlettđš @ 2025-01-30T13:42 (+6) in response to Toby Tremlett's Quick takes
I've made a public Forum Events calendar, which you can add to your Gcal. Hopefully, this will give people longer to think about and write for events like Debate Weeks or Theme Weeks. Let me know if you have any issues adding the calendar, or have suggestions for other ways to track Forum events.
Click to add calendarEdoArad @ 2025-01-30T13:32 (+2) in response to Meta Coordination Forum (2024) / Talent Need Survey
Minor nitpick
Should this be 1,000 times instead? (This also appears at the summary of key results as 10,000)
Nathan Young @ 2025-01-30T12:28 (+2) in response to Who's track record of AI predictions would you like to see evaluated?
Oppenheimer
Manuel Allgaier @ 2025-01-30T09:06 (+4) in response to The Game Board has been Flipped: Now is a good time to rethink what youâre doing
Anthropic released Claude everywhere but the EU first, and their EU release happened only months later, so to some extend labs are already deprioritizing the EU market. I guess this trend would continue? Not sure.
PabloAMC đ¸ @ 2025-01-30T12:02 (+2)
I think this had to do more with GDPR than the AI act, so the late release in the EU might be a one-off case. Once you figure out how to comply with data collection, it should be straightforward to extend to new models, if they want to.
MichaelDickens @ 2025-01-30T04:03 (+4) in response to Charity Cost-Effectiveness Really Does Follow a Power Law
Are you talking about this post? Looks like those cost-effectiveness estimates were written by Ambitious Impact so I don't know if there are some other estimates written by Vasco.
Vasco Grilođ¸ @ 2025-01-30T11:37 (+2)
Thanks for the interest, Michael!
I got the cost-effectiveness estimates I analysed in that post about global health and development directly from Ambitious Impact (AIM), and the ones about animal welfare adjusting their numbers based on Rethink Priorities' median welfare ranges[1].
I do not have my cost-effectiveness estimates collected in one place. I would be happy to put something together for you, such as a sheet with the name of the intervention, area, source, date of publication, and cost-effectiveness in DALYs averted per $. However, I wonder whether it would be better for you to look into sets of AIM's estimates respecting a given stage of a certain research round. AIM often uses them in weighted factor models to inform which ones to move to the next stage or recommend, so they are supposed to be specially comparable. In contrast, mine often concern different assumptions simply because they span a long period of time. For example, I now guess disabling pain is 10 % as intense as I assumed until October.
I could try to quickly adjust all my estimates such that they all reflect my current assumptions, but I suspect it would not be worth it. I believe AIM's estimates by stage of a particular research round would still be more methodologically aligned, and credible to a wider audience. I am also confident that a set with all my estimates, at least if interpreted at face value, much more closely follow a Pareto, lognormal or loguniform distribution than a normal or uniform distribution. I estimate broiler welfare and cage-free campaigns are 168 and 462 times as cost-effective as GiveWellâs top charities, and that the Shrimp Welfare Project (SWP) has been 64.3 k times as cost-effective as such charities.
AIM used to assume welfare ranges conditional on sentience equal to 1 before moving to estimating the benefits of animal welfare interventions in suffering-adjusted days (SADs) in 2024. I believe the new system still dramatically underestimates the intensity of excruciating pain, and therefore the cost-effectiveness of interventions decreasing it. I estimate the past cost-effectiveness of SWP is 639 DALY/$. For AIMâs pain intensities, and my guess that hurtful pain is as intense as fully healthy life, I get 0.484 DALY/$, which is only 0.0757 % (= 0.484/639) of my estimate. Feel free to ask Vicky Cox, senior animal welfare researcher at AIM, for the sheet with their pain intensities, and the doc with my suggestions for improvement.
Ebenezer Dukakis @ 2025-01-30T10:48 (+9) in response to Capitalism and the Very Long Term
The track record of anticapitalist advocacy seems quite poor. See this free book: Socialism: The failed idea that never dies
If you're doing anticapitalist advocacy for EA reasons, I think you need a really clear understanding of why such advocacy has caused so much misery in the past, and how your advocacy will avoid those traps.
I'd say what's needed is not anticapitalist advocacy, so much as small-scale prototyping of alternative economic systems that have strong theoretical arguments for how they will align incentives better, and scale way past Dunbar's number.
You don't need a full replacement for capitalism to test ideas and see results. For example, central planning often fails due to corruption. A well-designed alternative system will probably need a solution for corruption. And such a solution could be usefully applied to an ordinary capitalist democracy.
I concede that AI companies are behaving in a harmful way, but I doubt that anticapitalist advocacy is a particularly tractable way to address that, at least in the short term.
huw @ 2025-01-30T11:19 (+6)
Yes. Thatâs why I only scoped my comment around weak anticapitalism (specifically: placing strong restrictions on wealth accumulation when it leads to market failures), rather than full-scale revolution. Iâm personally probably more reformist and generally pretty pro-market, but anti-accumulation, FWIW.
As I said, I think that instead of siloed advocacy in distinct cause areas, EAs could realise that they have common cause around opposing bad economic incentives. AI safety, farmed animal welfare, and some global health concerns come from the same roots, and there are already large movements well-placed to solve these problems on the political left (ex. labour unions, veganism, environmentalism, internationalist political groups). Indeed, vegan EAs have already allied well with the existing movement to huge success, but this is the exception.
Frankly, I donât see how that leads to bread lines but I am open to a clearer mechanism if you have one?
huw @ 2025-01-30T04:26 (+5) in response to Capitalism and the Very Long Term
Great paper & a strong argument. I would even take it further to argue that most EAs and indeed, longtermists, probably already agree with weak anticapitalism; most EA projects are trying to compensate for externalities or market failures in one form or another, and the increasing turn to policy, rather than altruism, to settle these issues is a good sign.
I think a bigger issue, as youâve confronted on this forum before, is an unwillingness (mostly down to optics / ideological inoculation), to identify these issues as having structural causes in capitalism. This arrests EAs/longtermists from drawing on centuries of knowledge & movement-building, or more to the matter, even representing a coherent common cause that could be addressed through deploying pooled resources (for instance, donating to anticapitalist candidates in US elections). It breaks my heart a bit tbh, but Iâve long accepted it probably wonât happen.
Ebenezer Dukakis @ 2025-01-30T10:48 (+9)
The track record of anticapitalist advocacy seems quite poor. See this free book: Socialism: The failed idea that never dies
If you're doing anticapitalist advocacy for EA reasons, I think you need a really clear understanding of why such advocacy has caused so much misery in the past, and how your advocacy will avoid those traps.
I'd say what's needed is not anticapitalist advocacy, so much as small-scale prototyping of alternative economic systems that have strong theoretical arguments for how they will align incentives better, and scale way past Dunbar's number.
You don't need a full replacement for capitalism to test ideas and see results. For example, central planning often fails due to corruption. A well-designed alternative system will probably need a solution for corruption. And such a solution could be usefully applied to an ordinary capitalist democracy.
I concede that AI companies are behaving in a harmful way, but I doubt that anticapitalist advocacy is a particularly tractable way to address that, at least in the short term.
James Herbert @ 2025-01-30T10:46 (+2) in response to Meta Coordination Forum (2024) / Talent Need Survey
How confident are respondents, or people in general, about the results of the valuable skills question? This feels particularly relevant to me at the moment as a community builder.
Mo Putera @ 2025-01-29T03:11 (+11) in response to Charity Cost-Effectiveness Really Does Follow a Power Law
In 2011, GiveWell published the blog post Errors in DCP2 cost-effectiveness estimate for deworming, which made me lose a fair bit of confidence in DCP2 estimates (and by extension DCP3):
I agree with their key takeaways, in particular (emphasis mine)
That said, my best guess is such spreadsheet errors probably don't change your bottomline finding that charity cost-effectiveness really does follow a power law â in fact I expect the worst cases to be actively harmful (e.g. PlayPump International), i.e. negative DALYs/$. My prior essentially comes from 80K's How much do solutions to social problems differ in their effectiveness? A collection of all the studies we could find, who find:
Regarding your future work I'd like to see section, maybe Vasco's corpus of cost-effectiveness estimates would be a good starting point. His quantitative modelling spans nearly every category of EA interventions, his models are all methodologically aligned (since it's just him doing them), and they're all transparent too (unlike the DCP estimates).
Vasco Grilođ¸ @ 2025-01-30T10:41 (+4)
Thanks for the suggestion, Mo! More transparent methodologically aligned estimates:
Estimates from Rethink Priorities' cross-cause cost-effectiveness model are also methodologically aligned within each area, but they are not transparent. No information at all is provided about the inputs.
AIM's estimates respecting a given stage of a certain research round[3] will be especially comparable, as AIM often uses them in weighted factor models to inform which ones to move to the next stage or recommend. So I think you had better look into such sets of estimates over one covering all my estimates.
Meanwhile, they have published more recommended for the early 2025 incubation program.
Only in-depth reports of recommeded interventions are necessarily published.
There are 3 research rounds per year. 2 on global health and development, and 1 on animal welfare.
Holly Elmore â¸ď¸ đ¸ @ 2025-01-30T07:46 (+8) in response to The Game Board has been Flipped: Now is a good time to rethink what youâre doing
What you write there makes sense but it's not free to have people in those positions, as I said. I did a lot of thinking about this when I was working on wild animal welfare. It seems superficially like you could get the right kind of WAW-sympathetic person into agencies like FWS and the EPA and they would be there to, say, nudge the agency in a way no one else cared about to help animals when the time came. I did some interviews and looked into some historical cases and I concluded this is not a good idea.
Therefore I think trying to influence the values and safety of labs by working there is a bad idea that would not be pulled off.
Ebenezer Dukakis @ 2025-01-30T10:30 (+4)
It seems like your model only has such influence going one way. The lab worker will influence their friends, but not the other way around. I think two-way influence is a more accurate model.
Another option is to ask your friends to monitor you so you don't get ideologically captured, and hold an intervention if it seems appropriate.
Nathan Young @ 2025-01-30T09:37 (+2) in response to Who's track record of AI predictions would you like to see evaluated?
Alexander Hamilton
Nathan Young @ 2025-01-30T09:37 (+2) in response to Who's track record of AI predictions would you like to see evaluated?
I would want to see a big database of the exact wording of the statement that I could look up or at least some large random sample.
Nathan Young @ 2025-01-30T09:36 (+2) in response to Who's track record of AI predictions would you like to see evaluated?
Paul Christiano
Nathan Young @ 2025-01-30T09:36 (+2) in response to Who's track record of AI predictions would you like to see evaluated?
Nixon
Nathan Young @ 2025-01-30T09:36 (+2) in response to Who's track record of AI predictions would you like to see evaluated?
Napoleon
Nathan Young @ 2025-01-30T09:36 (+2) in response to Who's track record of AI predictions would you like to see evaluated?
Kissinger
Nathan Young @ 2025-01-30T09:35 (+2) in response to Who's track record of AI predictions would you like to see evaluated?
Einstein
Nathan Young @ 2025-01-30T09:35 (+2) in response to Who's track record of AI predictions would you like to see evaluated?
Von Neumann
Nathan Young @ 2025-01-30T09:35 (+2) in response to Who's track record of AI predictions would you like to see evaluated?
Eliezer Yudkowsky
yams @ 2025-01-30T02:07 (+5) in response to The Game Board has been Flipped: Now is a good time to rethink what youâre doing
Two years ago short timelines to superintelligence meant decades. That you would structure this bet such that it resolves in just a few years is itself evidence that timelines are getting shorter.
That you would propose the bet at even odds also does not gesture toward your confidence.
Finally, what money means after superintelligence is itself hotly debated, especially for worst-case doomers (the people most likely to expect three year timelines to ASI are also the exact people who don't expect to be alive to collect their winnings).
I think it's possible for a betting structure to prove some kind of point about timelines, but this isn't it.
Vasco Grilođ¸ @ 2025-01-30T09:09 (+2)
Thanks, yams.
I am also potentially open to a bet where I transfer money to the person bullish on AI timelines now. I bet Greg Coulbourn 10 k$ this way. However, I would have to trust the person betting with me more than in the case of the bet I linked to above. On this, money being less valuable after superintellgence (including due to supposedly higher risk of death) has the net effect of moving the break-even resolution date forward. As I say in the post I linked to, "We can agree on another resolution date such that the bet is good for you". The resolution date I proposed (end of 2028) was supposed to make the bet just slightly positive for people bullish on AI timelines. However, my views are closer to those of the median expert in 2023, whose median date of full automation was 2073.
zdgroff @ 2025-01-29T13:54 (+15) in response to The Game Board has been Flipped: Now is a good time to rethink what youâre doing
InterestingâI've had the opposite take for the EU. The low likelihood of regulation in the US seems like it would make EU regulation more important since that might be all there is. (The second point still stands, but it's still unclear how much that retaliation will happen and what impact it will have.)
It depends on aspects of the Brussels' effect, and I guess it could be that a complete absence of US regulation means companies just pull out of the EU in response to regulation there. Maybe recent technical developments make that more likely. On net, I'm still inclined to think these updates increase the importance of EU stuff.
For the UK, I think I'd agreeâUK work seems to get a lot of its leverage from the relationship with the US.
Manuel Allgaier @ 2025-01-30T09:06 (+4)
Anthropic released Claude everywhere but the EU first, and their EU release happened only months later, so to some extend labs are already deprioritizing the EU market. I guess this trend would continue? Not sure.
Holly Elmore â¸ď¸ đ¸ @ 2025-01-29T07:11 (+14) in response to The Game Board has been Flipped: Now is a good time to rethink what youâre doing
Great pieceâ great prompt to rethink things and good digests of implications.
If you agree that mass movement building is a priority, check out PauseAI-US.org , or donate here: https://www.zeffy.com/donation-form/donate-to-help-pause-ai
One implication I strongly disagree with is that people should be getting jobs in AI labs. I donât see you connecting that to actual safety impact, and I sincerely doubt working as a researcher gives you any influence on safety at this point (if it ever did). There is a definite cost to working at a lab, which is capture and NDA-walling. Already so many EAs work at Anthropic that it is shielded from scrutiny within EA, and the attachment to âour playerâ Anthropic has made it hard for many EAs to do the obvious thing by supporting PauseAI. Put simply: I see no meaningful path to impact on safety working as an AI lab researcher, and I see serious risks to individual and community effectiveness and mission focus.
Ebenezer Dukakis @ 2025-01-30T08:58 (+16)
This is a very strong statement. I'm not following technical alignment research that closely, but my general sense is that exciting work is being done. I just wrote this comment advertising a line of research which strikes me as particularly promising.
I noticed the other day that the people who are particularly grim about AI alignment also don't seem to be engaging much with contemporary technical alignment research. That missing intersection seems suspicious. I'm interested in any counterexamples that come to mind.
My subjective sense is there's a good chance we lose because all the necessary insights to build aligned AI were lying around, they just didn't get sufficiently developed or implemented. This seems especially true for techniques like gradient routing which would need to be baked in to a big, expensive training run.
(I'm also interested in arguments for why unlearning won't work. I've thought about this a fair amount, and it seems to me that sufficiently good unlearning kind of just oneshots AI safety, as elaborated in the comment I linked.)
Ebenezer Dukakis @ 2025-01-30T08:33 (+2) in response to The Game Board has been Flipped: Now is a good time to rethink what youâre doing
With regard to Deepseek, it seems to me that the success of mixture-of-experts could be considered an update towards methods like gradient routing. If you could localize specific kinds of knowledge to specific experts in a reliable way, you could dynamically toggle off / ablate experts with unnecessary dangerous knowledge. E.g. toggle off experts knowledgeable in human psychology so the AI doesn't manipulate you.
I like this approach because if you get it working well, it's a general tool that could help address a lot of different catastrophe stories in a way that seems pretty robust. E.g. to mitigate a malicious AI from gaining root access to its datacenter, ablate knowledge of the OS it's running on. To mitigate sophisticated cooperation between AIs that are supposed to be monitoring one another, ablate knowledge of game theory. Etc. (The broader point is that unlearning seems very generally useful. But the "Expand-Route-Ablate" style approach from the gradient routing paper strikes me as a particularly promising, and could harmonize well with MoE.)
I think a good research goal would be to try to eventually replicate Deepseek's work, except with highly interpretable experts. The idea is to produce a "high-assurance" model which can be ablated so undesired behaviors, like deception, are virtually impossible to jailbreak out of it (since the weights that perform the behavior are inaccessible). I think the gradient routing paper is a good start. To achieve sufficient safety we'll need new methods that are more robust and easier to deploy, which should probably be prototyped on toy problems first.
Ebenezer Dukakis @ 2025-01-30T08:08 (+3) in response to The Game Board has been Flipped: Now is a good time to rethink what youâre doing
Some related points:
Trump seems like someone who likes winning, but hates losing, and also likes making deals. If Deepseek increases the probability that the US will lose, that makes it more attractive to negotiate an end to the race. This seems true from both a Trump-psychology perspective, and a rational-game-theorist perspective.
Elon Musk seems to have a good relationship with Chinese leadership.
Releasing open-source AI seems like more of a way to prevent someone else from winning than a way to win yourself.
Chinese leadership may be concerned about x-risks.
I suppose one possible approach would be to try to get some sort of back-channel dialogue going, to start drafting a treaty which can be invoked if political momentum appears.
Lutebemberwa Isa @ 2025-01-30T07:59 (+5) in response to Exercise for 'Our Final Century?'
Becoming a Good Ancestor
As I reflect on life today and the stories of those I admire, I am reminded of the kind of ancestor I hope to become. My grandfather, Rumunyiire, is remembered as a great man, a great ancestor. Though he passed away in the 1970s, the tales of his strength and courage continue to live on.
He was a man of the hills, vast, endless landscapes where he thrived alongside the lions that roamed them. While many feared these wild lands, he embraced them without hesitation. It was on these very hills that he built his home, crafting mud houses and cultivating the land. Yet, he did not seek to tame nature entirely. He left the drinking places and salt licks untouched, allowing the animals to partake in the rhythms of the wild, harmonizing with the landscape he so deeply respected.
To this day, when I walk those lands, I still see the mineral licks, marked by signs that animals continue to use them. He forbade us from ever tampering with them, ensuring that his legacy of coexistence with nature endured long after he was gone.
Conversations with the elders reveal that my grandfather was not only a strong and courageous man but also a kind and honest one. He was known for his generosity, often sharing food from his plantation with the village during times of hunger.
After each harvest, when he returned to the village, he would host grand gatherings, celebrations filled with laughter, food, and a deep sense of community. He ensured that no one left empty-handed. His presence brought warmth and abundance, and people eagerly anticipated his visits, knowing they would be filled with joy and the promise of another unforgettable celebration.
Beyond his generosity, he was a man of principle and justice. Whenever someone in the village committed an immoral act, he was among the first to speak out. He would forbid such behavior, discourage those involved, and ensure that corrective measures were taken within the community. He upheld the values of those around him, not out of self-righteousness, but because he believed in the importance of a strong, moral society.
However, despite the many stories that paint him as a saint, I do not believe he was one. I have walked this earth for 27 years, encountering countless people, yet I have never met anyone who truly fits the description of sainthood. By that logic, my grandfather couldnât have been one either.
What I do believe is that he was a good man, one who did a great deal of good for his children and his community. His kindness, courage, and generosity left behind stories that have endured through generations, so much so that they have overshadowed any faults he may have had. And perhaps that is the true measure of a life well lived, not the absence of flaws, but the presence of an undeniable and lasting impact.
What Does It Mean to Be a Good Ancestor?
When I am asked this question, my answer is simple:
To be a good ancestor is to do a great deal of good, not just for your immediate generation, but for the generations that follow. True greatness is measured by the lasting impact of oneâs actions, shaping a future where others can thrive. The highest form of good is one that prevents existential catastrophe, ensuring that those who come after us inherit a better world.
History offers many examples of such ancestors, figures like Nelson Mandela, who fought for the rights and dignity of South Africans, leaving behind a legacy that continues to inspire.
In my time, there are individuals whom we believe have the potential to become great ancestors, people like Bobi Wine, Kizza Besigye, and Justice Kisakye. However, history will be their ultimate judge. Time will test their actions, challenge their legacies, and determine whether their impact endures.
If future generations continue to feel the weight of their influence, if their work continues to shape lives long after they are gone, then they will have truly become good ancestors.
And that, above all else, is the legacy I, too, hope to leave behind.
Buck @ 2025-01-30T00:53 (+4) in response to The Game Board has been Flipped: Now is a good time to rethink what youâre doing
I agree with you that people seem to somewhat overrate getting jobs in AI companies.
However, I do think there's good work to do inside AI companies. Currently, a lot of the quality-adjusted safety research happens inside AI companies. And see here for my rough argument that it's valuable to have safety-minded people inside AI companies at the point where they develop catastrophically dangerous AI.
Holly Elmore â¸ď¸ đ¸ @ 2025-01-30T07:46 (+8)
What you write there makes sense but it's not free to have people in those positions, as I said. I did a lot of thinking about this when I was working on wild animal welfare. It seems superficially like you could get the right kind of WAW-sympathetic person into agencies like FWS and the EPA and they would be there to, say, nudge the agency in a way no one else cared about to help animals when the time came. I did some interviews and looked into some historical cases and I concluded this is not a good idea.
Therefore I think trying to influence the values and safety of labs by working there is a bad idea that would not be pulled off.
Austin @ 2025-01-30T05:48 (+4) in response to Ethical Veganism is Wrong
I'm a bit disappointed, if not surprised, with the community response here. I understand veganism is something of a sacred cow (apologies) in these parts, but that's precisely why Ben's post deserves a careful treatment -- it's the arguments you least agree with that you should extend the most charity to. While this post didn't cause me to reconsider my vegetarianism, historically Ben's posts have had an outsized impact on the way I see things, and I'm grateful for his thoughts here.
Ben's response to point 2 was especially interesting:
And I agree about the significance of human fertility decline. I expect that this comparison, of factory farming to modern human lives, will be a useful metaphor when thinking about how to improve the structures around us.
bruce @ 2025-01-30T07:30 (+4)
I didn't catch this post until I saw this comment, and it prompted a response. I'm not well calibrated on how much upvotes different posts should get,[1] but at least my personal view is that this wasn't a post where I felt disappointment that this didn't make the front page of the EA Forum, and I don't expect this is a post I'd share with e.g., non-vegans who I'd discuss the meat eater problem with.[2]
I'm assuming you're talking about the downvotes, rather than the comments? I may be mistaken though.
This isn't something I'd usually comment because I do think the EA Forum should be more welcoming on the margin and I think there are a lot of barriers to people posting. But just providing one data point given your disappointment/surprise.
bruce @ 2025-01-30T07:08 (+3) in response to Ethical Veganism is Wrong
One might object to animal suffering, rather than living/dying. So a utilitarian might say factory farming is bad because of the significantly net-negative states that animals endure while alive, while being OK with eating meat from a cow that is raised in a way such that it is living a robustly net positive life, for example.[1]
This isn't an obvious comparison to me, there are clear potential downsides of habitat destruction (loss of ecosystem services) that don't apply to reducing factory farming. There are also a lot of uncertainties around impacts of destroying habitats - it is much harder to recreate the ecosystem and its benefits than to re-introduce factory farming if we are wrong in either case. One might also argue that we might have a special obligation to reduce the harms we cause (via factory farming) than attempt habitat destruction, which is reducing suffering that exists ~independently of humans.
I'm not sure I'm understanding this correctly. Are you saying animals in factory farms have to be able to indicate to you that they don't want to go on existing in order for you to consider taking action on factory farming? What bar do you think is appropriate here?
If there were 100 billion humans being killed for meat / other products every year and living in the conditions of modern factory farms, I would most definitely prioritise and advocate for that as a priority over factory farming.
Can you say more about what you mean by "the domestication of humans"? It seems like you're trying to draw a parallel between domesticated animals and domesticated humans, or modern humans and wild animals in captivity, but I'm not sure what the parallel you are trying to draw is. Could you make this more explicit?
This seems like a confusing argument. Most vegans I know aren't against factory farming because it affects animal replacement rates. It's also seems unlikely to me that reduced fertility rates in humans is a good proxy/correlate for the amount of suffering that exists (it's possible that the relationship isn't entirely linear, but if anything, historically the opposite is more true - countries have reduced fertility rates as they develop and standards of living improve). It's weird that you use fertility rates as evidence for human suffering but seem to have a extremely high bar for animal suffering! Most of the evidence I'm aware of would strongly point to factory farmed animals in fact not tolerating their conditions well.
This is a good argument to work on things that might end humanity or severely diminish it's ability to meaningfully + positively affect the world. Of all the options that might do this, where would you rank reduced fertility rates?
Though (as you note) one might also object to farming animals for food for rights-based rather than welfare-based reasons.
Austin @ 2025-01-30T05:48 (+4) in response to Ethical Veganism is Wrong
I'm a bit disappointed, if not surprised, with the community response here. I understand veganism is something of a sacred cow (apologies) in these parts, but that's precisely why Ben's post deserves a careful treatment -- it's the arguments you least agree with that you should extend the most charity to. While this post didn't cause me to reconsider my vegetarianism, historically Ben's posts have had an outsized impact on the way I see things, and I'm grateful for his thoughts here.
Ben's response to point 2 was especially interesting:
And I agree about the significance of human fertility decline. I expect that this comparison, of factory farming to modern human lives, will be a useful metaphor when thinking about how to improve the structures around us.
MichaelDickens @ 2025-01-30T04:03 (+4) in response to Charity Cost-Effectiveness Really Does Follow a Power Law
Are you talking about this post? Looks like those cost-effectiveness estimates were written by Ambitious Impact so I don't know if there are some other estimates written by Vasco.
Mo Putera @ 2025-01-30T04:40 (+4)
I'm thinking of all of his cost-effectiveness writings on this forum.
huw @ 2025-01-30T04:26 (+5) in response to Capitalism and the Very Long Term
Great paper & a strong argument. I would even take it further to argue that most EAs and indeed, longtermists, probably already agree with weak anticapitalism; most EA projects are trying to compensate for externalities or market failures in one form or another, and the increasing turn to policy, rather than altruism, to settle these issues is a good sign.
I think a bigger issue, as youâve confronted on this forum before, is an unwillingness (mostly down to optics / ideological inoculation), to identify these issues as having structural causes in capitalism. This arrests EAs/longtermists from drawing on centuries of knowledge & movement-building, or more to the matter, even representing a coherent common cause that could be addressed through deploying pooled resources (for instance, donating to anticapitalist candidates in US elections). It breaks my heart a bit tbh, but Iâve long accepted it probably wonât happen.
Benquo @ 2025-01-30T02:05 (+3) in response to Ethical Veganism is Wrong
The clearest evidence of discontent is the unprecedented fertility decline across developed nations. Humans are increasingly choosing not to reproduce at replacement rate when given modern conditions. This isn't just about discomfort - it suggests our large-scale coordination systems (markets, governments, corporations, media) are actively hostile to the welfare of the governed in a way that factory farming isn't.
Nearly no one wants to torture broiler chickens at massive and increasing scale. If we're doing that, this suggests our interlocking coordination systems are already producing outcomes severely misaligned from individual human values and preferences.
Either we restore human agency enough to avoid relying on distasteful and repugnant systems like the worst aspects of factory farming, or we lose the capacity for meaningful ethical action entirely as our systems drift toward whatever our failing coordination mechanisms were optimizing for, or civilization collapses and takes factory farming with it (along with most humans and domesticated animals). Only the first path offers hope of addressing animal welfare systematically.
The decision calculus would be substantially different if we were near the end rather than the beginning of expansion through the universe, just as one should usually focus more on improving one's own capacities earlier in life and on contributing to others' development later on.
MichaelStJules @ 2025-01-30T04:19 (+2)
Why is this clear evidence of discontent? Aren't there many other plausible explanations for the decline in fertility rates, like changes in values and life goals, like ideal family size, prioritization of careers and other interests.
I agree with the first sentence, but I'm not sure about the second. I think a primary reason is that it's not usually a political priority, because it's not actually important to the average voter. If it's not that important, the outcomes are not severely misaligned from individual human values and preferences.
But it can be made a priority through political advocacy. The outcomes of ballot measures seem like pretty good evidence of what people prefer.
I doubt we have ever really had more human agency in the past than now.
This seems wrong to me. While factory farming is increasing, it's primarily because of increasing populations and incomes, and there are effective targeted ways to systematically reduce and mitigate factory farming that don't require increasing human agency as a whole. Basically what the animal welfare side of EA does.
Possibly! But I'd like to see actual intervention proposals and estimates of their effects and cost-effectiveness. If the decision calculus is so obvious, you should be able to easily give a lower bound on the cost-effectiveness that drastically beats targeted animal welfare work (and being fair, should consider long-term effects of animal welfare work).
Joe Rogero @ 2025-01-29T18:32 (+14) in response to The Game Board has been Flipped: Now is a good time to rethink what youâre doing
This seems more extreme than the linked comment suggests? I can't find anything in the comment justifying "99% automation of fully-remote jobs".
Frankly I think we get ASI and everyone dies before we get anything like 99% automation of current remote jobs, due to bureaucratic inertia and slow adoption. Automation of AI research comes first on the jagged frontier. I don't think Ajeya disagrees?
Josh Thorsteinson @ 2025-01-30T04:16 (+5)
They were referring to this quote, from the linked post: "Median Estimate for when 99% of currently fully remote jobs will be automatable."
Automatable doesn't necessarily imply that the jobs are actually automated.
Mo Putera @ 2025-01-29T03:11 (+11) in response to Charity Cost-Effectiveness Really Does Follow a Power Law
In 2011, GiveWell published the blog post Errors in DCP2 cost-effectiveness estimate for deworming, which made me lose a fair bit of confidence in DCP2 estimates (and by extension DCP3):
I agree with their key takeaways, in particular (emphasis mine)
That said, my best guess is such spreadsheet errors probably don't change your bottomline finding that charity cost-effectiveness really does follow a power law â in fact I expect the worst cases to be actively harmful (e.g. PlayPump International), i.e. negative DALYs/$. My prior essentially comes from 80K's How much do solutions to social problems differ in their effectiveness? A collection of all the studies we could find, who find:
Regarding your future work I'd like to see section, maybe Vasco's corpus of cost-effectiveness estimates would be a good starting point. His quantitative modelling spans nearly every category of EA interventions, his models are all methodologically aligned (since it's just him doing them), and they're all transparent too (unlike the DCP estimates).
MichaelDickens @ 2025-01-30T04:03 (+4)
Are you talking about this post? Looks like those cost-effectiveness estimates were written by Ambitious Impact so I don't know if there are some other estimates written by Vasco.
ElliotTep @ 2025-01-30T03:59 (+5) in response to In defense of the certifiers
Just wanted to express that I really appreciate your newsletter as a source of thought leadership. Often I find myself wondering about one aspect of the animal advocacy movement or another, and then a few months later you provide a thoughtful summary of the state of play and next steps.
Ozzie Gooen @ 2025-01-30T02:36 (+8) in response to Retrospective for Inaugural EA SA Summit 2024: Its Impact and What We Learnt
Great work with this! I particularly enjoyed the photos - it's great to see community members across the globe.
I've heard some really positive things about Cape Town before, and generally am excited for more activity in Africa, so I'm quite happy to hear of events like this.
Vasco Grilođ¸ @ 2025-01-29T23:15 (+1) in response to The Game Board has been Flipped: Now is a good time to rethink what youâre doing
Thanks for the post, Lintz. I would be happy to bet 10 k$ against short AI timelines.
yams @ 2025-01-30T02:07 (+5)
Two years ago short timelines to superintelligence meant decades. That you would structure this bet such that it resolves in just a few years is itself evidence that timelines are getting shorter.
That you would propose the bet at even odds also does not gesture toward your confidence.
Finally, what money means after superintelligence is itself hotly debated, especially for worst-case doomers (the people most likely to expect three year timelines to ASI are also the exact people who don't expect to be alive to collect their winnings).
I think it's possible for a betting structure to prove some kind of point about timelines, but this isn't it.
MichaelStJules @ 2025-01-28T15:41 (+21) in response to Ethical Veganism is Wrong
Why do you believe discontent (or its expression) is increasing? On what time scale? And do you expect this trend to continue for long?
Plus, factory farming is also increasing, especially in developing countries and for farmed insects.
Your response to 2 in general seems totally insensitive to the relative numbers involved wrt farmed animals and humans, and our potential impacts on each group. Shouldn't there be a point where you'd prioritize farmed animals?
Benquo @ 2025-01-30T02:05 (+3)
The clearest evidence of discontent is the unprecedented fertility decline across developed nations. Humans are increasingly choosing not to reproduce at replacement rate when given modern conditions. This isn't just about discomfort - it suggests our large-scale coordination systems (markets, governments, corporations, media) are actively hostile to the welfare of the governed in a way that factory farming isn't.
Nearly no one wants to torture broiler chickens at massive and increasing scale. If we're doing that, this suggests our interlocking coordination systems are already producing outcomes severely misaligned from individual human values and preferences.
Either we restore human agency enough to avoid relying on distasteful and repugnant systems like the worst aspects of factory farming, or we lose the capacity for meaningful ethical action entirely as our systems drift toward whatever our failing coordination mechanisms were optimizing for, or civilization collapses and takes factory farming with it (along with most humans and domesticated animals). Only the first path offers hope of addressing animal welfare systematically.
The decision calculus would be substantially different if we were near the end rather than the beginning of expansion through the universe, just as one should usually focus more on improving one's own capacities earlier in life and on contributing to others' development later on.
David_Moss @ 2025-01-29T17:12 (+2) in response to Meta Coordination Forum (2024) / Talent Need Survey
Thanks for the comment Jessica! This makes sense. I have a few thoughts about this:
Patrick Gruban đ¸ @ 2025-01-30T01:21 (+8)
I would have preferred working groups especially for the questions around monetary value of talent which seemed especially hard to get a sense for.
jessica_mccurdyđ¸ @ 2025-01-29T14:50 (+28) in response to Meta Coordination Forum (2024) / Talent Need Survey
Very quickly: I feel like it's useful to share that I did this survey and found it very hard, and a lot of other people did too. In particular, it did feel pretty rushed for such difficult questions that we didn't necessarily have a fully informed pre-existing take on. OP does mention this, but I wanted to stress that for people reading this post.
I still think it has a lot of useful information and is directionally very informative. I might get a chance to write up more thoughts here, but I am not sure I will be able to. I mostly wanted to give a quick additional flag :)
Patrick Gruban đ¸ @ 2025-01-30T01:17 (+14)
I had a similar sense of feeling underprepared and rushed while taking the survey and think my input would have been better with more time and a different setting. At the same time I can see that it could have been hard to get the same group of people to answer without these constraints.
For the monetary value of talent Iâm especially cautious on putting much weight on them as I havenât seen much discussion on such estimates and coming up with a numbers in minutes is hard.
Rather than accepting the numbers at face value, they may be more useful for illustrating directional thinking at a specific moment in time.
Greg_Colbourn @ 2025-01-29T22:35 (+6) in response to Where I Am Donating in 2024
This seems like completely the wrong focus! We need huge numbers of normies involved to get the political pressure necessary to act on AI x-risk before it's too late. We've already tried the "EA's lobbying behind closed doors" approach, and it has failed (/been co-opted by the big AGI companies).
MichaelDickens @ 2025-01-30T01:07 (+4)
I do think there's concern with a popular movement that the movement will move in a direction you didn't want, but empirically this has already happened for "behind closed doors" lobbying so I don't think a popular movement can do worse.
There's also an argument that a popular movement would be too anti-AI and end up excessively delaying a post-AGI utopia, but I discussed in my post why I don't think that's a sufficiently big concern.
(I agree with you, I'm just anticipating some likely counter-arguments)
Holly Elmore â¸ď¸ đ¸ @ 2025-01-29T07:11 (+14) in response to The Game Board has been Flipped: Now is a good time to rethink what youâre doing
Great pieceâ great prompt to rethink things and good digests of implications.
If you agree that mass movement building is a priority, check out PauseAI-US.org , or donate here: https://www.zeffy.com/donation-form/donate-to-help-pause-ai
One implication I strongly disagree with is that people should be getting jobs in AI labs. I donât see you connecting that to actual safety impact, and I sincerely doubt working as a researcher gives you any influence on safety at this point (if it ever did). There is a definite cost to working at a lab, which is capture and NDA-walling. Already so many EAs work at Anthropic that it is shielded from scrutiny within EA, and the attachment to âour playerâ Anthropic has made it hard for many EAs to do the obvious thing by supporting PauseAI. Put simply: I see no meaningful path to impact on safety working as an AI lab researcher, and I see serious risks to individual and community effectiveness and mission focus.
Buck @ 2025-01-30T00:53 (+4)
I agree with you that people seem to somewhat overrate getting jobs in AI companies.
However, I do think there's good work to do inside AI companies. Currently, a lot of the quality-adjusted safety research happens inside AI companies. And see here for my rough argument that it's valuable to have safety-minded people inside AI companies at the point where they develop catastrophically dangerous AI.
Comments on 2025-01-29
yanni kyriacos @ 2025-01-29T23:58 (+2) in response to Yanni Kyriacos's Quick takes
One axis where Capabilities and Safety people pull apart the most, with high consequences is on "asking for forgiveness instead of permission."
1) Safety people need to get out there and start making stuff without their high prestige ally nodding first
2) Capabilities people need to consider more seriously that they're building something many people simply do not want
Dustin Crummett @ 2025-01-29T23:37 (+3) in response to The Lightcone solution to the transmitter room problem
I accept the bullet biting response. I think someone who doesn't should say the utility of the observers may outweigh Jones' utility but that you should save Jones for some deontic reason (which is what Scanlon says), or maybe that many small bits of utility spread across people don't sum in a straightforward way, and so can't add up to outweigh Jones' suffering (I think this is incorrect, but that something like it is probably what's actually driving the intuition). I think the infinite disutility response is wrong, but that someone who accepts it should probably adopt some view in infinite ethics according to which two people suffering infinite disutility is worse than one--adopting some such view may be needed to avoid other problems anyway.
The solution you propose is interesting, but I don't think I find it plausible:
1. If Jones' disutility is finite, presumably there is some sufficiently large number of spectators, X, such that their aggregate utility would outweigh his disutility. Why think that, in fact, the physically possible number of observers is lower than X?
2. Suppose Jones isn't suffering the worst torment possible, but merely "extremely painful" shocks, as in Scanlon's example. So the number of observers needed to outweigh his suffering is not X, but the lower number Y. I suppose the intuitive answer is still that you should save him. But why think the physically possible number of observers is below Y?
3. Even if, in fact, the physically possible number of observers is lower than X, presumably the fundamental moral rules should work across possible worlds. And anyway, that seems to be baked into the thought experiment, as there is in fact no Galactic Cup. But even if the physically possible number of observers is in fact lower than X, it could be higher than X in another possible world.
4. Even if the possible number of observers is in fact finite, presumably there are possible worlds with an infinite number of possible observers (the laws of physics are very different, or time is infinite into the future, or there are disembodied ghosts watching, etc.). If we think the solution should work across possible worlds, the fact that there can only be a finite number of observers in our world is then irrelevant.
5. You assume our lightcone is finite "with certainty." I assume this is because of the expected utility concern if there is some chance that it turns out not to be finite. But I think you shouldn't have epistemic certainty that there can only be a finite number of observers.
6. The solution seems to get the intuitive answer for a counterintuitive reason. People find letting Jones get shocked in the transmitter case counterintuitive because they think there is something off about weighing one really bad harm against all these really small benefits, not because of anything having to do with whether there can only be a finite number of observers, and especially not because of anything having that could depend on the specific number of possible observers. Once we grant that the reason for the intuition is off, I'm not sure why we should trust the intuition itself.
*I think your answer to 1-3 may be that there is no set-in-stone number of observers needed to outweigh Jones' suffering: we just pick some arbitrarily large amount and assign it to Jones, such that it's higher than the total utility possessed by however many observers there might happen to be. I am a realist about utility in such a way that we can't do this. But anyway, here is a potential argument against this:
Forget about what number we arbitrarily assign to represent Jones' suffering. Two people each suffering very slightly less than Jones is worse than Jones' suffering. Four people each suffering very slightly worse than them is worse than their suffering. Etc. If we keep going, we will reach some number of people undergoing some trivial amount of suffering which, intuitively, can be outweighed by enough people watching the Galactic Cup--call that number of observers Z. The suffering of those trivially suffering people is worse than the suffering of Jones, by transitivity. So the enjoyment of Z observers outweighs the suffering of Jones, by transitivity. And there is no reason to think the actual number of possible observers is smaller than Z.
idea21 @ 2025-01-29T23:34 (+1) in response to Capitalism and the Very Long Term
Impressive work, but it is not difficult to convince people of the risks of capitalism when it comes to facing longtermism challenges. We have the "social market economy", in which there are supposedly democratic controls on capitalism.
But from an imaginative perspective, an alternative to capitalism based on a purely altruistic economy is not inconceivable. An altruistic economy should not be confused with a socialist economy (legislation for the common good), but rather should be related to individualistic cultural conceptions such as the ethics of caring. https://en.wikipedia.org/wiki/Ethics_of_care
OllieBase @ 2025-01-29T16:49 (+25) in response to OllieBase's Quick takes
I found this finding in the MCF 2024 survey interesting:
This survey was hard and only given to a small number of people, so we shouldn't read too much into the specific numbers, but I think it's still a data point against putting significant weight on replacability concerns if you have a job offer for an org you consider impactful.
Survey respondents here (who all work at EA orgs like Open Phil, 80k, CEA, Giving What We Can) are saying that if they make someone a job offer, they would need to receive, in the typical case for junior staff, tens of thousands of dollars to be indifferent about that person taking the job instead of the next best candidate. As someone who's been involved in several hiring rounds, this sounds plausible to me.
If you get a job offer from an org you consider impactful, I suggest not putting significant weight on the idea that the next best candidate could also take the role and have just as much or more impact as you, unless you have a good reason to think you're in an atypical situation. There's often a (very) large gap!
FYI the question posed was:
(there's a debate to be had about how "EA org receiving X in financial compensation" compares to "value to the world in $ terms" or "value in EA-aligned donations" but I stand by the above bolded claim).
Full disclosure: I work at CEA and helped build the survey, so I'm somewhat incentivised to say this work was interesting and valuable.
Brad Westđ¸ @ 2025-01-29T23:26 (+5)
Another set of actors that would be incentivized in this would be the survey respondents, to say higher counterfactual values of first vs second choices. Saying otherwise could go against their goals of attracting more of the EA talent pool to their positions. The framing of irreplaceability for their staff also tends to lend to the prestige of their organizations and staff.
With limited applicants, especially in very specialized areas, I think there is definitely a case for a high value of first vs. second choice applicant. But I suspect that this set of survey respondents would be biased in the direction of overestimating the counterfactual impact.
quinn @ 2025-01-29T03:07 (+16) in response to The Game Board has been Flipped: Now is a good time to rethink what youâre doing
nitpick: you say open source which implies I can read it and rebuild it on my machine. I can't really "read" the weights in this way, I can run it on my machine but I can't compile it without a berjillion chips. "open weight" is the preferred nomenclature, it fits the situation better.
(epistemic status: a pedantry battle, but this ship has sailed as I can see other commenters are saying open source rather than open weight).
Davidmanheim @ 2025-01-29T23:16 (+3)
Thank you for fighting the good fight!
Vasco Grilođ¸ @ 2025-01-29T23:15 (+1) in response to The Game Board has been Flipped: Now is a good time to rethink what youâre doing
Thanks for the post, Lintz. I would be happy to bet 10 k$ against short AI timelines.
Esben Kran @ 2025-01-29T17:50 (+1) in response to The Game Board has been Flipped: Now is a good time to rethink what youâre doing
The main effect of regulation is to control certain net negative outcomes and hence slowing down negative AGIs. RSPs that require stopping developing at ASL-4 or otherwise are also under the pausing agenda. It might be a question of semantics due to how Pause AI and the Pause AI Letter have become the memetic sink for the term pause AI?
PabloAMC đ¸ @ 2025-01-29T23:12 (+2)
My point is that slowing AI down is often an unwanted side effect, from the regulator perspective. Thus, the main goal is raising the bar for safety practices across developers.
david_reinstein @ 2025-01-29T22:13 (+2) in response to Unjournal: "Research with potential for impact" database
I just updated it (same link) to include a much larger set of papers. The 'other' category now includes papers we may not yet have prioritized, may have deprioritized somewhat (but still found interesting), or may have set aside for other reasons (e.g., slightly outside our scope, not timely for evaluation as part of our model, etc.)
Now grouped by main outcome/cause/field cluster.
david_reinstein @ 2025-01-29T22:51 (+2)
Sharmake @ 2025-01-24T19:30 (+1) in response to Are AI safetyists crying wolf?
I think this might not be irrationality, but a genuine difference in values.
In particular, I think something like a discount rate disagreement is at the core of a lot of disagreements on AI safety, and to be blunt, you shouldn't expect convergence unless you successfully persuade them of this.
Greg_Colbourn @ 2025-01-29T22:42 (+2)
I don't think it's discount rate (esp given short timelines); I think it's more that people haven't really thought about why their p(doom|ASI) is low. But people seem remarkably resistant to actually tackle the cruxes of the object level arguments, or fully extrapolate the implications of what they do agree on. When they do, they invariably come up short.
MikhailSamin @ 2024-12-17T15:06 (+5) in response to Where I Am Donating in 2024
Update: I've received feedback from the SFF round; we got positive evaluations from two recommenders (so my understanding is the funding allocated to us in the s-process was lower than the speculation grant) and one piece of negative feedback. The negative feedback mentioned that our project might lead to EA getting swamped by normies with high inferential distances, which can have negative consequences; and that because of that risk, "This initiative may be worthy of some support, but unfortunately other orgs in this rather impressive lineup must take priority".
If you're considering donating to AIGSI/AISGF, please reach out! My email is ms@contact.ms.
Greg_Colbourn @ 2025-01-29T22:35 (+6)
This seems like completely the wrong focus! We need huge numbers of normies involved to get the political pressure necessary to act on AI x-risk before it's too late. We've already tried the "EA's lobbying behind closed doors" approach, and it has failed (/been co-opted by the big AGI companies).
Joe Rogero @ 2025-01-29T18:35 (+3) in response to The Game Board has been Flipped: Now is a good time to rethink what youâre doing
Honestly this writeup did update me somewhat in favor of at least a few competent safety-conscious people working at major labs, if only so the safety movement has some access to what's going on inside the labs if/when secrecy grows. The marginal extra researcher going to Anthropic, though? Probably not.
Holly Elmore â¸ď¸ đ¸ @ 2025-01-29T22:25 (+11)
Connect the rest of the dots for me-- how does that researcher's access become community knowledge? How does the community do anything productive with this knowledge? How do you think people working at the labs detracts from other strategies?
OllieBase @ 2025-01-29T16:26 (+7) in response to The Game Board has been Flipped: Now is a good time to rethink what youâre doing
What makes you think this? Zach's post is a clear counterexample here (though comments are friendlier to Anthropic) and I've heard of criticism of the RSPs (though I'm not watching closely).
Maybe you think there should be much more criticism?
Holly Elmore â¸ď¸ đ¸ @ 2025-01-29T22:22 (+11)
There should be protests against them (PauseAI US will be protesting them in SF 2/28) and we should all consider them evil for building superintelligence when it is not safe! Dario is now openly calling for recursive self-improvement. They are the villains-- this is not hard. The fact that you would think Zach's post with "maybe" in the title is scrutiny is evidence of the problem.