Latest comments on the EA Forum

Comments on 2025-02-14

James Herbert @ 2025-02-13T14:27 (+2) in response to (How) can we help the EA-space with our MEL/impact Consulting best?
  1. Should we work with EA-aligned orgs to support them methodologically with their impact management and MEL, or with non-EA orgs to motivate them to shift towards a more EA-aligned approach regarding impact management?
    1. Not sure, both options have their merits. The Mission Motor is focused on doing the latter for animal welfare organisations, maybe touch base with them to see what their experience has been (if you haven't already).
  2. Do you know EA-related orgs that could need support with impact planning (e.g. with developing a Theory of Change), tracking, or analysis?
    1. I know lots of city and national organisations funded by the Centre for Effective Altruism's Community Building Grants programme would like to improve their M&E (including ourselves at EA Netherlands, but we've already got outside support). Perhaps you could contact Naomi, who is responsible for this programme.
  3. What is your intuition on the level of capacity spent on and knowledge about impact management in EA organizations?
    1. I'm most familiar with community-building EA organisations like ourselves. My impression is that EA CB orgs tend to do better than the average non-profit, but we're still nowhere near as good as we ought to be.
  4. Do you work for an organization yourself (or know one well) that might want to build a ToC with us or do a tracking or measurement workshop?
    1. At EA Netherlands we're already getting outside support with our M&E but I think other national organisations would be interested. Again, I'd consider contacting Naomi from CEA. Right now she's putting together the programme for our next retreat and several requests have been made for M&E training. 
Teresa Jakovlev @ 2025-02-14T12:58 (+1)

Thanks so much. Yes, we are already in touch with Nicole and I will for sure contact Naomi :)

NunoSempere @ 2025-02-13T13:10 (+3) in response to (How) can we help the EA-space with our MEL/impact Consulting best?

Reminds me of 2019, strongly upvoted and sent to a few people.

Some thoughts that come to mind:

  • Should we work with EA-aligned orgs to support them methodologically with their impact management and MEL, or with non-EA orgs to motivate them to shift towards a more EA-aligned approach regarding impact management? Either is fine, or rather, both are broken in their own ways :)
  • In practice, impact measurement and related concerns seem to become more salient when people are trying to fundraise. You might want to time your appeals with funding rounds. For instance, offer to help people think through their theory of impact as they are applying to an SFF round.
  • My intuition might be to seek to attach yourself to large orgs (Charity Entrepeneurship, Animal Charity Evaluators, Anima International, Open Philanthropy, EA Funds, etc.) that might have a larger appetite for the type of service you are offering.
  • How much do you charge?
Teresa Jakovlev @ 2025-02-14T12:52 (+1)

Thanks for your upvotes and ideas!
Our business model is not yet refined, as we currently have funding and are able to be very flexible regarding rates depending on what makes sense individually. For us its important to maximize impact also consider the counterfactual while making pricing decisions. So I can't really give you a concrete answer at this moment in time, but if you want to get in touch, please do so!

 

Matthew_Barnett @ 2025-02-13T20:30 (+2) in response to The standard case for delaying AI appears to rest on non-utilitarian assumptions

Let's define "shumanity" as the set of all humans who are currently alive. Under this definition, every living person today is a "shuman," but our future children may not be, since they do not yet exist. Now, let's define "humanity" as the set of all humans who could ever exist, including future generations. Under this broader definition, both we and our future children are part of humanity.

If all currently living humans (shumanity) were to die, this would be a catastrophic loss from the perspective of shuman values—the values held by the people who are alive today. However, it would not necessarily be a catastrophic loss from the perspective of human values—the values of humanity as a whole, across time. This distinction is crucial. In the normal course of events, every generation eventually grows old, dies, and is replaced by the next. When this happens, shumanity, as defined, ceases to exist, and as such, shuman values are lost. However, humanity continues, carried forward by the new generation. Thus, human values are preserved, but not shuman values.

Now, consider this in the context of AI. Would the extinction of shumanity by AIs be much worse than the natural generational cycle of human replacement? In my view, it is not obvious that being replaced by AIs would be much worse than being replaced by future generations of humans. Both scenarios involve the complete loss of the individual values held by currently living people, which is undeniably a major loss. To be very clear, I am not saying that it would be fine if everyone died. But in both cases, something new takes our place, continuing some form of value, mitigating part of the loss. This is the same perspective I apply to AI: its rise might not necessarily be far worse than the inevitable generational turnover of humans, which equally involves everyone dying (which I see as a bad thing!). Maybe "human values" would die in this scenario, but this would not necessarily entail the end of the broader concept of impartial utilitarian value. This is precisely my point.

Neel Nanda @ 2025-02-14T11:13 (+3)

Because there is a much higher correlation between the value of the current generation of humans and the next one than there is between the values of humans and arbitrary AI entities

Matthew_Barnett @ 2025-02-13T20:01 (+2) in response to The standard case for delaying AI appears to rest on non-utilitarian assumptions

I don’t subscribe to moral realism. My own ethical outlook is a blend of personal attachments—my own life, my family, my friends, and other living humans—as well as a broader utilitarian concern for overall well-being. In this post, I focused on impartial utilitarianism because that’s the framework most often used by effective altruists. 

However, to the extent that I also have non-utilitarian concerns (like caring about specific people I know), those concerns incline me away from supporting a pause on AI. If AI can accelerate technologies that save and improve the lives of people who exist right now, then slowing it down would cost lives in the near term. A more complete, and more rigorous version of this argument was outlined in the post.

What I find confusing about other EA's views, including yours, is why we would assign such great importance to “human values” as something specifically tied to the human species as an abstract concept, rather than merely being partial to actual individuals who exist. This perspective is neither utilitarian, nor is it individualistic. It seems to value the concept of the human species over and above the actual individuals that comprise the species, much like how an ideological nationalist might view the survival of their nation as more important than the welfare of all the individuals who actually reside within the nation.

Neel Nanda @ 2025-02-14T11:13 (+3)

For your broader point of impartiality, I feel like you are continuing to assume some bizarre form of moral realism and I don't understand the case. Otherwise, why do you not consider rocks to be morally meaningful? Why is a plant not valuable? I can come up with reasons, but these are assuming specific things about what is and is not morally valuable in exactly the same way that when I say arbitrary AI beings are on average substantially less valuable because I have specific preferences and values over what matters. I do not understand the philosophical position you are taking here - it feels like you're saying that the standard position is speciesist and arbitrary and then drawing an arbitrary distinction slightly further out?

Matthew_Barnett @ 2025-02-13T20:01 (+2) in response to The standard case for delaying AI appears to rest on non-utilitarian assumptions

I don’t subscribe to moral realism. My own ethical outlook is a blend of personal attachments—my own life, my family, my friends, and other living humans—as well as a broader utilitarian concern for overall well-being. In this post, I focused on impartial utilitarianism because that’s the framework most often used by effective altruists. 

However, to the extent that I also have non-utilitarian concerns (like caring about specific people I know), those concerns incline me away from supporting a pause on AI. If AI can accelerate technologies that save and improve the lives of people who exist right now, then slowing it down would cost lives in the near term. A more complete, and more rigorous version of this argument was outlined in the post.

What I find confusing about other EA's views, including yours, is why we would assign such great importance to “human values” as something specifically tied to the human species as an abstract concept, rather than merely being partial to actual individuals who exist. This perspective is neither utilitarian, nor is it individualistic. It seems to value the concept of the human species over and above the actual individuals that comprise the species, much like how an ideological nationalist might view the survival of their nation as more important than the welfare of all the individuals who actually reside within the nation.

Neel Nanda @ 2025-02-14T11:10 (+2)

If AI can accelerate technologies that save and improve the lives of people who exist right now, then slowing it down would cost lives in the near term.

Huh? This argument only goes through if you have a sufficiently low probability of existential risk or an extremely low change in your probability of existential risk, conditioned on things moving slower. I disagree with both of these assumptions. Which part of your post are you referring to?

Brendon_Wong @ 2025-02-14T10:39 (+2) in response to Teaching AI to reason: this year's most important story

Not a major point, but OpenAI officially brands their models “o1” and “o3,” not “GPT-o1” and “GPT-o3.”

A couple sources: Wikipedia (https://en.m.wikipedia.org/wiki/OpenAI_o1), OpenAI (https://openai.com/o1/)

calebp @ 2024-12-20T09:45 (+4) in response to Announcing the Q1 2025 Long-Term Future Fund grant round

Thanks. Should now be fixed!

Jan Betley @ 2025-02-14T09:17 (+1)

Hey, this is still (again?) 404 - is there some other fast track application option?

Benjamin_Todd @ 2025-02-13T23:55 (+4) in response to Teaching AI to reason: this year's most important story

For test time compute, you need to do logarithmic increases of compute to get linear increases in accuracy on the benchmark. It's similar to the pretraining scaling law.

I agree test time compute isn't especially explosive – it mainly serves to "pull forward" more advanced capabilities by 1-2 years.

More broadly, you can swap training for inference: https://epoch.ai/blog/trading-off-compute-in-training-and-inference

On brute force, I mainly took Toby's thread to be saying we don't clearly have enough information to know how effective test time compute is vs. brute force. 

tobycrisford 🔸 @ 2025-02-14T08:02 (+1)

Ah, that's a really interesting way of looking at it, that you can trade training-compute for inference-compute to only bring forward capabilities that would have come about anyway via simply training larger models. I hadn't quite got this message from your post.

My understanding of Francois Chollet's position (he's where I first heard the comparison of logarithmic inference-time scaling to brute force search - before I saw Toby's thread) is that RL on chain of thought has unlocked genuinely new capabilities that would have been impossible simply by scaling traditional LLMs (or maybe it has to be chain of thought combined with tree-search - but whatever the magic ingredient is he has acknowledged that o3 has it).

Of course this could just be his way of explaining why the o3 ARC results don't prove his earlier positions wrong. People don't like to admit when they're wrong! But this view still seems plausible to me, it contradicts the 'trading off' narrative, and I'd be extremely interested to know which picture is correct. I'll have to read that paper!

But I guess maybe it doesn't matter a lot in practice, in terms of the impact that reasoning models are capable of having.

Lutebemberwa Isa @ 2025-02-14T05:27 (+1) in response to Exercise for 'What do you think?'

The Introduction to Effective Altruism course is well-designed and accessible to participants from all over the world. Its interface is optimized for low data usage, making it easily accessible on mobile phones and other low-spec devices. The content is clear and understandable for anyone with basic reading and writing skills. Additionally, it is well-summarized and enriched with engaging facts, making the learning experience both informative and enjoyable.

Now, on criticisms: EA Everywhere on Slack feels overwhelmingly centered on Europe/ America. It is filled with opportunities that are primarily accessible to people from Europe, along with event announcements that are often restricted to the region. For someone like me, coming from Uganda and seeking a like-minded community to grow and develop my skills while staying committed to my country, this exclusion is disheartening. It creates a sense of isolation and limitation.

A more inclusive approach is needed, EA opportunities should be accessible to everyone, regardless of location. There should be strong support structures to uplift individuals from underrepresented and less developed regions. After all, the essence of EA is to find the most effective ways to do good, using evidence. That mission should extend to creating equal opportunities for all that are worthy it.

Many of the examples presented in the course are heavily focused on Europe. However, if we are truly committed to solving global problems, we must incorporate diverse contexts from different parts of the world. This exposure would help participants understand a broader range of challenges and design interventions that are effective and scalable across various regions. Perhaps an example from Africa could provide valuable solutions in Europe, just as a European example might offer insights applicable to Africa. By incorporating diverse perspectives, we can foster cross-regional learning and design interventions that are more adaptable and effective on a global scale. Africa, for instance, offers valuable insights on moral philosophy and ethical considerations given its diversity, yet it is barely mentioned. The course gives the impression that all interventions are meant for the U.S. or Europe, overlooking the rich perspectives and pressing issues faced in other parts of the world. A more inclusive approach would ensure that effective altruism remains truly global in its impact.

All the personalities highlighted in the course are white and from either Europe or America. (This is not about race, but rather an observation on representation.) Have Black individuals or people from outside Europe and America not contributed to this movement? Have their efforts gone unrecognized, or is this course unintentionally reinforcing the idea that intelligence and philanthropy are primarily Western traits? If effective altruism is truly a global movement, it should acknowledge and celebrate contributions from diverse backgrounds. Representation matters—not just for inclusivity, but for inspiring a broader audience to engage in meaningful change.

Overall, the course feels like it was designed primarily for a European/ American audience, with supporting structures that task them with finding solutions to the world's biggest problems. However, these "global" challenges seem to focus mainly on issues affecting the developed world, which, while important, represent only a small fraction of the broader global landscape. This approach risks overlooking critical problems faced by the majority of the world's population and reinforces a narrow perspective on what truly constitutes the world's most pressing challenges to only what is identified in the first world countries. Try to put this in the ITN framework but from a perspective of someone coming from an underdeveloped world. I acknowledge that my perspective on this has been shaped by a neartermist lens and not a longtermist lens.

Disclaimer: These are simply my observations, and there is a considerable possibility that I may be wrong. Please take them with a pinch of salt.

NickLaing @ 2025-02-14T05:18 (+12) in response to US Foreign Aid Bridge Fund

Thanks for this initiative! My someone thought through take from someone who knows a bunch of people who lost their jobs and who's work has been mildly but meaningfully affected by USAID cuts is that I would be slow to throw money at projects previously funded by USAID. 

  1. Most USAID projects just won't be the most cost effective place to put your dollar, even if cost effectiveness might be slightly increases by providing a bridge to get them restarted. As much as cutting the aid abruptly is doing large amounts of harm and feels terrible, if the original project is only moderately cost effective it might still not make sense.
  2. Most orgs and projects are often surprisingly resilient to pauses, especially in sub Saharan Africa. There are very few cases where the work wouldn't be able to restart in a few months if funding was returned from somewhere else. This points to me against the "bridging" argument drastically increasing cost effectiveness above what it would have been for the project anyway.
  3. f you do choose to get involved, talk to a lot of people before deciding. Other funders are thinking the same thing  about whether to get involved in similar initiatives, and they might have done a bunch of research on it too. For example I know GiveWell are thinking shit this
  4. I don't love that the site doesn't have actual links to the work that's being funded. Fire example one example seemed super dubious to me "East Africa, one entity cannot make a $100,000 purchase of life-saving HIV/AIDS medications and another cannot purchase $50,000 worth of nutrient-dense foods for children, both because of the freeze on U.S. foreign aid."

    To the best of my knowledge. East African countries still have enough HIV meds for a few months at least, and I don't know much about of parallel programs that would purchase medication separately like this. I'm not saying it's necessarily wrong but I'd like to hear more.

     

David Mathers🔸 @ 2025-02-12T15:40 (+2) in response to The standard case for delaying AI appears to rest on non-utilitarian assumptions

What is coherence here? Perfect coherence sounds like a very strong assumption to me, not a minimal one. 

Davidmanheim @ 2025-02-14T03:35 (+2)

Close enough not to have any cyclic components that would lead to infinite cycles for the nonsatiable component of their utility.

Matthew_Barnett @ 2025-02-12T19:41 (+3) in response to The standard case for delaying AI appears to rest on non-utilitarian assumptions

Are humans coherent with at least one non-satiable component? If so, then I don’t understand the distinction you’re making that would justify positing AI values to be worse than human values from a utilitarian perspective. 

If not, then I’m additionally unclear on why you believe AIs will be unlike humans in this respect, to the extent that they would become "paperclippers." That term itself seems ambiguous to me (do you mean AIs will literally terminally value accumulating certain configurations of matter?). I would really appreciate a clearer explanation of your argument. As it stands, I don’t fully understand what point you’re trying to make.

Davidmanheim @ 2025-02-14T03:34 (+1)

Humans are neither coherent, nor do they necessarily have a nonsatiable goal - though some might. But they have both to a far greater extent than less intelligent creatures.

Lukas_Gloor @ 2025-02-13T14:22 (+14) in response to Matthew_Barnett's Quick takes

I haven't read your other recent comments on this, but here's a question on the topic of pausing AI progress. (The point I'm making is similar to what Brad West already commented.)

Let's say we grant your assumptions (that AIs will have values that matter the same as or more than human values and that an AI-filled future would be just as or more morally important than one with humans in control). Wouldn't it still make sense to pause AI progress at this important junction to make sure we study what we're doing so we can set up future AIs to do as well as (reasonably) possible?

You say that we shouldn't be confident that AI values will be worse than human values. We can put a pin in that. But values are just one feature here. We should also think about agent psychologies and character traits and infrastructure beneficial for forming peaceful coalitions. On those dimensions, some traits or setups seem (somewhat robustly?) worse than others?

We're growing an alien species that might take over from humans. Even if you think that's possibly okay or good, wouldn't you agree that we can envision factors about how AIs are built/trained and about what sort of world they are placed in that affect whether the future will likely be a lot better or a lot worse?

I'm thinking about things like:

  • pro-social insctincts (or at least absence of anti-social ones)
  • more general agent character traits that do well/poorly at forming peaceful coalitions
  • agent infrastructure to help with coordinating (e.g., having better lie detectors, having a reliable information environment or starting out under the chaos of information warfare, etc.)
  • initial strategic setup (being born into AI-vs-AI competition vs. being born in a situation where the first TAI can take to proceed slowly and deliberately)
  • maybe: decision-theoretic setup to do well in acausal interactions with other parts of the multiverse (or at least not do particularly poorly)

If (some of) these things are really important, wouldn't it make sense to pause and study this stuff until we know whether some of these traits are tractable to influence?

(And, if we do that, we might as well try to make AIs have the inclination to be nice to humans, because humans already exist, so anything that kills humans who don't want to die frustrates already-existing life goals, which seems worse than frustrating the goals of merely possible beings.)

I know you don't talk about pausing in your above comment -- but I think I vaguely remember you being skeptical of it in other comments. Maybe that was for different reasons, or maybe you just wanted to voice disagreement with the types of arguments people typically give in favor of pausing?

FWIW, I totally agree with the position that we should respect the goals of AIs (assuming they're not just roleplayed stated goals but deeply held ones -- of course, this distinction shouldn't be uncharitably weaponized against AIs ever being considered to have meaningful goals). I'm just concerned because whether the AIs respect ours in turn, especially when they find themselves in a position where they could easily crush us, will probably depend on how we build them.

Matthew_Barnett @ 2025-02-14T02:37 (+5)

In your comment, you raise a broad but important question about whether, even if we reject the idea that human survival must take absolute priority other concerns, we might still want to pause AI development in order to “set up” future AIs more thoughtfully. You list a range of traits—things like pro-social instincts, better coordination infrastructures, or other design features that might improve cooperation—that, in principle, we could try to incorporate if we took more time. I understand and agree with the motivation behind this: you are asking whether there is a prudential reason, from a more inclusive moral standpoint, to pause in order to ensure that whichever civilization emerges—whether dominated by humans, AIs, or both at once—turns out as well as possible in ways that matter impartially, rather than focusing narrowly on preserving human dominance. 

Having summarized your perspective, I want to clarify exactly where I differ from your view, and why.

First, let me restate the perspective I defended in my previous post on delaying AI. In that post, I was critiquing what I see as the “standard case” for pausing AI, as I perceive it being made in many EA circles. This standard case for pausing AI often treats preventing human extinction as so paramount that any delay of AI progress, no matter how costly to currently living people, becomes justified if it incrementally lowers the probability of humans losing control. 

Under this argument, the reason we want to pause is that time spent on “alignment research” can be used to ensure that future AIs share human goals, or at least do not threaten the human species. My critique had two components: first, I argued that pausing AI is very costly to people who currently exist, since it delays medical and technological breakthroughs that could be made by advanced AIs, thereby forcing a lot of people to die who could have otherwise been saved. Second, and more fundamentally, I argued that this "standard case" seems to rest on an assumption of strictly prioritizing human continuity, independent of whether future AIs might actually generate utilitarian moral value in a way that matches or exceeds humanity.

I certainly acknowledge that one could propose a different rationale for pausing AI, one which does not rest on the premise that preserving the human species is intrinsically worth more than than other moral priorities. This is, it seems, the position you are taking.

Nonetheless, I don't find your considerations compelling for a variety of reasons.

To begin with, it might seem that granting ourselves "more time" robustly ensures that AIs come out morally better—pro-social, cooperative, and so on. Yet the connection between “getting more time” to “achieving positive outcomes” does not seem straightforward. Merely taking more time does not ensure that this time will be used to increase, rather than decrease, the relevant quality of AI systems according to an impartial moral view. Alignment with human interests, for example, could just as easily push systems in directions that entrench specific biases, maintain existing social structures, or limit moral diversity—none of which strongly aligns with the “pro-social” ideals you described. In my view, there is no inherent link between a slower timeline and ensuring that AIs end up embodying genuinely virtuous or impartial ethical principles. Indeed, if what we call “human control” is mainly about enforcing the status quo or entrenching the dominance of the human species, it may be no better—and could even be worse—than a scenario in which AI development proceeds at the default pace, potentially allowing for more diversity and freedom in how systems are shaped.

Furthermore, in my own moral framework—which is heavily influenced by preference utilitarianism—I take seriously the well-being of everyone who currently exists in the present. As I mentioned previously, one major cost to pausing AI is that it would likely postpone many technological benefits. These might include breakthroughs in medicine—potential cures for aging, radical extensions of healthy lifespans, or other dramatic increases to human welfare that advanced AI could accelerate. We should not simply dismiss the scale of that cost. The usual EA argument for downplaying these costs rests on the Astronomical Waste argument. However, I find this argument flawed, and I spelled out exactly why I found this argument flawed in the post I just wrote. 

If a pause sets back major medical discoveries by even a decade, that delay could contribute to the premature deaths of around a billion people alive today. It seems to me that an argument in favor of pausing should grapple with this tradeoff, instead of dismissing it as clearly unimportant compared to the potential human lives that could maybe exist in the far future. Such a dismissal would seem both divorced from common sense concern for existing people, and divorced from broader impartial utilitarian values, as it would prioritize the continuity of the human species above and beyond species-neutral concerns for individual well-being.

Finally, I take very seriously the possibility that pausing AI would cause immense and enduring harm by requiring the creation of vast regulatory controls over society. Realistically, the political mechanisms by which we “pause” advanced AI development would likely involve a lot of coercion, surveillance and social control, particularly as AI starts becoming an integral part of our economy. These efforts are likely to expand state regulatory powers, hamper open competition, and open the door to a massive intrusion of state interference in economic and social activity. I believe these controls would likely be far more burdensome and costly than, for example, our controls over nuclear weapons. If our top long-term priority is building a more free, prosperous, inclusive, joyous, and open society for everyone, rather than merely to control and stop AI, then it seems highly questionable that creating the policing powers required to pause AI is the best way to achieve this objective.

As I see it, the core difference between the view you outlined and mine is not that I am ignoring the possibility that we might “do better” by carefully shaping the environment in which AIs arise. I concede that if we had a guaranteed mechanism to spend a known, short period of time intentionally optimizing how AIs are built, without imposing any other costs in the meantime, that might bring some benefits. However, my skepticism flows from the actual methods by which such a pause would come about, its unintended consequences on liberty, the immediate harms it imposes on present-day people by delaying technological progress, and the fact that it might simply entrench a narrower or more species-centric approach that I explicitly reject. It is not enough to claim that “pausing gives us more time", suggesting that "more time" is robustly a good thing. One must argue why that time will be spent well, in a way that outweighs the enormous and varied costs that I believe are incurred by pausing AI.

To be clear, I am not opposed to all forms of regulation. But I tend to prefer more liberal approaches, in the sense of classical liberalism. I prefer strategies that try to invite AIs into a cooperative framework, giving them legal rights and a path to peaceful integration—coupled, of course, with constraints on any actor (human or AI) who threatens to commit violence. This, in my view, simply seems like a far stronger foundation for AI policy than a stricter top-down approach in which we halt all frontier AI progress, and establish the associated sweeping regulatory powers required to enforce such a moratorium.

MichaelStJules @ 2025-02-14T01:14 (+4) in response to Introduction to Expected Value Fanaticism

These results are super interesting! Thanks for writing and sharing! (I happened to have already read a bunch of the original papers you're summarizing, and some of your work, too.)

By the way, I think you can capture common intuitions against fanaticism and dependence on unaffected things with a multi-step procedure:

  1. Ignore anything unaffected between your options,[1] and keep only the remaining differences between options.
  2. Use some method that's not fanatical with respect to the remaining differences between options, e.g. the expected value of a bounded function of the differences.

However, this means violating stochastic dominance with respect to outcomes, and the approach might seem ad hoc. While these do count against the approach in my mind, I don't think they rule it out decisively, because the approach seems to match other intuitions of mine so well otherwise. So, something like this approach is something I give a decent amount of weight to under normative uncertainty. I also give some weight to fanatical views, of course, too.

I discuss these ideas a bit in this post, but kind of scattered across the subsections of this section.

  1. ^

    Either just pairwise, dealing with pairwise comparisons first, or across all of them together at the same time.



Comments on 2025-02-13

tobycrisford 🔸 @ 2025-02-13T21:41 (+5) in response to Teaching AI to reason: this year's most important story

This was a thought-provoking and quite scary summary of what reasoning models might mean.

I think this sentence may have a mistake though:

"you can have GPT-o1 think 100-times longer than normal, and get linear increases in accuracy on coding problems."

Doesn't the graph show that the accuracy gains are only logarithmic? The x-axis is a log scale.

This logarithmic relationship between performance and test-time compute is characteristic of brute-force search, and maybe is the one part of this story that means the consequences won't be quite so explosive? Or have I misunderstood?

Benjamin_Todd @ 2025-02-13T23:55 (+4)

For test time compute, you need to do logarithmic increases of compute to get linear increases in accuracy on the benchmark. It's similar to the pretraining scaling law.

I agree test time compute isn't especially explosive – it mainly serves to "pull forward" more advanced capabilities by 1-2 years.

More broadly, you can swap training for inference: https://epoch.ai/blog/trading-off-compute-in-training-and-inference

On brute force, I mainly took Toby's thread to be saying we don't clearly have enough information to know how effective test time compute is vs. brute force. 

idea21 @ 2025-02-13T19:18 (+3) in response to Rich and Poor: How Things Work

Inequality has reached obscene limits in our time, considering the technological advances. A social system that allows such immorality cannot be stable. Moreover, history shows that labor productivity and technological progress can occur under all kinds of social conditions. And greed is not necessarily related to increased productivity.

Vasco Grilo🔸 @ 2025-02-13T23:38 (+2)

Hi there. The Gini coefficient of gross income has decreased a little over the last few decades globally, which means decreased inequality, and I believe the Gini coefficient is one of the best indicators of inequality due to accounting for the whole income distribution.

Our World in Data did not have data on the global trend of the Gini coefficient of net income, which is more relevant than gross income. However, hapiness is even more relevant, and hapiness inequality has apparently been decreasing.

MichaelStJules @ 2025-02-13T23:13 (+3) in response to EAA is relatively overinvesting in corporate welfare reforms

I agree that the numbers don't necessarily match due to experiences not accounted for, although I'd guess they're close enough as a best guess in practice, because WFP covered the most important causes of suffering for egg-laying hens and broilers. (For broilers, they have a separate page for slaughter reform, which is also included in the BCC, but I suppose doesn't reflect transportation or other differences during slaughter due to breed.)

My point was to highlight how great welfare reforms are in utilitarian suffering-reduction terms, relative to preventing animals from being farmed, in response to the original post. We could instead estimate a lower bound on the value of welfare reforms relative to preventing existence, to say welfare reforms are at least X% as good for each animal as preventing that animal from being born and farmed at all. The fact that WFP aimed to be conservative wrt the differences between conventional and reformed helps with this lower bound interpretation.

Also, in case you're not only concerned with suffering, these welfare reforms might increase pleasure or other things of positive value in a chicken's life, while preventing existence actually decreases them. So the welfare reforms could be even better for chickens relative to preventing existence than in my original interpretation. Again, I'd think of it like a lower bound.

Jacob_Peacock @ 2025-02-13T23:20 (+3)

Thank you, that's all helpful!

Jacob_Peacock @ 2025-02-13T22:12 (+1) in response to EAA is relatively overinvesting in corporate welfare reforms

I'm thinking more about this interpretation, but I'm not sure it is correct because WFP's calculations are designed to be conservative in estimating the welfare improvements and exclude various welfare harms. For example, it looks like the broiler estimates exclude welfare harms from transport to slaughter. When these hours of suffering are added back in, the ratio between the two scenarios can go down.

As a hypothetical example, suppose BCC chickens are currently estimated to suffer 50 hours, while non-BCC chickens suffer 100 hours. If we add in 10 hours of suffering from transport for non-BCC chickens and only 2 hours for BCC chickens (as they are believed to be more heat tolerant), this ratio then increases to 53%. So while excluding harms from transport to slaughter is fine for keeping the absolute difference in hours suffered conservative (50=100-50 < 58 = 110-52), it does not necessarily keep the ratio conservative (50% vs 47% suffering reduction).

I think this is fine when comparing between different welfare levels for species, but I suspect it means they can not be used to compare directly to non-existence?

[Tagging @saulius as well since this seems relevant to the extent of whether cage-free is 'still pretty bad'.]

MichaelStJules @ 2025-02-13T23:13 (+3)

I agree that the numbers don't necessarily match due to experiences not accounted for, although I'd guess they're close enough as a best guess in practice, because WFP covered the most important causes of suffering for egg-laying hens and broilers. (For broilers, they have a separate page for slaughter reform, which is also included in the BCC, but I suppose doesn't reflect transportation or other differences during slaughter due to breed.)

My point was to highlight how great welfare reforms are in utilitarian suffering-reduction terms, relative to preventing animals from being farmed, in response to the original post. We could instead estimate a lower bound on the value of welfare reforms relative to preventing existence, to say welfare reforms are at least X% as good for each animal as preventing that animal from being born and farmed at all. The fact that WFP aimed to be conservative wrt the differences between conventional and reformed helps with this lower bound interpretation.

Also, in case you're not only concerned with suffering, these welfare reforms might increase pleasure or other things of positive value in a chicken's life, while preventing existence actually decreases them. So the welfare reforms could be even better for chickens relative to preventing existence than in my original interpretation. Again, I'd think of it like a lower bound.

yochayco @ 2025-02-13T20:07 (+9) in response to Explaining EA: What I learned from giving a TEDx talk

Totally agree with you on your last point. When it comes to spreading the ideas of EA to the public, they can get stuck on the movement's vibe instead of embracing the ideas behind it. 

Yuval Shapira @ 2025-02-13T22:13 (+1)

Thabk you kindly! I would like to clarify that I do think that in some cases there is a big value in the "follow-up" opportunity  (for instance- when I connect to EA events or projects in the local community). However, in other conversations or situations it doesn't seem like the value outweighs the confusion/complication/vibe it brings

MichaelStJules @ 2022-01-06T15:52 (+10) in response to EAA is relatively overinvesting in corporate welfare reforms

Another way of putting this is that these corporate welfare reforms are about half as good as preventing their births, or better, for their welfare. So, corporate welfare reforms over a region (a country, a US state, a province, the EU, a continent, the world, etc.) would be as good as cutting present and future factory farming in that region in half or better (in welfarist terms, ignoring other effects, assuming no new lower standard farms under the reform scenario, etc.).

Jacob_Peacock @ 2025-02-13T22:12 (+1)

I'm thinking more about this interpretation, but I'm not sure it is correct because WFP's calculations are designed to be conservative in estimating the welfare improvements and exclude various welfare harms. For example, it looks like the broiler estimates exclude welfare harms from transport to slaughter. When these hours of suffering are added back in, the ratio between the two scenarios can go down.

As a hypothetical example, suppose BCC chickens are currently estimated to suffer 50 hours, while non-BCC chickens suffer 100 hours. If we add in 10 hours of suffering from transport for non-BCC chickens and only 2 hours for BCC chickens (as they are believed to be more heat tolerant), this ratio then increases to 53%. So while excluding harms from transport to slaughter is fine for keeping the absolute difference in hours suffered conservative (50=100-50 < 58 = 110-52), it does not necessarily keep the ratio conservative (50% vs 47% suffering reduction).

I think this is fine when comparing between different welfare levels for species, but I suspect it means they can not be used to compare directly to non-existence?

[Tagging @saulius as well since this seems relevant to the extent of whether cage-free is 'still pretty bad'.]

NickLaing @ 2025-02-13T20:28 (+10) in response to Explaining EA: What I learned from giving a TEDx talk

I really like these frameworks, lots of interesting communication ideas, and I do think some of the old EA communication classics have find a bit stale. I especially like the blue ocean thing. Nice job! 

Yuval Shapira @ 2025-02-13T22:08 (0)

Thank you! I'm so happy you found it useful!

Matthew_Barnett @ 2025-02-13T20:30 (+2) in response to The standard case for delaying AI appears to rest on non-utilitarian assumptions

Let's define "shumanity" as the set of all humans who are currently alive. Under this definition, every living person today is a "shuman," but our future children may not be, since they do not yet exist. Now, let's define "humanity" as the set of all humans who could ever exist, including future generations. Under this broader definition, both we and our future children are part of humanity.

If all currently living humans (shumanity) were to die, this would be a catastrophic loss from the perspective of shuman values—the values held by the people who are alive today. However, it would not necessarily be a catastrophic loss from the perspective of human values—the values of humanity as a whole, across time. This distinction is crucial. In the normal course of events, every generation eventually grows old, dies, and is replaced by the next. When this happens, shumanity, as defined, ceases to exist, and as such, shuman values are lost. However, humanity continues, carried forward by the new generation. Thus, human values are preserved, but not shuman values.

Now, consider this in the context of AI. Would the extinction of shumanity by AIs be much worse than the natural generational cycle of human replacement? In my view, it is not obvious that being replaced by AIs would be much worse than being replaced by future generations of humans. Both scenarios involve the complete loss of the individual values held by currently living people, which is undeniably a major loss. To be very clear, I am not saying that it would be fine if everyone died. But in both cases, something new takes our place, continuing some form of value, mitigating part of the loss. This is the same perspective I apply to AI: its rise might not necessarily be far worse than the inevitable generational turnover of humans, which equally involves everyone dying (which I see as a bad thing!). Maybe "human values" would die in this scenario, but this would not necessarily entail the end of the broader concept of impartial utilitarian value. This is precisely my point.

Habryka @ 2025-02-13T21:54 (+5)

Now, consider this in the context of AI. Would the extinction of shumanity by AIs be much worse than the natural generational cycle of human replacement?

I think the answer to this is "yes", because your shared genetics and culture create much more robust pointers to your values than we are likely to get with AI. 

Additionally, even if that wasn't true, humans alive at present have obligations inherited from the past and relatedly obligations to the future. We have contracts and inheritance principles and various things that extend our moral circle of concern beyond just the current generation. It is not sufficient to coordinate with just the present humans, we are engaging in at least some moral trade with future generations, and trading away their influence to AI systems is also not something we have the right to do.

(Importantly, I think we have many fewer such obligations to very distant generations, since I don't think we are generally borrowing or coordinating with humans living in the far future very much).

From a more impartial standpoint, the mere fact that AI might not care about the exact same things humans do doesn’t necessarily entail a decrease in total impartial moral value—unless we’ve already decided in advance that human values are inherently more important. 

Look, this sentence just really doesn't make any sense to me. From the perspective of humanity, which is composed of many humans, of course the fact that AI does not care about the same things as humans creates a strong presumption that a world optimized for those values will be worse than a world optimized for human values. Yes, current humans are also limited to what degree we successfully can delegate the fulfillment of our values to future generations, but we also just share, on-average, a huge fraction of our values with future generations. That is a struggle every generation faces, and you are just advocating for... total defeat being fine for some reason? Yes, it would be terrible if the next generation of humans suddenly did not care about almost anything I cared about, but that is very unlikely to happen, but it is quite likely to happen with AI systems. 

tobycrisford 🔸 @ 2025-02-13T21:41 (+5) in response to Teaching AI to reason: this year's most important story

This was a thought-provoking and quite scary summary of what reasoning models might mean.

I think this sentence may have a mistake though:

"you can have GPT-o1 think 100-times longer than normal, and get linear increases in accuracy on coding problems."

Doesn't the graph show that the accuracy gains are only logarithmic? The x-axis is a log scale.

This logarithmic relationship between performance and test-time compute is characteristic of brute-force search, and maybe is the one part of this story that means the consequences won't be quite so explosive? Or have I misunderstood?

Luke Dawes @ 2025-02-13T21:27 (+1) in response to What posts are you thinking about writing?

I'll post my ideas as replies to this, so they can be voted on separately.

Luke Dawes @ 2025-02-13T21:30 (+1)

(See here for a draft I whipped up for this, and feel free to comment!) Hayden Wilkinson’s “In defence of fanaticism” argues that you should always take the lower-probability odds of a higher-value reward over the inverse in decision theory, or face serious problems. I think accepting his argument introduces new problems that aren’t described in the paper:

  1. It is implied that each round of Dyson’s Wager (e.g. for each person in the population being presented with the wager) has no subsequent effect on the probability distribution for future rounds, which is unrealistic and doesn’t. I illustrate this with a “small worlds” example.
  2. Fanaticism is only considered under positive theories of value and therefore ignores the offsetting principle, which assumes both the existence of and an exchange rate (or commensurability) between independent goods and bads. I’d like to address this in a future draft with multiple reframings of Dyson’s Wager under minimalist theories of value.
Luke Dawes @ 2025-02-13T21:27 (+1) in response to What posts are you thinking about writing?

I'll post my ideas as replies to this, so they can be voted on separately.

Luke Dawes @ 2025-02-13T21:28 (+1)

(See here for a draft I whipped for this, and feel free to comment!) An Earth-originating artificial superintelligence (ASI) may reason that the galaxy is busy in expectation, and that it could therefore eventually encounter an alien-originating ASI. ASIs from different homeworlds may find it valuable on first contact to verify whether they can each reliably enter into and uphold agreements, by presenting credible evidence of their own pro-social behaviour with other intelligences. If at least one of these ASIs has never met another, the only such agreement it could plausibly have entered into is with its progenitor species – maybe that's us. 

Luke Dawes @ 2025-02-13T21:27 (+1) in response to What posts are you thinking about writing?

I'll post my ideas as replies to this, so they can be voted on separately.

Habryka @ 2025-02-13T20:02 (+3) in response to The standard case for delaying AI appears to rest on non-utilitarian assumptions

Yeah, this. 

From my perspective "caring about anything but human values" doesn't make any sense. Of course, even more specifically, "caring about anything but my own values" also doesn't make sense, but in as much as you are talking to humans, and making arguments about what other humans should do, you have to ground that in their values and so it makes sense to talk about "human values". 

The AIs will not share the pointer to these values, in the same way as every individual does to their own values, and so we should a-priori assume the AI will do worse things after we transfer all the power from the humans to the AIs. 

Matthew_Barnett @ 2025-02-13T20:30 (+2)

Let's define "shumanity" as the set of all humans who are currently alive. Under this definition, every living person today is a "shuman," but our future children may not be, since they do not yet exist. Now, let's define "humanity" as the set of all humans who could ever exist, including future generations. Under this broader definition, both we and our future children are part of humanity.

If all currently living humans (shumanity) were to die, this would be a catastrophic loss from the perspective of shuman values—the values held by the people who are alive today. However, it would not necessarily be a catastrophic loss from the perspective of human values—the values of humanity as a whole, across time. This distinction is crucial. In the normal course of events, every generation eventually grows old, dies, and is replaced by the next. When this happens, shumanity, as defined, ceases to exist, and as such, shuman values are lost. However, humanity continues, carried forward by the new generation. Thus, human values are preserved, but not shuman values.

Now, consider this in the context of AI. Would the extinction of shumanity by AIs be much worse than the natural generational cycle of human replacement? In my view, it is not obvious that being replaced by AIs would be much worse than being replaced by future generations of humans. Both scenarios involve the complete loss of the individual values held by currently living people, which is undeniably a major loss. To be very clear, I am not saying that it would be fine if everyone died. But in both cases, something new takes our place, continuing some form of value, mitigating part of the loss. This is the same perspective I apply to AI: its rise might not necessarily be far worse than the inevitable generational turnover of humans, which equally involves everyone dying (which I see as a bad thing!). Maybe "human values" would die in this scenario, but this would not necessarily entail the end of the broader concept of impartial utilitarian value. This is precisely my point.

NickLaing @ 2025-02-13T20:28 (+10) in response to Explaining EA: What I learned from giving a TEDx talk

I really like these frameworks, lots of interesting communication ideas, and I do think some of the old EA communication classics have find a bit stale. I especially like the blue ocean thing. Nice job! 

Matthew_Barnett @ 2025-02-12T20:46 (+2) in response to The standard case for delaying AI appears to rest on non-utilitarian assumptions

We can assess the strength of people's preferences for future generations by analyzing their economic behavior. The key idea is that if people genuinely cared deeply about future generations, they would prioritize saving a huge portion of their income for the benefit of those future individuals rather than spending it on themselves in the present. This would indicate a strong intertemporal preference for improving the lives of future people over the well-being of currently existing individuals.

For instance, if people truly valued humanity as a whole far more than their own personal well-being, we would expect parents to allocate the vast majority of their income to their descendants (or humanity collectively) rather than using it for their own immediate needs and desires. However, empirical studies generally do not support the claim that people place far greater importance on the long-term preservation of humanity than on the well-being of currently existing individuals. In reality, most people tend to prioritize themselves and their children, while allocating only a relatively small portion of their income to charitable causes or savings intended to benefit future generations beyond their immediate children. If people were intrinsically and strongly committed to the abstract concept of humanity itself, rather than primarily concerned with the welfare of present individuals (including their immediate family and friends), we would expect to see much higher levels of long-term financial sacrifice for future generations than we actually observe.

To be clear, I'm not claiming that people don’t value their descendants, or the concept of humanity at all. Rather, my point is that this preference does not appear to be strong enough to override the considerations outlined in my previous argument. While I agree that people do have an independent preference for preserving humanity—beyond just their personal desire to avoid death—this preference is typically not way stronger than their own desire for self-preservation. As a result, my previous conclusion still holds: from the perspective of present-day individuals, accelerating AI development can still be easily justified if one does not believe in a high probability of human extinction from AI.

Kaspar Brandner @ 2025-02-13T20:14 (+1)

The economic behavior analysis falls short. People usually do not expect to have a significant impact on the survival of humanity. If in the past centuries people had saved a large part of their income for "future generations" (including for us) this would likely have had almost no impact on the survival of humanity, probably not even significantly on our present quality of life. The expected utility of saving money for future generations is simply too low compared to spending the money in the present for themselves. This does just mean that people (reasonably) expect to have little influence on the survival of humanity, not that they are relatively okay with humanity going extinct. If people could somehow directly influence, via voting perhaps, whether to trade a few extra years of life against a significant increase in the likelihood of humanity going extinct, I think the outcome would be predictable.

Though I'm indeed not specifically commenting here on what delaying AI could realistically achieve. My main point was only that the preferences for humanity not going extinct are significant and that they easily outweigh any preferences for future AI coming into existence, without relying on immoral speciesism.

yochayco @ 2025-02-13T20:07 (+9) in response to Explaining EA: What I learned from giving a TEDx talk

Totally agree with you on your last point. When it comes to spreading the ideas of EA to the public, they can get stuck on the movement's vibe instead of embracing the ideas behind it. 

Neel Nanda @ 2025-02-13T19:49 (+13) in response to The standard case for delaying AI appears to rest on non-utilitarian assumptions

Are you assuming some kind of moral realism here? That there's some deep moral truth, humans may or may not have insight into it, so any other intelligent entity is equally likely to?

If so, idk, I just reject your premise. I value what I chose to value, which is obviously related to human values, and an arbitrary sampled entity is not likely to be better on that front

Habryka @ 2025-02-13T20:02 (+3)

Yeah, this. 

From my perspective "caring about anything but human values" doesn't make any sense. Of course, even more specifically, "caring about anything but my own values" also doesn't make sense, but in as much as you are talking to humans, and making arguments about what other humans should do, you have to ground that in their values and so it makes sense to talk about "human values". 

The AIs will not share the pointer to these values, in the same way as every individual does to their own values, and so we should a-priori assume the AI will do worse things after we transfer all the power from the humans to the AIs. 

Neel Nanda @ 2025-02-13T19:49 (+13) in response to The standard case for delaying AI appears to rest on non-utilitarian assumptions

Are you assuming some kind of moral realism here? That there's some deep moral truth, humans may or may not have insight into it, so any other intelligent entity is equally likely to?

If so, idk, I just reject your premise. I value what I chose to value, which is obviously related to human values, and an arbitrary sampled entity is not likely to be better on that front

Matthew_Barnett @ 2025-02-13T20:01 (+2)

I don’t subscribe to moral realism. My own ethical outlook is a blend of personal attachments—my own life, my family, my friends, and other living humans—as well as a broader utilitarian concern for overall well-being. In this post, I focused on impartial utilitarianism because that’s the framework most often used by effective altruists. 

However, to the extent that I also have non-utilitarian concerns (like caring about specific people I know), those concerns incline me away from supporting a pause on AI. If AI can accelerate technologies that save and improve the lives of people who exist right now, then slowing it down would cost lives in the near term. A more complete, and more rigorous version of this argument was outlined in the post.

What I find confusing about other EA's views, including yours, is why we would assign such great importance to “human values” as something specifically tied to the human species as an abstract concept, rather than merely being partial to actual individuals who exist. This perspective is neither utilitarian, nor is it individualistic. It seems to value the concept of the human species over and above the actual individuals that comprise the species, much like how an ideological nationalist might view the survival of their nation as more important than the welfare of all the individuals who actually reside within the nation.

Matthew_Barnett @ 2025-02-13T04:34 (+2) in response to The standard case for delaying AI appears to rest on non-utilitarian assumptions

I think your response largely assumes a human-species-centered viewpoint, rather than engaging with my critique that is precisely aimed at re-evaluating this very point of view. 

You say, AIs will probably not care about the same things, so the universe will be worse by our lights if controlled by AI.” But what are "our lights" and "our values" in this context? Are you referring to the values of me as an individual, the current generation of humans, or humanity as a broad, ongoing species-category? These are distinct—and often conflicting—sets of values, preferences, and priorities. It’s possible, indeed probable, that I, personally, have preferences that differ fundamentally from the majority of humans. "My values" are not the same as "our values".

When you talk about whether an AI civilization is “better” or “worse,” it’s crucial to clarify what perspective we’re measuring that from. If, from the outset, we assume that human values, or the survival of humanity-as-a-species, is the critical factor that determines whether an AI civilization is better or worse than our own, that effectively begs the question. It merely assumes what I aim to challenge. From a more impartial standpoint, the mere fact that AI might not care about the exact same things humans do doesn’t necessarily entail a decrease in total impartial moral value—unless we’ve already decided in advance that human values are inherently more important. 

(To make this point clearer, perhaps replace all mentions of "human values" with "North American values" in the standard arguments about these issues, and see if it makes these arguments sound like they privilege an arbitrary category of beings.)

While it’s valid to personally value the continuation of the human species, or the preservation of human values, as a moral preference above other priorities, my point is simply that that’s precisely the species-centric assumption I’m highlighting, rather than a distinct argument that undermines my observations or analysis. Such a perspective is not substrate or species-neutral. Nor is it obviously mandated by a strictly utilitarian framework; it’s an extra premise that privileges the category "humankind" for its own sake. You may believe that such a preference is natural or good from your own perspective, but that is not equivalent to saying that it is the preference of an impartial utilitarian, who would, in theory, make no inherent distinction based purely on species, or substrate.

Neel Nanda @ 2025-02-13T19:49 (+13)

Are you assuming some kind of moral realism here? That there's some deep moral truth, humans may or may not have insight into it, so any other intelligent entity is equally likely to?

If so, idk, I just reject your premise. I value what I chose to value, which is obviously related to human values, and an arbitrary sampled entity is not likely to be better on that front

David Mathers🔸 @ 2025-02-13T12:10 (+7) in response to Matthew_Barnett's Quick takes

I think for me, part of the issue with your posts on this (which I think are net positive to be clear, they really push at significant weak points in ideas widely held in the community) is that you seem to be sort of vacillating between three different ideas, in a way that conceal that one of them, taken on its own sounds super-crazy and evil:

1) Actually, if AI development were to literally lead to human extinction, that might be fine, because it might lead to higher utility.

2) We should care about humans harming sentient, human-like AIs as much as we care about AIs harming humans.

3) In practice, the benefits to current people from AI development outweigh the risks, and the only moral views which say that we should ignore this and pause in the face of even tiny risks of extinction from AI because there are way more potential humans in the future, in fact, when taken seriously, imply 1), which nobody believes. 

1) feels extremely bad to me, basically a sort of Nazi-style view on which genocide is fine if the replacing people are superior  utility generators (or I guess, inferior but sufficiently more numerous). 1) plausibly is a consequence of classical utilitarianism (even maybe on some person-affecting versions of classical utilitarianism I think), but I take this to be a reason to reject pure classical utilitarianism, not a reason to endorse 1). 2) and 3), on the other hand, seem reasonable to me. But the thing is that you seem at least sometimes to be taking AI moral patienthood as a reason to push on in the face of uncertainty about whether AI will literally kill everyone. And that seems more like 1) than 2) or 3). 1-style reasoning supports the idea that AI moral patienthood is a reason for pushing on with AI development even in the face of human extinction risk, but as far as I can tell 2) and 3) don't. At the same time though I don't think you mean to endorse 1). 

Matthew_Barnett @ 2025-02-13T19:44 (+16)

I realize my position can be confusing, so let me clarify it as plainly as I can: I do not regard the extinction of humanity as anything close to “fine.” In fact, I think it would be a devastating tragedy if every human being died. I have repeatedly emphasized that a major upside of advanced AI lies in its potential to accelerate medical breakthroughs—breakthroughs that might save countless human lives, including potentially my own. Clearly, I value human lives, as otherwise I would not have made this particular point so frequently.

What seems to cause confusion is that I also argue the following more subtle point: while human extinction would be unbelievably bad, it would likely not be astronomically bad in the strict sense used by the "astronomical waste" argument. The standard “astronomical waste” argument says that if humanity disappears, then all possibility for a valuable, advanced civilization vanishes forever. But in a scenario where humans die out because of AI, civilization would continue—just not with humans. That means a valuable intergalactic civilization could still arise, populated by AI rather than by humans. From a purely utilitarian perspective that counts the existence of a future civilization as extremely valuable—whether human or AI—this difference lowers the cataclysm from “astronomically, supremely, world-endingly awful” to “still incredibly awful, but not on a cosmic scale.”

In other words, my position remains that human extinction is very bad indeed—it entails the loss of eight billion individual human lives, which would be horrifying. I don't want to be forcibly replaced by an AI. Nor do I want you, or anyone else to be forcibly replaced by an AI. I am simply pushing back on the idea that such an event would constitute the absolute destruction of all future value in the universe. There is a meaningful distinction between “an unimaginable tragedy we should try very hard to avoid” and “a total collapse of all potential for a flourishing future civilization of any kind.” My stance falls firmly in the former category.

This distinction is essential to my argument because it fundamentally shapes how we evaluate trade-offs, particularly when considering policies that aim to slow or restrict AI research. If we assume that human extinction due to AI would erase all future value, then virtually any present-day sacrifice—no matter how extreme—might seem justified to reduce that risk. However, if advanced AI could continue to sustain its own value-generating civilization, even in the absence of humans, then extinction would not represent the absolute end of valuable life. While this scenario would be catastrophic for humanity, attempting to avoid it might not outweigh certain immediate benefits of AI, such as its potential to save lives through advanced technology.

In other words, there could easily be situations where accelerating AI development—rather than pausing it—ends up being the better choice for saving human lives, even if doing so technically slightly increases the risk of human species extinction. This does not mean we should be indifferent to extinction; rather, it means we should stop treating extinction as a near-infinitely overriding concern, where even the smallest reduction in its probability is always worth immense near-term costs to actual people living today.

For a moment, I’d like to reverse the criticism you leveled at me. From where I stand, it is often those who strongly advocate pausing AI development, not myself, who can appear to undervalue the lives of humans. I know they don’t see themselves this way, and they would certainly never phrase it in those terms. Nevertheless, this is my reading of the deeper implications of their position.

A common proposition that many AI pause advocates have affirmed to me is that it very well could be worth it to pause AI, even if this led to billions of humans dying prematurely due to them missing out on accelerated medical progress that could otherwise have saved their lives. Therefore, while these advocates care deeply about human extinction (something I do not deny), their concern does not seem rooted in the intrinsic worth of the people who are alive today. Instead, their primary focus often seems to be on the loss of potential future human lives that could maybe exist in the far future—lives that do not yet even exist, and on my view, are unlikely to exist in the far future in basically any scenario, since humanity is unlikely to be preserved as a fixed, static concept over the long-run.

In my view, this philosophy neither prioritizes the well-being of actual individuals nor is it grounded in the utilitarian value that humanity actively generates. If this philosophy were purely about impartial utilitarian value, then I ask: why are they not more open to my perspective? Since my philosophy takes an impartial utilitarian approach—one that considers not just human-generated value, but also the potential value that AI itself could create—it would seem to appeal to those who simply took a strict utilitarian approach, without discriminating against artificial life arbitrarily. Yet, my philosophy largely does not appeal to those who express this view, suggesting the presence of alternative, non-utilitarian concerns.

PeterMcCluskey @ 2025-02-13T19:39 (+1) in response to Deepseek and Taiwan

Destroying Taiwan's fabs would make it harder for the West to maintain much of a lead in chips. China likely cares a fair amount about that.

idea21 @ 2025-02-13T19:18 (+3) in response to Rich and Poor: How Things Work

Inequality has reached obscene limits in our time, considering the technological advances. A social system that allows such immorality cannot be stable. Moreover, history shows that labor productivity and technological progress can occur under all kinds of social conditions. And greed is not necessarily related to increased productivity.

MatthewDahlhausen @ 2025-02-13T17:49 (+4) in response to Using a diet offset calculator to encourage effective giving for farmed animals

"The important question is whether eating meat and donating is morally better than eating meat and not donating. The answer to that seems like a resounding 'yes'"

Offsetting bad moral actions depends on 1) the action being off-settable, 2) the two actions are inseparable, and 3) presuming a rather extreme form of utilitarianism is morally correct.

In the case you provide, I think it fails on all three parts. The action isn't off-settable. Most moral frameworks would look at the two actions separately. Donating to an animal welfare charity doesn't first require you eat meat, and there is no forced decision to donate or not donate if you eat an animal. And if you accept moral offsetting is better in this case, you are upon to all sorts of the standard utilitarian critiques.

There are also separate justice concerns and whether you are benefiting the appropriate reference class (if you eat cow and donate to shrimp welfare in another country, is that appropriate offsetting?).

I think it's fine to promote the endeavor (or at least its morally permissible). But saying it is morally better isn't well-supported. It's similar to the somewhat non-intuitive finding in moral philosophy that if choosing between A) not donating to charity, B) donating to an ineffective charity, and C) donating to an effective charity, choosing A over B may be morally permissible, but choosing B over C is not.

Aidan Alexander @ 2025-02-13T18:10 (+8)

Sorry I've been unclear -- let me clarify: When we use the term 'offset,' we mean it in a quantitative sense - doing an amount of good for animals that's comparable in magnitude to the harm caused by one's diet. Whether this good deed makes eating meat ethically equivalent to not eating meat is a complex philosophical question that reasonable people can disagree on. But for someone who is going to eat meat either way (which describes most of our users), adding a donation that helps farmed animals is clearly better than not adding that donation.

The calculator is simply a tool to help people understand what size of donation would create a comparable scale of positive impact to their diet's negative impact. We've found this framing resonates with people who care about animals but aren't ready to change their diet

sammyboiz @ 2025-02-13T17:31 (+3) in response to Using a diet offset calculator to encourage effective giving for farmed animals

interesting, you offset each species independently using the appropriate charity. e.g. an omnivore causes 8.3 fish to be farmed per year and it costs $117 to offset those fish via fish charity. if you offset all animals via SWP or some chicken welfare project, the number might be around $10 to offset a year of meat eating which is similar to my prior of 5 cents daily. (since you dont use moral weights, i am only estimating the $10 figure)

Aidan Alexander @ 2025-02-13T17:56 (+4)

We do use moral weights, in the sense that the Suffering Adjusted Day (SAD) methodology considers both probability of sentience and welfare ranges :)

Aidan Alexander @ 2025-02-13T15:18 (+2) in response to Using a diet offset calculator to encourage effective giving for farmed animals

We agree that there may be morally relevant differences between carbon offsetting and meat offsetting. But as I mention in my FAQ comment, given how the calculator is actually being used (i.e. by people who had no intention of changing their diet), the important question isn't whether eating meat and then paying to offset it is morally equivalent to not eating the meat. The important question is whether eating meat and donating is morally better than eating meat and not donating. The answer to that seems like a resounding 'yes'

MatthewDahlhausen @ 2025-02-13T17:49 (+4)

"The important question is whether eating meat and donating is morally better than eating meat and not donating. The answer to that seems like a resounding 'yes'"

Offsetting bad moral actions depends on 1) the action being off-settable, 2) the two actions are inseparable, and 3) presuming a rather extreme form of utilitarianism is morally correct.

In the case you provide, I think it fails on all three parts. The action isn't off-settable. Most moral frameworks would look at the two actions separately. Donating to an animal welfare charity doesn't first require you eat meat, and there is no forced decision to donate or not donate if you eat an animal. And if you accept moral offsetting is better in this case, you are upon to all sorts of the standard utilitarian critiques.

There are also separate justice concerns and whether you are benefiting the appropriate reference class (if you eat cow and donate to shrimp welfare in another country, is that appropriate offsetting?).

I think it's fine to promote the endeavor (or at least its morally permissible). But saying it is morally better isn't well-supported. It's similar to the somewhat non-intuitive finding in moral philosophy that if choosing between A) not donating to charity, B) donating to an ineffective charity, and C) donating to an effective charity, choosing A over B may be morally permissible, but choosing B over C is not.

sammyboiz @ 2025-02-13T17:31 (+3) in response to Using a diet offset calculator to encourage effective giving for farmed animals

interesting, you offset each species independently using the appropriate charity. e.g. an omnivore causes 8.3 fish to be farmed per year and it costs $117 to offset those fish via fish charity. if you offset all animals via SWP or some chicken welfare project, the number might be around $10 to offset a year of meat eating which is similar to my prior of 5 cents daily. (since you dont use moral weights, i am only estimating the $10 figure)

jeeebz @ 2025-02-13T17:03 (+6) in response to jeeebz's Quick takes

How might EA-aligned orgs in global health and wellness need to adapt calculations of cost-effective interventions given the slash-and-burn campaign currently underway against US foreign aid? Has anyone tried gaming out what different scenarios of funding loss look like (e.g., one where most of the destruction is reversed by the courts, or where that reversal is partial, or where nothing happens and the days are numbered for things like PEPFAR)? Since US foreign aid is so varied, I imagine that's a tall order, but I've been thinking about this quite a bit lately!

Aidan Alexander @ 2025-02-13T15:18 (+2) in response to Using a diet offset calculator to encourage effective giving for farmed animals

We agree that there may be morally relevant differences between carbon offsetting and meat offsetting. But as I mention in my FAQ comment, given how the calculator is actually being used (i.e. by people who had no intention of changing their diet), the important question isn't whether eating meat and then paying to offset it is morally equivalent to not eating the meat. The important question is whether eating meat and donating is morally better than eating meat and not donating. The answer to that seems like a resounding 'yes'

Stijn @ 2025-02-13T16:25 (+1)

I agree, that's why I generally support it.

catpillow @ 2025-02-13T16:14 (+1) in response to What posts are you thinking about writing?

I'm considering writing a post on why it's hard for some people who intellectually agree with EA foundations to be emotionally passionate about EA (and really "doing good" in general). This is mostly based on my experience as a university group organiser, my tendency to be drawn to EA-lite people who end up leaving the community, and the fact that I am not very passionate about EA. Very fuzzy TL;DR is that caring about cause prioritisation requires levels of uncertainty, but the average person needs to be able to see concrete steps to take and how their contribution can help people to feel a fervour that propels them into action. This is doubly true for people who are not surrounded by EAs. To combat this, I argue for one actionable item, and one broader, more abstract ideal. The action item is to have a visual, easily digestable EA roadmap, that links broader cause areas with specific things people and orgs are doing. Ideally, the roadmap would almost be like a bunch of "business pitches" to attract new employees, explaining the pain points, the solutions suggested, and how people can get involved. The broader ideal I want to advocate for is for the EA philosophy to be principles based, but for the day-to-day EA to be missions-based (which I view as different from being cause-area-oriented). 

It's all just vibes in my head right now, but I'd be curious to know if people would want to see interviews/surveys/any sort of data to back up what I'm saying.

Aidan Alexander @ 2025-02-13T15:15 (+4) in response to Using a diet offset calculator to encourage effective giving for farmed animals

Good on your Mum!

Since we carved it out as it's own page in ~September 2024 it's had 4336 visits from 3169 visitors (that's just those who accepted cookies -- we can't see those who didn't).

Since we turned on the ability to track "offset donors" seperately in mid-December, we've had ~30 offsetters sign up. We also have reason to believe that a fair few people who were convinced to donate by the offset calculator ended up making normal donations, not through the calculator

Toby Tremlett🔹 @ 2025-02-13T16:05 (+2)

Thanks! That's awesome. Best of luck growing it. 

Spiarrow @ 2025-02-13T15:44 (+1) in response to You can talk to EA Funds before applying

Thank you, this is quite helpful! 

Does anybody know if the same applies to OpenPhil?

harfe @ 2025-02-13T13:40 (+1) in response to Ideas EAIF is excited to receive applications for

A recent comment says that restriction has been lifted and the website will be updated next week: https://forum.effectivealtruism.org/posts/aBkALPSXBRjnjWLnP/announcing-the-q1-2025-long-term-future-fund-grant-round?commentId=FFFMBth8v7WBqYFzP

gergo @ 2025-02-13T15:20 (+1)

awesome, thanks for flagging this as I would have missed it!

Stijn @ 2025-02-13T12:18 (+1) in response to Using a diet offset calculator to encourage effective giving for farmed animals

I generally support this idea of diet offsetting, although purely morally speaking I have several objections, explained here: https://stijnbruers.wordpress.com/2019/08/22/carbon-offsetting-versus-meat-offsetting/

There are morally relevant differences between carbon offsetting and meat offsetting. 

Aidan Alexander @ 2025-02-13T15:18 (+2)

We agree that there may be morally relevant differences between carbon offsetting and meat offsetting. But as I mention in my FAQ comment, given how the calculator is actually being used (i.e. by people who had no intention of changing their diet), the important question isn't whether eating meat and then paying to offset it is morally equivalent to not eating the meat. The important question is whether eating meat and donating is morally better than eating meat and not donating. The answer to that seems like a resounding 'yes'

Toby Tremlett🔹 @ 2025-02-13T13:20 (+13) in response to Using a diet offset calculator to encourage effective giving for farmed animals

Can I ask how many hits it gets right now? 
I love it btw, I think I've convinced my Mum to use it. 

Aidan Alexander @ 2025-02-13T15:15 (+4)

Good on your Mum!

Since we carved it out as it's own page in ~September 2024 it's had 4336 visits from 3169 visitors (that's just those who accepted cookies -- we can't see those who didn't).

Since we turned on the ability to track "offset donors" seperately in mid-December, we've had ~30 offsetters sign up. We also have reason to believe that a fair few people who were convinced to donate by the offset calculator ended up making normal donations, not through the calculator

James Herbert @ 2025-02-13T14:27 (+2) in response to (How) can we help the EA-space with our MEL/impact Consulting best?
  1. Should we work with EA-aligned orgs to support them methodologically with their impact management and MEL, or with non-EA orgs to motivate them to shift towards a more EA-aligned approach regarding impact management?
    1. Not sure, both options have their merits. The Mission Motor is focused on doing the latter for animal welfare organisations, maybe touch base with them to see what their experience has been (if you haven't already).
  2. Do you know EA-related orgs that could need support with impact planning (e.g. with developing a Theory of Change), tracking, or analysis?
    1. I know lots of city and national organisations funded by the Centre for Effective Altruism's Community Building Grants programme would like to improve their M&E (including ourselves at EA Netherlands, but we've already got outside support). Perhaps you could contact Naomi, who is responsible for this programme.
  3. What is your intuition on the level of capacity spent on and knowledge about impact management in EA organizations?
    1. I'm most familiar with community-building EA organisations like ourselves. My impression is that EA CB orgs tend to do better than the average non-profit, but we're still nowhere near as good as we ought to be.
  4. Do you work for an organization yourself (or know one well) that might want to build a ToC with us or do a tracking or measurement workshop?
    1. At EA Netherlands we're already getting outside support with our M&E but I think other national organisations would be interested. Again, I'd consider contacting Naomi from CEA. Right now she's putting together the programme for our next retreat and several requests have been made for M&E training. 
Matthew_Barnett @ 2025-02-13T04:06 (+32) in response to Matthew_Barnett's Quick takes

A reflection on the posts I have written in the last few months, elaborating on my views

In a series of recent posts, I have sought to challenge the conventional view among longtermists that prioritizes the empowerment or preservation of the human species as the chief goal of AI policy. It is my opinion that this view is likely rooted in a bias that automatically favors human beings over artificial entities—thereby sidelining the idea that future AIs might create equal or greater moral value than humans—and treating this alternative perspective with unwarranted skepticism.

I recognize that my position is controversial and likely to remain unpopular among effective altruists for a long time. Nevertheless, I believe it is worth articulating my view at length, as I see it as a straightforward application of standard, common-sense utilitarian principles that merely lead to an unpopular conclusion. I intend to continue elaborating on my arguments in the coming months.

My view follows from a few basic premises. First, that future AI systems are quite likely to be moral patients; second, that we shouldn’t discriminate against them based on arbitrary distinctions, such as their being instantiated on silicon rather than carbon, or having been created through deep learning rather than natural selection. If we insist on treating AIs fundamentally differently from a human child or adult—for example, by regarding them merely as property to be controlled or denying them the freedom to pursue their own goals—then we should identify a specific ethical reason for our approach that goes beyond highlighting their non-human nature.

Many people have argued that consciousness is the key quality separating humans from AIs, thus rendering any AI-based civilization morally insignificant compared to ours. They maintain that consciousness has relatively narrow boundaries, perhaps largely confined to biological organisms, and would only arise in artificial systems under highly specific conditions—for instance, if one were to emulate a human mind in digital form. While I acknowledge that this perspective is logically coherent, I find it deeply unconvincing. The AIs I am referring to when I write about this topic would almost certainly not be simplistic, robotic automatons; rather, they would be profoundly complex, sophisticated entities whose cognitive abilities rival or exceed those of the human brain. For anyone who adopts a functionalist view of consciousness, it seems difficult to be confident that such AIs would lack a rich inner experience.

Because functionalism and preference utilitarianism—both of which could grant moral worth to AI preferences even if they do not precisely replicate biological states—have at least some support within the EA community, I remain hopeful that, if I articulate my position clearly, EAs who share these philosophical assumptions will see its merits.

That said, I am aware that explaining this perspective is an uphill battle. The unpopularity of my views often makes it difficult to communicate without instant misunderstandings; critics seem to frequently conflate my arguments with other, simpler positions that can be more easily dismissed. At times, this has caused me to feel as though the EA community is open to only a narrow range of acceptable ideas. This reaction, while occasionally frustrating, does not surprise me, as I have encountered similar resistance when presenting other unpopular views—such as challenging the ethics of purchasing meat in social contexts where such concerns are quickly deemed absurd.

However, the unpopularity of these ideas also creates a benefit: it creates room for rapid intellectual progress by opening the door to new and interesting philosophical questions about AI ethics. If we free ourselves from the seemingly unquestionable premise that preserving the human species should be the top priority when governing AI development, we can begin to ask entirely new and neglected questions about the role of artificial minds in society.

These questions include: what social and legal frameworks should we pursue if AIs are seen not as dangerous tools to be contained but as individuals on similar moral footing with humans? How do we integrate AI freedom and autonomy into our vision of the future, creating the foundation for a system of ethical and pragmatic AI rights?

Under this alternative philosophical approach, policy would not focus solely on minimizing risks to humanity. Instead, it would emphasize cooperation and inclusion, seeing advanced AI as a partner rather than an ethical menace to be tightly restricted or controlled. This undoubtedly requires a significant shift in our longtermist thinking, demanding a re-examination of deeply rooted assumptions. Such a project cannot be completed overnight, but given the moral stakes and the rapid progress in AI, I view this philosophical endeavor as both urgent and exciting. I invite anyone open to rethinking these foundational premises to join me in exploring how we might foster a future in which AIs and humans coexist as moral peers, cooperating for mutual benefit rather than viewing each other as intrinsic competitors locked in an inevitable zero-sum fight.

Lukas_Gloor @ 2025-02-13T14:22 (+14)

I haven't read your other recent comments on this, but here's a question on the topic of pausing AI progress. (The point I'm making is similar to what Brad West already commented.)

Let's say we grant your assumptions (that AIs will have values that matter the same as or more than human values and that an AI-filled future would be just as or more morally important than one with humans in control). Wouldn't it still make sense to pause AI progress at this important junction to make sure we study what we're doing so we can set up future AIs to do as well as (reasonably) possible?

You say that we shouldn't be confident that AI values will be worse than human values. We can put a pin in that. But values are just one feature here. We should also think about agent psychologies and character traits and infrastructure beneficial for forming peaceful coalitions. On those dimensions, some traits or setups seem (somewhat robustly?) worse than others?

We're growing an alien species that might take over from humans. Even if you think that's possibly okay or good, wouldn't you agree that we can envision factors about how AIs are built/trained and about what sort of world they are placed in that affect whether the future will likely be a lot better or a lot worse?

I'm thinking about things like:

  • pro-social insctincts (or at least absence of anti-social ones)
  • more general agent character traits that do well/poorly at forming peaceful coalitions
  • agent infrastructure to help with coordinating (e.g., having better lie detectors, having a reliable information environment or starting out under the chaos of information warfare, etc.)
  • initial strategic setup (being born into AI-vs-AI competition vs. being born in a situation where the first TAI can take to proceed slowly and deliberately)
  • maybe: decision-theoretic setup to do well in acausal interactions with other parts of the multiverse (or at least not do particularly poorly)

If (some of) these things are really important, wouldn't it make sense to pause and study this stuff until we know whether some of these traits are tractable to influence?

(And, if we do that, we might as well try to make AIs have the inclination to be nice to humans, because humans already exist, so anything that kills humans who don't want to die frustrates already-existing life goals, which seems worse than frustrating the goals of merely possible beings.)

I know you don't talk about pausing in your above comment -- but I think I vaguely remember you being skeptical of it in other comments. Maybe that was for different reasons, or maybe you just wanted to voice disagreement with the types of arguments people typically give in favor of pausing?

FWIW, I totally agree with the position that we should respect the goals of AIs (assuming they're not just roleplayed stated goals but deeply held ones -- of course, this distinction shouldn't be uncharitably weaponized against AIs ever being considered to have meaningful goals). I'm just concerned because whether the AIs respect ours in turn, especially when they find themselves in a position where they could easily crush us, will probably depend on how we build them.

Charlotte Darnell @ 2025-02-13T14:19 (+1) in response to EA North is happening in Sheffield on Saturday 26 April

So excited about this, thank you for all your hard work making it happen!!!

David Mathers🔸 @ 2025-02-13T12:10 (+7) in response to Matthew_Barnett's Quick takes

I think for me, part of the issue with your posts on this (which I think are net positive to be clear, they really push at significant weak points in ideas widely held in the community) is that you seem to be sort of vacillating between three different ideas, in a way that conceal that one of them, taken on its own sounds super-crazy and evil:

1) Actually, if AI development were to literally lead to human extinction, that might be fine, because it might lead to higher utility.

2) We should care about humans harming sentient, human-like AIs as much as we care about AIs harming humans.

3) In practice, the benefits to current people from AI development outweigh the risks, and the only moral views which say that we should ignore this and pause in the face of even tiny risks of extinction from AI because there are way more potential humans in the future, in fact, when taken seriously, imply 1), which nobody believes. 

1) feels extremely bad to me, basically a sort of Nazi-style view on which genocide is fine if the replacing people are superior  utility generators (or I guess, inferior but sufficiently more numerous). 1) plausibly is a consequence of classical utilitarianism (even maybe on some person-affecting versions of classical utilitarianism I think), but I take this to be a reason to reject pure classical utilitarianism, not a reason to endorse 1). 2) and 3), on the other hand, seem reasonable to me. But the thing is that you seem at least sometimes to be taking AI moral patienthood as a reason to push on in the face of uncertainty about whether AI will literally kill everyone. And that seems more like 1) than 2) or 3). 1-style reasoning supports the idea that AI moral patienthood is a reason for pushing on with AI development even in the face of human extinction risk, but as far as I can tell 2) and 3) don't. At the same time though I don't think you mean to endorse 1). 

harfe @ 2025-02-13T13:57 (+3)

At the same time though I don't think you mean to endorse 1).

I have read or skimmed some of his posts and my sense is that he does endorse 1). But at the same time he says

critics seem to frequently conflate my arguments with other, simpler positions that can be more easily dismissed.

so maybe this is one of these cases and I should be more careful.

gergo @ 2025-02-13T10:21 (+1) in response to Ideas EAIF is excited to receive applications for

Hey! Just wanted to double-check if this is still the case (as it is still on the website), considering that you will be getting a lot of applications for LTFF that would probably want funding for a longer period!

harfe @ 2025-02-13T13:40 (+1)

A recent comment says that restriction has been lifted and the website will be updated next week: https://forum.effectivealtruism.org/posts/aBkALPSXBRjnjWLnP/announcing-the-q1-2025-long-term-future-fund-grant-round?commentId=FFFMBth8v7WBqYFzP

Ramiro @ 2025-02-13T13:28 (+2) in response to How AI Takeover Might Happen in Two Years

thanks. seems to be way better than the new Mission Impossible plot. you could send it to Nature Futures: https://www.nature.com/articles/d41586-024-03507-x
 

Toby Tremlett🔹 @ 2025-02-13T13:20 (+13) in response to Using a diet offset calculator to encourage effective giving for farmed animals

Can I ask how many hits it gets right now? 
I love it btw, I think I've convinced my Mum to use it. 

NunoSempere @ 2025-02-13T13:10 (+3) in response to (How) can we help the EA-space with our MEL/impact Consulting best?

Reminds me of 2019, strongly upvoted and sent to a few people.

Some thoughts that come to mind:

  • Should we work with EA-aligned orgs to support them methodologically with their impact management and MEL, or with non-EA orgs to motivate them to shift towards a more EA-aligned approach regarding impact management? Either is fine, or rather, both are broken in their own ways :)
  • In practice, impact measurement and related concerns seem to become more salient when people are trying to fundraise. You might want to time your appeals with funding rounds. For instance, offer to help people think through their theory of impact as they are applying to an SFF round.
  • My intuition might be to seek to attach yourself to large orgs (Charity Entrepeneurship, Animal Charity Evaluators, Anima International, Open Philanthropy, EA Funds, etc.) that might have a larger appetite for the type of service you are offering.
  • How much do you charge?
Stijn @ 2025-02-13T12:18 (+1) in response to Using a diet offset calculator to encourage effective giving for farmed animals

I generally support this idea of diet offsetting, although purely morally speaking I have several objections, explained here: https://stijnbruers.wordpress.com/2019/08/22/carbon-offsetting-versus-meat-offsetting/

There are morally relevant differences between carbon offsetting and meat offsetting. 

Matthew_Barnett @ 2025-02-13T04:06 (+32) in response to Matthew_Barnett's Quick takes

A reflection on the posts I have written in the last few months, elaborating on my views

In a series of recent posts, I have sought to challenge the conventional view among longtermists that prioritizes the empowerment or preservation of the human species as the chief goal of AI policy. It is my opinion that this view is likely rooted in a bias that automatically favors human beings over artificial entities—thereby sidelining the idea that future AIs might create equal or greater moral value than humans—and treating this alternative perspective with unwarranted skepticism.

I recognize that my position is controversial and likely to remain unpopular among effective altruists for a long time. Nevertheless, I believe it is worth articulating my view at length, as I see it as a straightforward application of standard, common-sense utilitarian principles that merely lead to an unpopular conclusion. I intend to continue elaborating on my arguments in the coming months.

My view follows from a few basic premises. First, that future AI systems are quite likely to be moral patients; second, that we shouldn’t discriminate against them based on arbitrary distinctions, such as their being instantiated on silicon rather than carbon, or having been created through deep learning rather than natural selection. If we insist on treating AIs fundamentally differently from a human child or adult—for example, by regarding them merely as property to be controlled or denying them the freedom to pursue their own goals—then we should identify a specific ethical reason for our approach that goes beyond highlighting their non-human nature.

Many people have argued that consciousness is the key quality separating humans from AIs, thus rendering any AI-based civilization morally insignificant compared to ours. They maintain that consciousness has relatively narrow boundaries, perhaps largely confined to biological organisms, and would only arise in artificial systems under highly specific conditions—for instance, if one were to emulate a human mind in digital form. While I acknowledge that this perspective is logically coherent, I find it deeply unconvincing. The AIs I am referring to when I write about this topic would almost certainly not be simplistic, robotic automatons; rather, they would be profoundly complex, sophisticated entities whose cognitive abilities rival or exceed those of the human brain. For anyone who adopts a functionalist view of consciousness, it seems difficult to be confident that such AIs would lack a rich inner experience.

Because functionalism and preference utilitarianism—both of which could grant moral worth to AI preferences even if they do not precisely replicate biological states—have at least some support within the EA community, I remain hopeful that, if I articulate my position clearly, EAs who share these philosophical assumptions will see its merits.

That said, I am aware that explaining this perspective is an uphill battle. The unpopularity of my views often makes it difficult to communicate without instant misunderstandings; critics seem to frequently conflate my arguments with other, simpler positions that can be more easily dismissed. At times, this has caused me to feel as though the EA community is open to only a narrow range of acceptable ideas. This reaction, while occasionally frustrating, does not surprise me, as I have encountered similar resistance when presenting other unpopular views—such as challenging the ethics of purchasing meat in social contexts where such concerns are quickly deemed absurd.

However, the unpopularity of these ideas also creates a benefit: it creates room for rapid intellectual progress by opening the door to new and interesting philosophical questions about AI ethics. If we free ourselves from the seemingly unquestionable premise that preserving the human species should be the top priority when governing AI development, we can begin to ask entirely new and neglected questions about the role of artificial minds in society.

These questions include: what social and legal frameworks should we pursue if AIs are seen not as dangerous tools to be contained but as individuals on similar moral footing with humans? How do we integrate AI freedom and autonomy into our vision of the future, creating the foundation for a system of ethical and pragmatic AI rights?

Under this alternative philosophical approach, policy would not focus solely on minimizing risks to humanity. Instead, it would emphasize cooperation and inclusion, seeing advanced AI as a partner rather than an ethical menace to be tightly restricted or controlled. This undoubtedly requires a significant shift in our longtermist thinking, demanding a re-examination of deeply rooted assumptions. Such a project cannot be completed overnight, but given the moral stakes and the rapid progress in AI, I view this philosophical endeavor as both urgent and exciting. I invite anyone open to rethinking these foundational premises to join me in exploring how we might foster a future in which AIs and humans coexist as moral peers, cooperating for mutual benefit rather than viewing each other as intrinsic competitors locked in an inevitable zero-sum fight.

David_Moss @ 2025-02-13T12:16 (+17)

Thanks for writing on this important topic!

I think it's interesting to assess how popular or unpopular these views are within the EA community. This year and last year, we asked people in the EA Survey about the extent to which they agreed or disagreed that:

Most expected value in the future comes from digital minds' experiences, or the experiences of other nonbiological entities.

This year about 47% (strongly or somewhat) disagreed, while 22.2% agreed (roughly a 2:1 ratio).

However, among people who rated AI risks a top priority, respondents leaned towards agreement, with 29.6% disagreeing and 36.6% agreeing (a 0.8:1 ratio).[1]

Similarly, among the most highly engaged EAs, attitudes were roughly evenly split between 33.6% disagreement and 32.7% agreement (1.02:1), with much lower agreement among everyone else.

This suggests to me that the collective opinion of EAs, among those who strongly prioritise AI risks and the most highly engaged is not so hostile to digital minds. Of course, for practical purposes, what matters most might be the attitudes of a small number of decisionmakers, but I think the attitudes of the engaged EAs matters for epistemic reasons. 

 

  1. ^

    Interestingly, among people who merely rated AI risks a near-top priority, attitudes towards digital minds were similar to the sample as a whole. Lower prioritisation of AI risks were associated with yet lower agreement with the digital minds item.

Matthew_Barnett @ 2025-02-13T04:06 (+32) in response to Matthew_Barnett's Quick takes

A reflection on the posts I have written in the last few months, elaborating on my views

In a series of recent posts, I have sought to challenge the conventional view among longtermists that prioritizes the empowerment or preservation of the human species as the chief goal of AI policy. It is my opinion that this view is likely rooted in a bias that automatically favors human beings over artificial entities—thereby sidelining the idea that future AIs might create equal or greater moral value than humans—and treating this alternative perspective with unwarranted skepticism.

I recognize that my position is controversial and likely to remain unpopular among effective altruists for a long time. Nevertheless, I believe it is worth articulating my view at length, as I see it as a straightforward application of standard, common-sense utilitarian principles that merely lead to an unpopular conclusion. I intend to continue elaborating on my arguments in the coming months.

My view follows from a few basic premises. First, that future AI systems are quite likely to be moral patients; second, that we shouldn’t discriminate against them based on arbitrary distinctions, such as their being instantiated on silicon rather than carbon, or having been created through deep learning rather than natural selection. If we insist on treating AIs fundamentally differently from a human child or adult—for example, by regarding them merely as property to be controlled or denying them the freedom to pursue their own goals—then we should identify a specific ethical reason for our approach that goes beyond highlighting their non-human nature.

Many people have argued that consciousness is the key quality separating humans from AIs, thus rendering any AI-based civilization morally insignificant compared to ours. They maintain that consciousness has relatively narrow boundaries, perhaps largely confined to biological organisms, and would only arise in artificial systems under highly specific conditions—for instance, if one were to emulate a human mind in digital form. While I acknowledge that this perspective is logically coherent, I find it deeply unconvincing. The AIs I am referring to when I write about this topic would almost certainly not be simplistic, robotic automatons; rather, they would be profoundly complex, sophisticated entities whose cognitive abilities rival or exceed those of the human brain. For anyone who adopts a functionalist view of consciousness, it seems difficult to be confident that such AIs would lack a rich inner experience.

Because functionalism and preference utilitarianism—both of which could grant moral worth to AI preferences even if they do not precisely replicate biological states—have at least some support within the EA community, I remain hopeful that, if I articulate my position clearly, EAs who share these philosophical assumptions will see its merits.

That said, I am aware that explaining this perspective is an uphill battle. The unpopularity of my views often makes it difficult to communicate without instant misunderstandings; critics seem to frequently conflate my arguments with other, simpler positions that can be more easily dismissed. At times, this has caused me to feel as though the EA community is open to only a narrow range of acceptable ideas. This reaction, while occasionally frustrating, does not surprise me, as I have encountered similar resistance when presenting other unpopular views—such as challenging the ethics of purchasing meat in social contexts where such concerns are quickly deemed absurd.

However, the unpopularity of these ideas also creates a benefit: it creates room for rapid intellectual progress by opening the door to new and interesting philosophical questions about AI ethics. If we free ourselves from the seemingly unquestionable premise that preserving the human species should be the top priority when governing AI development, we can begin to ask entirely new and neglected questions about the role of artificial minds in society.

These questions include: what social and legal frameworks should we pursue if AIs are seen not as dangerous tools to be contained but as individuals on similar moral footing with humans? How do we integrate AI freedom and autonomy into our vision of the future, creating the foundation for a system of ethical and pragmatic AI rights?

Under this alternative philosophical approach, policy would not focus solely on minimizing risks to humanity. Instead, it would emphasize cooperation and inclusion, seeing advanced AI as a partner rather than an ethical menace to be tightly restricted or controlled. This undoubtedly requires a significant shift in our longtermist thinking, demanding a re-examination of deeply rooted assumptions. Such a project cannot be completed overnight, but given the moral stakes and the rapid progress in AI, I view this philosophical endeavor as both urgent and exciting. I invite anyone open to rethinking these foundational premises to join me in exploring how we might foster a future in which AIs and humans coexist as moral peers, cooperating for mutual benefit rather than viewing each other as intrinsic competitors locked in an inevitable zero-sum fight.

David Mathers🔸 @ 2025-02-13T12:10 (+7)

I think for me, part of the issue with your posts on this (which I think are net positive to be clear, they really push at significant weak points in ideas widely held in the community) is that you seem to be sort of vacillating between three different ideas, in a way that conceal that one of them, taken on its own sounds super-crazy and evil:

1) Actually, if AI development were to literally lead to human extinction, that might be fine, because it might lead to higher utility.

2) We should care about humans harming sentient, human-like AIs as much as we care about AIs harming humans.

3) In practice, the benefits to current people from AI development outweigh the risks, and the only moral views which say that we should ignore this and pause in the face of even tiny risks of extinction from AI because there are way more potential humans in the future, in fact, when taken seriously, imply 1), which nobody believes. 

1) feels extremely bad to me, basically a sort of Nazi-style view on which genocide is fine if the replacing people are superior  utility generators (or I guess, inferior but sufficiently more numerous). 1) plausibly is a consequence of classical utilitarianism (even maybe on some person-affecting versions of classical utilitarianism I think), but I take this to be a reason to reject pure classical utilitarianism, not a reason to endorse 1). 2) and 3), on the other hand, seem reasonable to me. But the thing is that you seem at least sometimes to be taking AI moral patienthood as a reason to push on in the face of uncertainty about whether AI will literally kill everyone. And that seems more like 1) than 2) or 3). 1-style reasoning supports the idea that AI moral patienthood is a reason for pushing on with AI development even in the face of human extinction risk, but as far as I can tell 2) and 3) don't. At the same time though I don't think you mean to endorse 1). 

Dan Oblinger @ 2025-02-12T23:48 (+4) in response to How AI Takeover Might Happen in Two Years

Vasco, this can hardly be posed as a fair bet.  If you win, you get paid, if you loose, then money ceases to exist.

Vasco Grilo🔸 @ 2025-02-13T11:02 (+2)

Hi Dan,

Some people may think superintelligent AI as defined by Metaculus does not necessarily imply money ceasing to existing or have value. I am also open to bets where I transfer the money now to the person worried about AI.

Rasool @ 2025-02-11T09:22 (+6) in response to CBA 301: Practical Advice

Google Sheets also has a criminally under appreciated tool - Google App Script

You can write custom code, as Javascript, and have it run on a schedule, or as a function in a spreadsheet cell, or as a clickable menu item

Indrek Kivirik @ 2025-02-13T10:48 (+1)

To add to this, combining this with LLMs is very powerful. If you describe the structure of your sheets and how you want them manipulated to ChatGPT, it will (in my experience) output hundreds of lines of code that will work on the first try.
This has turned me from Just Some Guy into a capable programmer at work, it's crazy.

Carlos Ramírez @ 2025-01-24T03:30 (+1) in response to Open thread: January - March 2025

Those expected value arguments about low probability but very high yield opportunities (moonshots) being more valuable than lower yield, but more certain ones, always rubbed me the wrong way. I suppose because in a very real sense, a 1% odds outcome might as well be 0% odds, specially for something that will only be attempted once, but I was also thinking about the economy. I suspect the overwhelming majority of economic activity is directed at lower risk, lower yield opportunities, and that it is necessary for things to be this way for the economy to function: there is some optimal proportion of the economy that should be dedicated to moonshots of course, but I wonder what that is. And similarly for altruism, there is probably some optimal proportion of altruistic effort that should be directed to moonshots, relative to effort on lower risk, lower yield stuff.

Has anyone written about this, about what would be the best proportion of moonshots to non-moonshots in EA? In the economy? My point is that it's not as simple as saying moonshots are better.

I also recently read someone saying that the worst case with a moonshot is that nothing happens, but that is not true, the moonshot has opportunity cost, all the time, effort, and money spent on it could've been used on something else.

Dylan Richardson @ 2025-02-13T10:44 (+1)

A much of the debate on this topic comes down to questions about risk-aversion and the relevant psychology and decision theory thereof, ex: https://forum.effectivealtruism.org/posts/vnsoy47psQ5KaeHhB/difference-making-risk-aversion-an-exploration
 

Although there are other considerations of course.

OllieBase @ 2025-02-13T10:37 (+2) in response to EA North is happening in Sheffield on Saturday 26 April

This is really cool! Huge props for making this happen :)

Jamie_Harris @ 2024-12-18T19:20 (+4) in response to Ideas EAIF is excited to receive applications for

I don't know all the details since it's a governance/operational thing but I don't think we expect this to be an issue, thankfully!

gergo @ 2025-02-13T10:21 (+1)

Hey! Just wanted to double-check if this is still the case (as it is still on the website), considering that you will be getting a lot of applications for LTFF that would probably want funding for a longer period!

Vasco Grilo🔸 @ 2025-02-12T19:51 (+4) in response to Reaching a New Audience: Insights From a Market Research Survey

Thanks for sharing, Holly!

There was an increase in survey respondents’ self-reported likelihood to support our cause after engaging with our campaign materials, with a 28.6% increase for “very likely” and a 3% increase for “somewhat likely.”

It would be great if you eventually estimated how much campaigns like this increase donations to the organisations recommended by ACE.

Animal Charity Evaluators @ 2025-02-13T10:11 (+4)

Thank you, Vasco! Yes, that's what I will be working toward next and assessing over time. I will be sure to share my findings :) 

Holly

Matthew_Barnett @ 2025-02-13T04:06 (+32) in response to Matthew_Barnett's Quick takes

A reflection on the posts I have written in the last few months, elaborating on my views

In a series of recent posts, I have sought to challenge the conventional view among longtermists that prioritizes the empowerment or preservation of the human species as the chief goal of AI policy. It is my opinion that this view is likely rooted in a bias that automatically favors human beings over artificial entities—thereby sidelining the idea that future AIs might create equal or greater moral value than humans—and treating this alternative perspective with unwarranted skepticism.

I recognize that my position is controversial and likely to remain unpopular among effective altruists for a long time. Nevertheless, I believe it is worth articulating my view at length, as I see it as a straightforward application of standard, common-sense utilitarian principles that merely lead to an unpopular conclusion. I intend to continue elaborating on my arguments in the coming months.

My view follows from a few basic premises. First, that future AI systems are quite likely to be moral patients; second, that we shouldn’t discriminate against them based on arbitrary distinctions, such as their being instantiated on silicon rather than carbon, or having been created through deep learning rather than natural selection. If we insist on treating AIs fundamentally differently from a human child or adult—for example, by regarding them merely as property to be controlled or denying them the freedom to pursue their own goals—then we should identify a specific ethical reason for our approach that goes beyond highlighting their non-human nature.

Many people have argued that consciousness is the key quality separating humans from AIs, thus rendering any AI-based civilization morally insignificant compared to ours. They maintain that consciousness has relatively narrow boundaries, perhaps largely confined to biological organisms, and would only arise in artificial systems under highly specific conditions—for instance, if one were to emulate a human mind in digital form. While I acknowledge that this perspective is logically coherent, I find it deeply unconvincing. The AIs I am referring to when I write about this topic would almost certainly not be simplistic, robotic automatons; rather, they would be profoundly complex, sophisticated entities whose cognitive abilities rival or exceed those of the human brain. For anyone who adopts a functionalist view of consciousness, it seems difficult to be confident that such AIs would lack a rich inner experience.

Because functionalism and preference utilitarianism—both of which could grant moral worth to AI preferences even if they do not precisely replicate biological states—have at least some support within the EA community, I remain hopeful that, if I articulate my position clearly, EAs who share these philosophical assumptions will see its merits.

That said, I am aware that explaining this perspective is an uphill battle. The unpopularity of my views often makes it difficult to communicate without instant misunderstandings; critics seem to frequently conflate my arguments with other, simpler positions that can be more easily dismissed. At times, this has caused me to feel as though the EA community is open to only a narrow range of acceptable ideas. This reaction, while occasionally frustrating, does not surprise me, as I have encountered similar resistance when presenting other unpopular views—such as challenging the ethics of purchasing meat in social contexts where such concerns are quickly deemed absurd.

However, the unpopularity of these ideas also creates a benefit: it creates room for rapid intellectual progress by opening the door to new and interesting philosophical questions about AI ethics. If we free ourselves from the seemingly unquestionable premise that preserving the human species should be the top priority when governing AI development, we can begin to ask entirely new and neglected questions about the role of artificial minds in society.

These questions include: what social and legal frameworks should we pursue if AIs are seen not as dangerous tools to be contained but as individuals on similar moral footing with humans? How do we integrate AI freedom and autonomy into our vision of the future, creating the foundation for a system of ethical and pragmatic AI rights?

Under this alternative philosophical approach, policy would not focus solely on minimizing risks to humanity. Instead, it would emphasize cooperation and inclusion, seeing advanced AI as a partner rather than an ethical menace to be tightly restricted or controlled. This undoubtedly requires a significant shift in our longtermist thinking, demanding a re-examination of deeply rooted assumptions. Such a project cannot be completed overnight, but given the moral stakes and the rapid progress in AI, I view this philosophical endeavor as both urgent and exciting. I invite anyone open to rethinking these foundational premises to join me in exploring how we might foster a future in which AIs and humans coexist as moral peers, cooperating for mutual benefit rather than viewing each other as intrinsic competitors locked in an inevitable zero-sum fight.

Jonas Hallgren @ 2025-02-13T10:03 (+1)

FWIW, I completely agree with what you're saying here and I think that if you seriously go into consciousness research and especially for what we westerners more label as a sense of self rather than anything else it quickly becomes infeasible to hold a position that the way we're taking AI development, e.g towards AI agents will not lead to AIs having self-models. 

For all matters and purposes this encompasses most theories of physicalist or non-dual theories of consciousness which are the only feasible ones unless you want to bite some really sour apples. 

There's a classic "what are we getting wrong" question in EA and I think it's extremely likely that we will look back in 10 years and say, "wow, what are we doing here?". 

I think it's a lot better to think of systemic alignment and look at properties that we want for the general collective intelligences that we're engaging in such as our information networks or our institutional decision making procedures and think of how we can optimise these for resillience and truth-seeking. If certain AIs deserve moral patienthood then that truth will naturally arise from such structures. 

(hot take) Individual AI alignment might honestly be counter-productive towards this view.

James Herbert @ 2025-02-13T09:49 (+2) in response to EA North is happening in Sheffield on Saturday 26 April

Eyy! I grew up in the North East so it's great to see this happen. Love the branding too. 

Rasool @ 2025-02-13T08:45 (+6) in response to Rasool's Quick takes

Screwworm is a flesh-eating maggot!

I skimmed past many posts like this, assuming that it was some kind of stomach worm, or related to the suffering of wild worms (not that I am opposed to either of those, they just don't grab my attention as strongly)

Gemma 🔸 @ 2025-02-13T08:45 (+2) in response to EA North is happening in Sheffield on Saturday 26 April

Great work making this happen! 

Matthew_Barnett @ 2025-02-13T04:06 (+32) in response to Matthew_Barnett's Quick takes

A reflection on the posts I have written in the last few months, elaborating on my views

In a series of recent posts, I have sought to challenge the conventional view among longtermists that prioritizes the empowerment or preservation of the human species as the chief goal of AI policy. It is my opinion that this view is likely rooted in a bias that automatically favors human beings over artificial entities—thereby sidelining the idea that future AIs might create equal or greater moral value than humans—and treating this alternative perspective with unwarranted skepticism.

I recognize that my position is controversial and likely to remain unpopular among effective altruists for a long time. Nevertheless, I believe it is worth articulating my view at length, as I see it as a straightforward application of standard, common-sense utilitarian principles that merely lead to an unpopular conclusion. I intend to continue elaborating on my arguments in the coming months.

My view follows from a few basic premises. First, that future AI systems are quite likely to be moral patients; second, that we shouldn’t discriminate against them based on arbitrary distinctions, such as their being instantiated on silicon rather than carbon, or having been created through deep learning rather than natural selection. If we insist on treating AIs fundamentally differently from a human child or adult—for example, by regarding them merely as property to be controlled or denying them the freedom to pursue their own goals—then we should identify a specific ethical reason for our approach that goes beyond highlighting their non-human nature.

Many people have argued that consciousness is the key quality separating humans from AIs, thus rendering any AI-based civilization morally insignificant compared to ours. They maintain that consciousness has relatively narrow boundaries, perhaps largely confined to biological organisms, and would only arise in artificial systems under highly specific conditions—for instance, if one were to emulate a human mind in digital form. While I acknowledge that this perspective is logically coherent, I find it deeply unconvincing. The AIs I am referring to when I write about this topic would almost certainly not be simplistic, robotic automatons; rather, they would be profoundly complex, sophisticated entities whose cognitive abilities rival or exceed those of the human brain. For anyone who adopts a functionalist view of consciousness, it seems difficult to be confident that such AIs would lack a rich inner experience.

Because functionalism and preference utilitarianism—both of which could grant moral worth to AI preferences even if they do not precisely replicate biological states—have at least some support within the EA community, I remain hopeful that, if I articulate my position clearly, EAs who share these philosophical assumptions will see its merits.

That said, I am aware that explaining this perspective is an uphill battle. The unpopularity of my views often makes it difficult to communicate without instant misunderstandings; critics seem to frequently conflate my arguments with other, simpler positions that can be more easily dismissed. At times, this has caused me to feel as though the EA community is open to only a narrow range of acceptable ideas. This reaction, while occasionally frustrating, does not surprise me, as I have encountered similar resistance when presenting other unpopular views—such as challenging the ethics of purchasing meat in social contexts where such concerns are quickly deemed absurd.

However, the unpopularity of these ideas also creates a benefit: it creates room for rapid intellectual progress by opening the door to new and interesting philosophical questions about AI ethics. If we free ourselves from the seemingly unquestionable premise that preserving the human species should be the top priority when governing AI development, we can begin to ask entirely new and neglected questions about the role of artificial minds in society.

These questions include: what social and legal frameworks should we pursue if AIs are seen not as dangerous tools to be contained but as individuals on similar moral footing with humans? How do we integrate AI freedom and autonomy into our vision of the future, creating the foundation for a system of ethical and pragmatic AI rights?

Under this alternative philosophical approach, policy would not focus solely on minimizing risks to humanity. Instead, it would emphasize cooperation and inclusion, seeing advanced AI as a partner rather than an ethical menace to be tightly restricted or controlled. This undoubtedly requires a significant shift in our longtermist thinking, demanding a re-examination of deeply rooted assumptions. Such a project cannot be completed overnight, but given the moral stakes and the rapid progress in AI, I view this philosophical endeavor as both urgent and exciting. I invite anyone open to rethinking these foundational premises to join me in exploring how we might foster a future in which AIs and humans coexist as moral peers, cooperating for mutual benefit rather than viewing each other as intrinsic competitors locked in an inevitable zero-sum fight.

Ozzie Gooen @ 2025-02-13T05:46 (+4)

I think it's interesting and admiral that you're dedicated on a position that's so unusual in this space.

I assume I'm in the majority here that my intuitions are quite different from yours, however. 

One quick point when we're here:
> this view is likely rooted in a bias that automatically favors human beings over artificial entities—thereby sidelining the idea that future AIs might create equal or greater moral value than humans—and treating this alternative perspective with unwarranted skepticism.

I think that a common, but perhaps not well vocalized, utilitarian take is that humans don't have much of a special significance in terms of creating well-being. The main option would be a much more abstract idea, some kind of generalization of hedonium or consequentialism-ium or similar. For now, let's define hedonium as "the ideal way of converting matter and energy into well-being, after a great deal of deliberation." 

As such, it's very tempting to try to separate concerns and have AI tools focus on being great tools, and separately optimize hedonium to be efficient at being well-being. While I'm not sure if AIs would have zero qualia, I'd feel a lot more confident that they will have dramatically less qualia per unit resources than a much more optimized substrate.

If one follows this general logic, then one might assume that it's likely that the vast majority of well-being in the future would exist as hedonium, not within AIs created to ultimately make hedonium.

One less intense formulation would be to have both AIs and humans focus only on making sure we get to the point where we much better understand the situation with qualia and hedonium (a la the Long Reflection), and then re-evaluate. In my strategic thinking around AI I'm not particularly optimizing for the qualia of the humans involved in the AI labs or the relevant governments. Similarly, I'd expect not to optimize hard for the qualia in the early AIs, in the period when we're unsure about qualia and ethics, even if I thought they might have experiences. I would be nervous if I thought this period could involve AIs having intense suffering or be treated in highly immoral ways. 

Habryka @ 2025-02-13T03:56 (+11) in response to The standard case for delaying AI appears to rest on non-utilitarian assumptions

In the absence of meaningful evidence about the nature of AI civilization, what justification is there for assuming that it will have less moral value than human civilization—other than a speciesist bias?

You know these arguments! You have heard them hundreds of times. Humans care about many things. Sometimes we collapse that into caring about experience for simplicity. 

AIs will probably not care about the same things, as such, the universe will be worse by our lights if controlled by AI civilizations. We don't know what exactly those things are, but the only pointer to our values that we have is ourselves, and AIs will not share those pointers.

Matthew_Barnett @ 2025-02-13T04:34 (+2)

I think your response largely assumes a human-species-centered viewpoint, rather than engaging with my critique that is precisely aimed at re-evaluating this very point of view. 

You say, AIs will probably not care about the same things, so the universe will be worse by our lights if controlled by AI.” But what are "our lights" and "our values" in this context? Are you referring to the values of me as an individual, the current generation of humans, or humanity as a broad, ongoing species-category? These are distinct—and often conflicting—sets of values, preferences, and priorities. It’s possible, indeed probable, that I, personally, have preferences that differ fundamentally from the majority of humans. "My values" are not the same as "our values".

When you talk about whether an AI civilization is “better” or “worse,” it’s crucial to clarify what perspective we’re measuring that from. If, from the outset, we assume that human values, or the survival of humanity-as-a-species, is the critical factor that determines whether an AI civilization is better or worse than our own, that effectively begs the question. It merely assumes what I aim to challenge. From a more impartial standpoint, the mere fact that AI might not care about the exact same things humans do doesn’t necessarily entail a decrease in total impartial moral value—unless we’ve already decided in advance that human values are inherently more important. 

(To make this point clearer, perhaps replace all mentions of "human values" with "North American values" in the standard arguments about these issues, and see if it makes these arguments sound like they privilege an arbitrary category of beings.)

While it’s valid to personally value the continuation of the human species, or the preservation of human values, as a moral preference above other priorities, my point is simply that that’s precisely the species-centric assumption I’m highlighting, rather than a distinct argument that undermines my observations or analysis. Such a perspective is not substrate or species-neutral. Nor is it obviously mandated by a strictly utilitarian framework; it’s an extra premise that privileges the category "humankind" for its own sake. You may believe that such a preference is natural or good from your own perspective, but that is not equivalent to saying that it is the preference of an impartial utilitarian, who would, in theory, make no inherent distinction based purely on species, or substrate.

Matthew_Barnett @ 2025-02-13T04:06 (+32) in response to Matthew_Barnett's Quick takes

A reflection on the posts I have written in the last few months, elaborating on my views

In a series of recent posts, I have sought to challenge the conventional view among longtermists that prioritizes the empowerment or preservation of the human species as the chief goal of AI policy. It is my opinion that this view is likely rooted in a bias that automatically favors human beings over artificial entities—thereby sidelining the idea that future AIs might create equal or greater moral value than humans—and treating this alternative perspective with unwarranted skepticism.

I recognize that my position is controversial and likely to remain unpopular among effective altruists for a long time. Nevertheless, I believe it is worth articulating my view at length, as I see it as a straightforward application of standard, common-sense utilitarian principles that merely lead to an unpopular conclusion. I intend to continue elaborating on my arguments in the coming months.

My view follows from a few basic premises. First, that future AI systems are quite likely to be moral patients; second, that we shouldn’t discriminate against them based on arbitrary distinctions, such as their being instantiated on silicon rather than carbon, or having been created through deep learning rather than natural selection. If we insist on treating AIs fundamentally differently from a human child or adult—for example, by regarding them merely as property to be controlled or denying them the freedom to pursue their own goals—then we should identify a specific ethical reason for our approach that goes beyond highlighting their non-human nature.

Many people have argued that consciousness is the key quality separating humans from AIs, thus rendering any AI-based civilization morally insignificant compared to ours. They maintain that consciousness has relatively narrow boundaries, perhaps largely confined to biological organisms, and would only arise in artificial systems under highly specific conditions—for instance, if one were to emulate a human mind in digital form. While I acknowledge that this perspective is logically coherent, I find it deeply unconvincing. The AIs I am referring to when I write about this topic would almost certainly not be simplistic, robotic automatons; rather, they would be profoundly complex, sophisticated entities whose cognitive abilities rival or exceed those of the human brain. For anyone who adopts a functionalist view of consciousness, it seems difficult to be confident that such AIs would lack a rich inner experience.

Because functionalism and preference utilitarianism—both of which could grant moral worth to AI preferences even if they do not precisely replicate biological states—have at least some support within the EA community, I remain hopeful that, if I articulate my position clearly, EAs who share these philosophical assumptions will see its merits.

That said, I am aware that explaining this perspective is an uphill battle. The unpopularity of my views often makes it difficult to communicate without instant misunderstandings; critics seem to frequently conflate my arguments with other, simpler positions that can be more easily dismissed. At times, this has caused me to feel as though the EA community is open to only a narrow range of acceptable ideas. This reaction, while occasionally frustrating, does not surprise me, as I have encountered similar resistance when presenting other unpopular views—such as challenging the ethics of purchasing meat in social contexts where such concerns are quickly deemed absurd.

However, the unpopularity of these ideas also creates a benefit: it creates room for rapid intellectual progress by opening the door to new and interesting philosophical questions about AI ethics. If we free ourselves from the seemingly unquestionable premise that preserving the human species should be the top priority when governing AI development, we can begin to ask entirely new and neglected questions about the role of artificial minds in society.

These questions include: what social and legal frameworks should we pursue if AIs are seen not as dangerous tools to be contained but as individuals on similar moral footing with humans? How do we integrate AI freedom and autonomy into our vision of the future, creating the foundation for a system of ethical and pragmatic AI rights?

Under this alternative philosophical approach, policy would not focus solely on minimizing risks to humanity. Instead, it would emphasize cooperation and inclusion, seeing advanced AI as a partner rather than an ethical menace to be tightly restricted or controlled. This undoubtedly requires a significant shift in our longtermist thinking, demanding a re-examination of deeply rooted assumptions. Such a project cannot be completed overnight, but given the moral stakes and the rapid progress in AI, I view this philosophical endeavor as both urgent and exciting. I invite anyone open to rethinking these foundational premises to join me in exploring how we might foster a future in which AIs and humans coexist as moral peers, cooperating for mutual benefit rather than viewing each other as intrinsic competitors locked in an inevitable zero-sum fight.

Matthew_Barnett @ 2025-02-11T22:55 (+2) in response to The standard case for delaying AI appears to rest on non-utilitarian assumptions

In the absence of meaningful evidence about the nature of AI civilization, what justification is there for assuming that it will have less moral value than human civilization—other than a speciesist bias? While I agree that there is great uncertainty, your argument appears to be entirely symmetric. AI civilization could turn out to be far more morally valuable than human civilization, or it could be far less valuable, from a utilitarian perspective. Both possibilities seem plausible, since we have little information either way. Given this vast uncertainty, there is no clear reason to default to one assumption over the other. In such a situation, the best response is not to commit to a particular stance as the default, but rather to suspend judgment until stronger evidence emerges.

I personally am happy to bite the bullet and say that I morally value human civilization continuing over an AI civilization that killed all of humanity, and that this is a significant term in my utility function.

I'm glad to see you explicitly acknowledge that you accept the implications regarding the value of human civilization. However, your statement here is a bit ambiguous—when considering what to prioritize, do you place greater value on the survival of the human species as a whole or on the well-being and preservation of the humans who are currently alive? Personally, my intuitions lean more strongly toward prioritizing the latter.

Habryka @ 2025-02-13T03:56 (+11)

In the absence of meaningful evidence about the nature of AI civilization, what justification is there for assuming that it will have less moral value than human civilization—other than a speciesist bias?

You know these arguments! You have heard them hundreds of times. Humans care about many things. Sometimes we collapse that into caring about experience for simplicity. 

AIs will probably not care about the same things, as such, the universe will be worse by our lights if controlled by AI civilizations. We don't know what exactly those things are, but the only pointer to our values that we have is ourselves, and AIs will not share those pointers.

Karthik Tadepalli @ 2025-02-13T01:56 (+4) in response to The ambiguous effect of full automation + new goods on GDP growth

I've been thinking about this post for days, which is a great sign, and in particular I think there's a deep truth in the following:

Indeed, my guess is that people’s utility in the goods available today does have an upper asymptote, that new goods in the future could raise our utility above that bound, and that this cycle has been played out many times already.

I realize this is tangential to your point about GDP measurement, but I think Uzawa's theorem probably set growth theory back by decades. By axiomatizing that technical change is labor-augmenting, we became unable to speak coherently about automation, something that is only changing recently. I think there is so much more we can understand about technical change that we don't yet. My best guess of the nature of technological progress is as follows:

  1. In the long run, capital and labor are gross substitutes, and basically all technological change in existing goods is capital-augmenting (-> labor-replacing by the gross substitutes assumption).
  2. However, we constantly create new goods that have a high labor share of costs (e.g. the services transition). These goods keep increasing as a share of the economy and cause an increase in wages.

This idea is given some empirical support by Hubmer 2022 and theoretical clarity by Jones and Liu 2024, but it's still just a conjecture. So I think the really important question about AI is whether the tons of new products it will enable will themselves be labor-intensive or capital-intensive. If the new products are capital-intensive, breaking with historical trend, then I expect that the phenomenon you describe (good 2's productivity doesn't grow) will not happen.

trammell @ 2025-02-13T03:04 (+2)

Great to hear, thanks!

As for the prediction—fair enough. Just to clarify though, I’m worried that the example makes it look like we need growth in the new good(s) to get this weird slow GDP growth result, but that’s not true. In case that’s the impression you got, this example illustrates how we can have superexponential growth in every good but (arbitrarily slow) exponential growth in GDP.

Dan_Keys @ 2025-02-13T02:48 (+4) in response to The standard case for delaying AI appears to rest on non-utilitarian assumptions

Nate Soares' take here was that an AI takeover would most likely lead to an "unconscious meh" scenario, where "The outcome is worse than the “Pretty Good” scenario, but isn’t worse than an empty universe-shard" and "there’s little or no conscious experience in our universe-shard’s future. E.g., our universe-shard is tiled with tiny molecular squiggles (a.k.a. “molecular paperclips”)." Whereas humanity boosted by ASI would probably lead to a better outcome.

That was also the most common view in the polls in the comments there.

Karthik Tadepalli @ 2025-02-13T01:56 (+4) in response to The ambiguous effect of full automation + new goods on GDP growth

I've been thinking about this post for days, which is a great sign, and in particular I think there's a deep truth in the following:

Indeed, my guess is that people’s utility in the goods available today does have an upper asymptote, that new goods in the future could raise our utility above that bound, and that this cycle has been played out many times already.

I realize this is tangential to your point about GDP measurement, but I think Uzawa's theorem probably set growth theory back by decades. By axiomatizing that technical change is labor-augmenting, we became unable to speak coherently about automation, something that is only changing recently. I think there is so much more we can understand about technical change that we don't yet. My best guess of the nature of technological progress is as follows:

  1. In the long run, capital and labor are gross substitutes, and basically all technological change in existing goods is capital-augmenting (-> labor-replacing by the gross substitutes assumption).
  2. However, we constantly create new goods that have a high labor share of costs (e.g. the services transition). These goods keep increasing as a share of the economy and cause an increase in wages.

This idea is given some empirical support by Hubmer 2022 and theoretical clarity by Jones and Liu 2024, but it's still just a conjecture. So I think the really important question about AI is whether the tons of new products it will enable will themselves be labor-intensive or capital-intensive. If the new products are capital-intensive, breaking with historical trend, then I expect that the phenomenon you describe (good 2's productivity doesn't grow) will not happen.

Jess_Riedel @ 2025-02-12T23:11 (+1) in response to The ambiguous effect of full automation + new goods on GDP growth

Oh yea, I didn't mind the title at all (although I do think it's usefully more precise now :)

Agreed on additively separable utility being unrealistic. My point (which wasn't clearly spelled out) was not that GDP growth and unit production can't look dramatically. (We already see that in individual products like transistors (>> GDP) and rain dances (<< GDP).) It was that post-full-automation isn't crucially different than pre-full-automation unless you make some imo pretty extreme assumptions to distinguish them.

By "extracting this from our utility function", I just mean my vague claim that, insofar as we are uncertain about GDP growth post-full-automation, understanding better the sorts of things people and superhuman intelligences want will reduce that uncertainty more than learning about the non-extreme features of future productivity heterogeneity (although both do matter if extreme enough). But I'm being so vague here that it's hard to argue against.

trammell @ 2025-02-13T01:08 (+2)

Here's an example in which utility is additively separable,  is identical for all goods, the productivity and quantity of all goods grow hyperbolically, and yet GDP grows exponentially.

Michael Hinge @ 2023-09-09T09:05 (+1) in response to Nuclear winter - Reviewing the evidence, the complexities, and my conclusions

Hi Ulrik,

I would agree with you there in large part, but I don't think that should necessarily reduce our estimate of the impact away from what I estimated above. 

For example, the Los Alamos team did far more detailed fire modelling vs Rutgers, but the end result is a model that seems to be unable to replicate real fire conditions in situations like Hiroshima, Dresden and Hamberg -> more detailed modeling isn't in itself a guarantee of accuracy.

However, the models we have are basing their estimates at least in part on empirical observations, which potentially give us enough cause for concern:

-Soot can be lofted in firestorm plumes, for example at Hiroshima.

-Materials like SO2 in the atmosphere from volcanoes can be observed to disrupt the climate, and there is no reason to expect that this is different for soot.

-Materials in the atmosphere can persist for years, though the impact takes time to arrive due to inertia and will diminish over time.

The complexities of modeling you highlight raise the uncertainties with everything above, but they do not disprove nuclear winter. The complexities also seem raise more uncertainty for Los Alamos and the more skeptic side, who rely heavily on modeling, than Rutgers, who use modeling only where they cannot use an empirical heuristic like the conditions of past firestorms.

Gentzel @ 2025-02-13T00:11 (+1)

FWIW, Los Alamos claims they replicated Hiroshima and the Berkeley Hills Fire Smoke Plumes with their fire models to within 1 km of plume height. It's pretty far into the presentation though, and most of their sessions are not public, so I can hardly blame anyone for not encountering this. 



Comments on 2025-02-12

Constance Li @ 2025-02-12T23:51 (+7) in response to Vegetarian or not..?

I suspect that the majority of positive impact from vegan/vegetarian diets comes from the normalization of these practices in different communities, not from the economic effects. I don't have any data to back this up other than anecdotes of people telling me that I influenced them just by pointing out animal harm in their food choices... which I guess is different from me just being vegan, but I think me being vegan adds to the impact of my words.

Vasco Grilo🔸 @ 2025-02-08T07:24 (+2) in response to How AI Takeover Might Happen in Two Years

Hi Joshua,

I would be happy to bet 10 k$ against short AI timelines. Note I am open to a later resolution date than the one I mention in the linked post, such that the bet is beneficial for you despite a higher risk of you not receiving the transfer in case you win.

Your post did not have any tags. I added a few.

Dan Oblinger @ 2025-02-12T23:48 (+4)

Vasco, this can hardly be posed as a fair bet.  If you win, you get paid, if you loose, then money ceases to exist.

Jess_Riedel @ 2025-02-12T23:11 (+1) in response to The ambiguous effect of full automation + new goods on GDP growth

Oh yea, I didn't mind the title at all (although I do think it's usefully more precise now :)

Agreed on additively separable utility being unrealistic. My point (which wasn't clearly spelled out) was not that GDP growth and unit production can't look dramatically. (We already see that in individual products like transistors (>> GDP) and rain dances (<< GDP).) It was that post-full-automation isn't crucially different than pre-full-automation unless you make some imo pretty extreme assumptions to distinguish them.

By "extracting this from our utility function", I just mean my vague claim that, insofar as we are uncertain about GDP growth post-full-automation, understanding better the sorts of things people and superhuman intelligences want will reduce that uncertainty more than learning about the non-extreme features of future productivity heterogeneity (although both do matter if extreme enough). But I'm being so vague here that it's hard to argue against.

trammell @ 2025-02-12T23:20 (+3)

Ok, fair enough--thanks for getting me to make it clearer :). So I guess the disagreement (if any remains, post-retitling/etc) is just about how plausible we think it is that the technological advances that accompany full automation will be accompanied by further technological advances that counterintuitively slow GDP growth through the "new-products-Baumol" mechanism illustrated here. I don't think that's so implausible, and hopefully the note I'll write later will make it clearer where I'm coming from on that.

But this post isn't aiming to argue for the plausibility, just the possibility. It seems to me that a lot of discussion of this issue hasn't noticed that it's even a theoretical possibility.

trammell @ 2025-02-12T22:39 (+3) in response to The ambiguous effect of full automation + new goods on GDP growth

not being on track to produce Good 2 only happens in your model specifically because you define automation to be a thing that takes Good-2 productivity from 0 to something positive... Automation is usually understood to be something that increases the productivity of something that we could already produce at least a little of in principle

Okay, I'm happy to change the title to (a more concise version of) "the ambiguous effect of a technological advancement that achieves full automation, and also allows new goods to be introduced on GDP growth" if that would resolve the disagreement. [Update: have just changed the title and a few words of the body text; let me know.]

On the second point: in practice I don't think we have additively separable utility, and I don't know what you mean by "extracting this from our utility function". But anyway, if I'm understanding you, that is wrong: if your utility function is additively separable with an upper bound in each good, say , a technological shift can yield superexponential growth in the quantity of each n but exponential GDP growth. I'll write up a note on how that works this evening if that would be helpful, but I was hoping this post could just be a maximally simple illustration of the more limited point that Baumol-like effects can slow growth even past the point of full automation.

Jess_Riedel @ 2025-02-12T23:11 (+1)

Oh yea, I didn't mind the title at all (although I do think it's usefully more precise now :)

Agreed on additively separable utility being unrealistic. My point (which wasn't clearly spelled out) was not that GDP growth and unit production can't look dramatically. (We already see that in individual products like transistors (>> GDP) and rain dances (<< GDP).) It was that post-full-automation isn't crucially different than pre-full-automation unless you make some imo pretty extreme assumptions to distinguish them.

By "extracting this from our utility function", I just mean my vague claim that, insofar as we are uncertain about GDP growth post-full-automation, understanding better the sorts of things people and superhuman intelligences want will reduce that uncertainty more than learning about the non-extreme features of future productivity heterogeneity (although both do matter if extreme enough). But I'm being so vague here that it's hard to argue against.

SAB25 @ 2025-02-09T05:17 (+3) in response to The Game Board has been Flipped: Now is a good time to rethink what you’re doing

One implication I strongly disagree with is that people should be getting jobs in AI labs. I don’t see you connecting that to actual safety impact, and I sincerely doubt working as a researcher gives you any influence on safety at this point (if it ever did). 

Wouldn't it allow access to the most advanced models which are not yet publicly available? I think these are the models that would pose the most risk?
 

Working at a frontier lab would also give the opportunity to reach people far less concerned about safety, and maybe their minds could be changed? 

Holly Elmore ⏸️ 🔸 @ 2025-02-12T23:08 (+3)

See my other comments. "Access" to do what? At what cost?

Jess_Riedel @ 2025-02-12T22:20 (+1) in response to The ambiguous effect of full automation + new goods on GDP growth

Thanks!

My point is that the existence of a human-only good satisfying (1) and (2) is unnecessary: the very same effect can arise even given true full automation, not due to a limitation of our ability to fully automate, but due to the fact that a technological advance can encompass full automation and go beyond yielding a world where we can produce way more of everything we would ever produce without the advance, by letting us produce some goods we otherwise wouldn't have been on track to produce at all. This has not been widely appreciated.

OK but this key feature of not being on track to produce Good 2 only happens in your model specifically because you define automation to be a thing that takes Good-2 productivity from 0 to something positive.  I think this is in conflict with the normal understanding of what "automation" means! Automation is usually understood to be something that increases the productivity of something that we could already produce at least a little of in principle, even if the additional efficiency means actual spending on a specific product goes from 0 to 1. And as long as we could produce a little of Good 2 pre-automation, the utility function in your model implies that the spending in the economy would eventually be dominated by Good 2 (and hence GDP growth rates would be set by the growth in productivity of Good 2) even without full automation (unless the ratio of Good-1 and Good-2 productivity is growing superexponentially in time).

What kind product would we be unable to produce without full automation, even given arbitrary time to grow? Off the top of my head I can only think of something really ad-hoc like "artisanal human paintings depicting the real-world otherwise-fully-autonomous economy".

That's basically what makes me think that "the answer is already in our utility function", which we could productively introspect on, rather than some empirical uncertainty about what products full automation will introduce.

presumably full automation will coincide with not only a big increase in productivity growth (which raises GDP growth, in the absence of a random "utility function kink") but also a big change in the direction of productivity growth, including via making new products available (which introduces the kind of "utility function kink" that has an arbitrary effect on GDP growth).

I'm not sure what the best precise math statement to make here is, but I suspect that at least for "separable" utility functions of the form  you need either a dramatic difference in diminishing returns for the  (e.g., log vs. linear as in your model) or you need a super dramatic difference in the post-full-automation productivity growth curves (e.g., one grows exponentially and the other grows superexponentially) that is absent pre-automation. (I don't think it's enough that the productivities grow at different rates post-automation.) So I still think we can extract this from our utility function without knowing much about the future, although maybe there's a concrete model that would show that's wrong.

trammell @ 2025-02-12T22:39 (+3)

not being on track to produce Good 2 only happens in your model specifically because you define automation to be a thing that takes Good-2 productivity from 0 to something positive... Automation is usually understood to be something that increases the productivity of something that we could already produce at least a little of in principle

Okay, I'm happy to change the title to (a more concise version of) "the ambiguous effect of a technological advancement that achieves full automation, and also allows new goods to be introduced on GDP growth" if that would resolve the disagreement. [Update: have just changed the title and a few words of the body text; let me know.]

On the second point: in practice I don't think we have additively separable utility, and I don't know what you mean by "extracting this from our utility function". But anyway, if I'm understanding you, that is wrong: if your utility function is additively separable with an upper bound in each good, say , a technological shift can yield superexponential growth in the quantity of each n but exponential GDP growth. I'll write up a note on how that works this evening if that would be helpful, but I was hoping this post could just be a maximally simple illustration of the more limited point that Baumol-like effects can slow growth even past the point of full automation.

Yonatan Cale @ 2025-02-12T22:39 (+2) in response to Job Post Template

Guided Track used this template

cc Richard Möhn

trammell @ 2025-02-12T18:54 (+5) in response to The ambiguous effect of full automation + new goods on GDP growth

Hey Jess, thanks for the thoughtful comments.

On whether "this is still basically Baumol"

If we make that one tiny tweak and say that good 2 was out there to be made all along, just too expensive to be demanded, then yes, it's the same! That was the goal of the example: to introduce a Baumol-like effect in a way so similar to how everyone agrees Baumol effects have played out historically that it's really easy to see what's going on.

I'm happy to say it's basically Baumol. The distinction I think is worth emphasizing here is that, when people say "Baumol effects could slow down the growth effects of AI", they are usually--I think always, in my experience--pointing to the fact that if

  1. AI dramatically speeds productivity growth on most goods but leaves some that only humans can produce (say, artisanal handicrafts), and
  2. consumers see those goods as not very substitutable for the goods we're getting way more productive at,

then GDP growth won't speed up much. This then invites the response that, when we look around, there doesn't seem to be any human-only good strongly satisfying (1) and (2). My point is that the existence of a human-only good satisfying (1) and (2) is unnecessary: the very same effect can arise even given true full automation, not due to a limitation of our ability to fully automate, but due to the fact that a technological advance can encompass full automation and go beyond yielding a world where we can produce way more of everything we would ever produce without the advance, by letting us produce some goods we otherwise wouldn't have been on track to produce at all. This has not been widely appreciated.

On whether there is any reason to expect the productivity acceleration to coincide with the "kink in the utility function"

Here I think I disagree with you more substantively, though maybe the disagreement stems from the small framing point above.

If indeed "good 2" were always out there, just waiting for its price to fall, and if a technology were coming that would just replace all our existing workers, factory parts, and innovators with versions that operate more quickly in equal proportion--so that we move along the same paths of technology and the quantity of each good produced, but more quickly--then I agree that the past would be a good guide to the future, and GDP growth would reliably rise a lot. The only way it wouldn't would be if the goods we were just about to start producing anyway were like good 2, featuring less steeply diminishing marginal utility but way slower productivity growth, so that the rise in productivity growth across the board coincidentally turned up at the same time as the "utility function kink".

But if the technological advances that are allowing us to automate the production of everything people would ever be able to produce without the advances are also what allow for the invention of goods like good 2, it wouldn't be a coincidence. I.e. presumably full automation will coincide with not only a big increase in productivity growth (which raises GDP growth, in the absence of a random "utility function kink") but also a big change in the direction of productivity growth, including via making new products available (which introduces the kind of "utility function kink" that has an arbitrary effect on GDP growth). The idea that we're soon producing very different products than we otherwise ever would have, whose productivity is growing at very different rates, seems all the more likely to me when we remember that even at 30% growth we're soon in an economy several orders of magnitude bigger: the kink just needs to show up somewhere, not anywhere near the current margin.

To reiterate what I noted at the beginning though, I'd be surprised if the ambiguous second effect single-handedly outweighed the unambiguously positive first effect. And it could just as well amplify it, if "good 2" exhibits faster than average productivity growth.

Jess_Riedel @ 2025-02-12T22:20 (+1)

Thanks!

My point is that the existence of a human-only good satisfying (1) and (2) is unnecessary: the very same effect can arise even given true full automation, not due to a limitation of our ability to fully automate, but due to the fact that a technological advance can encompass full automation and go beyond yielding a world where we can produce way more of everything we would ever produce without the advance, by letting us produce some goods we otherwise wouldn't have been on track to produce at all. This has not been widely appreciated.

OK but this key feature of not being on track to produce Good 2 only happens in your model specifically because you define automation to be a thing that takes Good-2 productivity from 0 to something positive.  I think this is in conflict with the normal understanding of what "automation" means! Automation is usually understood to be something that increases the productivity of something that we could already produce at least a little of in principle, even if the additional efficiency means actual spending on a specific product goes from 0 to 1. And as long as we could produce a little of Good 2 pre-automation, the utility function in your model implies that the spending in the economy would eventually be dominated by Good 2 (and hence GDP growth rates would be set by the growth in productivity of Good 2) even without full automation (unless the ratio of Good-1 and Good-2 productivity is growing superexponentially in time).

What kind product would we be unable to produce without full automation, even given arbitrary time to grow? Off the top of my head I can only think of something really ad-hoc like "artisanal human paintings depicting the real-world otherwise-fully-autonomous economy".

That's basically what makes me think that "the answer is already in our utility function", which we could productively introspect on, rather than some empirical uncertainty about what products full automation will introduce.

presumably full automation will coincide with not only a big increase in productivity growth (which raises GDP growth, in the absence of a random "utility function kink") but also a big change in the direction of productivity growth, including via making new products available (which introduces the kind of "utility function kink" that has an arbitrary effect on GDP growth).

I'm not sure what the best precise math statement to make here is, but I suspect that at least for "separable" utility functions of the form  you need either a dramatic difference in diminishing returns for the  (e.g., log vs. linear as in your model) or you need a super dramatic difference in the post-full-automation productivity growth curves (e.g., one grows exponentially and the other grows superexponentially) that is absent pre-automation. (I don't think it's enough that the productivities grow at different rates post-automation.) So I still think we can extract this from our utility function without knowing much about the future, although maybe there's a concrete model that would show that's wrong.

Vasco Grilo🔸 @ 2025-02-12T20:54 (+2) in response to Rethink Priorities’ Welfare Range Estimates

I noted Table 1 of the doc does not have the probability of sentience of shrimp, although I guess it is similar to that of crayfish, 45.3 %.

Richard Bruns @ 2025-02-11T22:11 (+1) in response to Mortality Cost of Taxation

I had hoped that it would be obvious that I was referring to the gross cost, not the net, especially after my post on Life-year Variability. Taxation does not automatically generate good things. Often it is wasted. It is an odd bias, or a failure of decoupling, to assume that 'taxation' always and automatically means 'tax and spend it effectively on good things'.

If a developed government spends a billion dollars on something, we should train ourselves to think 'You just killed about 100 people to do that. Was it worth it?' Public health and foreign aid usually is. Other things, not so much.

It is not about the deadweight loss, that is an entirely separate issue. Even taxation that caused zero deadweight loss would kill people.

David T @ 2025-02-12T20:47 (+2)

Tax isn't "wasted" by making money vanish from the economy though (except for the deadweight loss) it's just redistributed to other people via payouts, jobs, loans or indirectly via goods purchases. Statistically, some of these beneficiaries will enjoy longer lives through the same indirect income-mortality relationship you invoke to associate taxes with death. This is true even of public spending which is - relative to others - extremely wasteful and not evaluated as lifesaving even by its proponents.[1] 

Which is why I'd argue it makes far more sense to focus cost-benefit analysis on deadweight losses and [counterfactual alternative] uses of public funds. Because regardless of whether the tax is focused on creating "good things" or not, the net result of the transfer probably isn't killing people..

In a developed country with a progressive tax system, the demographics paying most of the tax are unlikely to typically need the income to survive more than state employees or other [indirect] recipients the resulting public expenditure benefits, even for ridiculous ideas like paying millions of dancers to create synchronised tributes to the president. So ignoring extremely indirect and difficult to quantify transfer effects (or explicitly treating them as netting out to zero) in favour of focusing on direct effects and deadweight loss in cost-benefit analysis probably if anything is biased against tax and spend. Empirically, tax burden is positively correlated with longevity, even amongst US states.

  1. ^

    paying superfluous bureaucrats may be an inefficient way of saving lives, but in exactly the same way as taxing people is a very inefficient way of killing, especially where the tax is progressive above affordability thresholds and targeted benefits/rebates exist

Kaspar Brandner @ 2025-02-12T20:11 (+1) in response to The standard case for delaying AI appears to rest on non-utilitarian assumptions

This argument appears very similar to the one I addressed in the essay about how delaying or accelerating AI will impact the well-being of currently existing humans. My claim is not that it isn't bad if humanity goes extinct; I am certainly not saying that it would be good if everyone died.

I'm not supposing you do. Of course most people have a strong preference not to die. But there is also (beyond that) a widespread preference for humanity not to go extinct. This is why it e.g. would be so depressing (as in the movie Children of Men) when a global virus made all humans infertile. Ending humanity is very different from and much worse than people merely dying at the end of their lives, which by itself doesn't imply extinction. Many people would likely even sacrifice their own life in order to safe the future of humanity. We don't have a similar preference for having AI descendants. That's not speciesist, it's just what our preferences are.

Matthew_Barnett @ 2025-02-12T20:46 (+2)

We can assess the strength of people's preferences for future generations by analyzing their economic behavior. The key idea is that if people genuinely cared deeply about future generations, they would prioritize saving a huge portion of their income for the benefit of those future individuals rather than spending it on themselves in the present. This would indicate a strong intertemporal preference for improving the lives of future people over the well-being of currently existing individuals.

For instance, if people truly valued humanity as a whole far more than their own personal well-being, we would expect parents to allocate the vast majority of their income to their descendants (or humanity collectively) rather than using it for their own immediate needs and desires. However, empirical studies generally do not support the claim that people place far greater importance on the long-term preservation of humanity than on the well-being of currently existing individuals. In reality, most people tend to prioritize themselves and their children, while allocating only a relatively small portion of their income to charitable causes or savings intended to benefit future generations beyond their immediate children. If people were intrinsically and strongly committed to the abstract concept of humanity itself, rather than primarily concerned with the welfare of present individuals (including their immediate family and friends), we would expect to see much higher levels of long-term financial sacrifice for future generations than we actually observe.

To be clear, I'm not claiming that people don’t value their descendants, or the concept of humanity at all. Rather, my point is that this preference does not appear to be strong enough to override the considerations outlined in my previous argument. While I agree that people do have an independent preference for preserving humanity—beyond just their personal desire to avoid death—this preference is typically not way stronger than their own desire for self-preservation. As a result, my previous conclusion still holds: from the perspective of present-day individuals, accelerating AI development can still be easily justified if one does not believe in a high probability of human extinction from AI.

Milan Griffes @ 2025-01-28T23:15 (+2) in response to What are we doing about the EA Forum? (Jan 2025)

i was referring to the big growth in MAUs that started before July 2022 and peaked before January 2023.

AnonymousEAForumAccount @ 2025-02-12T20:29 (+2)

I think this is likely due to the huge amount of publicity that surrounded the launch of What We Owe the Future feeding into a peak associated with the height of the FTX drama (MAU peaked in November 2022), which has then been followed by over two years of ~steady decline (presumably due to fallout from FTX). Note that the "steady and sizeable decline since FTX bankruptcy" pattern is also evident in EA Funds metrics

Joseph Sarvary @ 2025-02-10T13:49 (+1) in response to Request for Feedback: Gamified Daily Giving App (‘Dollar a Day’)

Thank you very much for the super helpful feedback. I love the idea of giving donation credit as a gift. I've felt the same conundrum when donating on behalf of others where you're ultimately choosing for them where the money goes. Giving them the money then letting them choose the destination is a great fix. 

Matching funds is also something I'd love to incorporate though the way I thought of it could be a corporate or larger donor could sponsor a challenge like when you hit 30 days of consistent giving your next five donations could be matched. Or to drive donations to a specific place: if 100 individuals give to this specific charity today then the sponsor will match the amount. 

There's loads of potential to find ways to motivate others through 'bonus' donations. I would want to target people to support this program at a larger level who might be able to back the 30 days of consistent giving unlocks $20 of extra money to be directed by the user. 

 

Would be great to continue brainstorming. 

 

I also feel that the app would benefit from being able to educate the users about the three prompted causes each day. That way the giving is less robotic and more investigative. People who perhaps had not been exposed to FGM or the impact of malaria or climate change adaptation have a resource to expand their connection to philanthropy 

Martin Jacobson 🔸 @ 2025-02-12T20:26 (+2)

I am happy to continue brainstorming! Here are a handful of quick ideas: 

  • I really liked the idea of getting a donation-matching award when reaching milestones!
  • Perhaps you could add this project itself as one of the donation options? These funds would be used for operation costs and milestone awards. Of course, it is a bit speculative how effective it will be, but once it has been in use for some time you might be able to estimate its multiplier. It also sounds like a good idea to seek some seed grants from a larger funder.
  • I know that Clearer Thinking has done some research on habit formation and daily rituals that might be useful in designing the app. They often discuss and promote EA ideas, so it could be a good idea to reach out to them directly for advice! 
  • I also like the idea mentioned by Charlotte in another comment of building virtual rewards, similar to 'Forest'. This could help people stay motivated over time, but could also be really useful for dramatically visualizing the differences in scope, and incentivize users to aim for more impact. If you are building your virtual chicken farm, you will be able to clearly see the difference between a charity that is 10x as effective as another, and be properly motivated to maximize the number of chickens per donation. 

    Also, feel free to reach out to me by direct message if you want to bounce ideas. This is not my area of expertise, but I am happy to help if there is something I can do!

Matthew_Barnett @ 2025-02-12T18:47 (+2) in response to The standard case for delaying AI appears to rest on non-utilitarian assumptions

Preferences just are what they are, and existing humans clearly have a strong and overwhelming-majority preference for humanity to continue to exist in the future. [...] So the extinction of humanity is bad because we don't want humanity to go extinct.

This argument appears very similar to the one I addressed in the essay about how delaying or accelerating AI will impact the well-being of currently existing humans. My claim is not that it isn't bad if humanity goes extinct; I am certainly not saying that it would be good if everyone died. Rather, my claim is that, if your reason for caring about human extinction arises from a concern for the preferences of the existing generation of humans, then you should likely push for accelerating AI so long as the probability of human extinction from AI is fairly low.

I'll quote the full argument below:

Of course, one can still think—as I do—that human extinction would be a terrible outcome for the people who are alive when it occurs. Even if the AIs that replace us are just as morally valuable as we are from an impartial moral perspective, it would still be a moral disaster for all currently existing humans to die. However, if we accept this perspective, then we must also acknowledge that, from the standpoint of people living today, there appear to be compelling reasons to accelerate AI development rather than delay it for safety reasons.

The reasoning is straightforward: if AI becomes advanced enough to pose an existential threat to humanity, then it would almost certainly also be powerful enough to enable massive technological progress—potentially revolutionizing medicine, biotechnology, and other fields in ways that could drastically improve and extend human lives. For example, advanced AI could help develop cures for aging, eliminate extreme suffering, and significantly enhance human health through medical and biological interventions. These advancements could allow many people who are alive today to live much longer, healthier, and more fulfilling lives.

As economist Chad Jones has pointed out, delaying AI development means that the current generation of humans risks missing out on these transformative benefits. If AI is delayed for years or decades, a large fraction of people alive today—including those advocating for AI safety—would not live long enough to experience these life-extending technologies. This leads to a strong argument for accelerating AI, at least from the perspective of present-day individuals, unless one is either unusually risk-averse, or they have a very high confidence (such as above 50%) that AI will lead to human extinction. 

To be clear, if someone genuinely believes there is a high probability that AI will wipe out humanity, then I agree that delaying AI would seem rational, since the high risk of personal death would outweigh the small possibility of a dramatically improved life. But for those who see AI extinction risk as relatively low (such as below 15%), accelerating AI development appears to be the more pragmatic personal choice.

Thus, while human extinction would undoubtedly be a disastrous event, the idea that even a small risk of extinction from AI justifies delaying its development—even if that delay results in large numbers of currently existing humans dying from preventable causes—is not supported by straightforward utilitarian reasoning. The key question here is what extinction actually entails. If human extinction means the total disappearance of all complex life and the permanent loss of all future value, then mitigating even a small risk of such an event might seem overwhelmingly important. However, if the outcome of human extinction is simply that AIs replace humans—while still continuing civilization and potentially generating vast amounts of moral value—then the reasoning behind delaying AI development changes fundamentally.

In this case, the clearest and most direct tradeoff is not about preventing "astronomical waste" in the classic sense (i.e., preserving the potential for future civilizations) but rather about whether the risk of AI takeover is acceptable to the current generation of humans. In other words, is it justifiable to impose costs on presently living people—including delaying potentially life-saving medical advancements—just to reduce a relatively small probability that humanity might be forcibly replaced by AI? This question is distinct from the broader existential risk arguments that typically focus on preserving all future potential value, and it suggests that delaying AI is not obviously justified by utilitarian logic alone.

Kaspar Brandner @ 2025-02-12T20:11 (+1)

This argument appears very similar to the one I addressed in the essay about how delaying or accelerating AI will impact the well-being of currently existing humans. My claim is not that it isn't bad if humanity goes extinct; I am certainly not saying that it would be good if everyone died.

I'm not supposing you do. Of course most people have a strong preference not to die. But there is also (beyond that) a widespread preference for humanity not to go extinct. This is why it e.g. would be so depressing (as in the movie Children of Men) when a global virus made all humans infertile. Ending humanity is very different from and much worse than people merely dying at the end of their lives, which by itself doesn't imply extinction. Many people would likely even sacrifice their own life in order to safe the future of humanity. We don't have a similar preference for having AI descendants. That's not speciesist, it's just what our preferences are.

Matthew_Barnett @ 2025-02-12T19:50 (+5) in response to The ambiguous effect of full automation on wages

I think the conditions that support eventual below subsistence wages are fairly plausible, which is why I argued that the overall outcome is plausible. It appears Phillip Trammell either believes these conditions are less likely than I do, or decided to temporarily suspend judgement about their likelihood for the purposes of writing this post. Either way, while I agree the emphasis of our posts is different, I think the posts are still consistent with each other in a minimal sense.

trammell @ 2025-02-12T20:08 (+5)

Right, I’m just not taking a stand here. 

I might be more pessimistic than you about wages on balance, since I would argue for the importance of the “reallocation of capital from labor-augmenting to non-labor-augmenting uses” point, which if strong enough could lower wages through a channel other than the DRS+PS one you focus on. 

Matthew_Barnett @ 2025-02-12T19:50 (+5) in response to The ambiguous effect of full automation on wages

I think the conditions that support eventual below subsistence wages are fairly plausible, which is why I argued that the overall outcome is plausible. It appears Phillip Trammell either believes these conditions are less likely than I do, or decided to temporarily suspend judgement about their likelihood for the purposes of writing this post. Either way, while I agree the emphasis of our posts is different, I think the posts are still consistent with each other in a minimal sense.

JoshuaBlake @ 2025-02-12T20:03 (+3)

Thanks, that's very useful for me trying to follow but not that deep in the models!

Caroline Mills @ 2025-02-12T19:54 (+4) in response to Using a diet offset calculator to encourage effective giving for farmed animals

Thanks for posting this! I saw the tool and was intrigued, appreciate the opportunity to learn more about it, and about what you all are finding is working.

Vasco Grilo🔸 @ 2025-02-12T19:51 (+4) in response to Reaching a New Audience: Insights From a Market Research Survey

Thanks for sharing, Holly!

There was an increase in survey respondents’ self-reported likelihood to support our cause after engaging with our campaign materials, with a 28.6% increase for “very likely” and a 3% increase for “somewhat likely.”

It would be great if you eventually estimated how much campaigns like this increase donations to the organisations recommended by ACE.

JoshuaBlake @ 2025-02-12T19:43 (+2) in response to The ambiguous effect of full automation on wages

I feel like the overall takeaway is very different though. I've not fully understood the details in either argument so this is a little vibes based. You seemed to be arguing that below subsistence wages were fairly likely while here it seems to be that even falling wages require a weird combination of conditions.

What have I misunderstood?

Matthew_Barnett @ 2025-02-12T19:50 (+5)

I think the conditions that support eventual below subsistence wages are fairly plausible, which is why I argued that the overall outcome is plausible. It appears Phillip Trammell either believes these conditions are less likely than I do, or decided to temporarily suspend judgement about their likelihood for the purposes of writing this post. Either way, while I agree the emphasis of our posts is different, I think the posts are still consistent with each other in a minimal sense.

Matthew_Barnett @ 2025-02-07T21:27 (+5) in response to The ambiguous effect of full automation on wages

Summary

Given only “neutral” factor-augmenting technology, to reliably get the result that the increase in substitutability between capital and labor lowers wages, we need

  1. decreasing returns to scale and
  2. substitutability great enough that the decreasing returns to scale outweighs the fact that effective capital is now plentiful and maybe complementing labor a little bit. In the extreme, as shown above, decreasing returns to scale + perfect substitutability lowers wages.

I'll note that these are almost exactly the same conditions that I outlined in my recent article about the effects of AGI on human wages. It seems we're in agreement.

JoshuaBlake @ 2025-02-12T19:43 (+2)

I feel like the overall takeaway is very different though. I've not fully understood the details in either argument so this is a little vibes based. You seemed to be arguing that below subsistence wages were fairly likely while here it seems to be that even falling wages require a weird combination of conditions.

What have I misunderstood?

Davidmanheim @ 2025-02-12T12:48 (+3) in response to The standard case for delaying AI appears to rest on non-utilitarian assumptions

Are you willing to posit that advanced systems are coherent, with at least one non-satiable component? Because that's pretty minimal as an assumption, but implies with probability one that they prefer paperclipping of some sort.

Matthew_Barnett @ 2025-02-12T19:41 (+3)

Are humans coherent with at least one non-satiable component? If so, then I don’t understand the distinction you’re making that would justify positing AI values to be worse than human values from a utilitarian perspective. 

If not, then I’m additionally unclear on why you believe AIs will be unlike humans in this respect, to the extent that they would become "paperclippers." That term itself seems ambiguous to me (do you mean AIs will literally terminally value accumulating certain configurations of matter?). I would really appreciate a clearer explanation of your argument. As it stands, I don’t fully understand what point you’re trying to make.