Latest comments on the EA Forum

Comments on 2024-05-20

BrownHairedEevee @ 2024-05-18T18:29 (+8) in response to BrownHairedEevee's Quick takes

Are there currently any safety-conscious people on the OpenAI Board?

huw @ 2024-05-20T01:49 (+1)

The current board is:

  • Bret Taylor (chair): Co-created Google Maps, ex-Meta CTO, ex-Twitter Chairperson, current co-founder of Sierra (AI company)
  • Larry Summers: Ex U.S. Treasury Secretary, Ex Harvard president
  • Adam D'Angelo: Co-founder, CEO Quora
  • Dr. Sue Desmond-Hellmann: Ex-director P&G, Meta, Bill & Melinda Gates; Ex-chancellor UCSF. Pfizer board member
  • Nicole Seligman: Ex-Sony exec, Paramount board member
  • Fidji Simo: CEO & Chair Instacart, Ex-Meta VP

The only person here who even has rumours of being safety-conscious (AFAIK) is Adam D'Angelo, who allegedly played a role in kickstarting last year's board incident. God knows why Larry Summers is there (give it an air of professionalism?), the rest seem to me like your typical professional board members (i.e. unlikely to understand OpenAI's unique charter & structure). In my opinion, any hope of restraint from this board or OpenAI's current leadership is misplaced.

Ozzie Gooen @ 2024-05-20T01:34 (+2) in response to Project idea: AI for epistemics

Happy to see conversation and excitement on this!

Some quick points:
- Eli Lifland and I had a podcast episode about this topic a few weeks back. This goes into some detail on the details and viability of forecasting+AI being a cost-effective EA intervention.
- We at QURI have been investigating a certain thread of ambitious forecasting (which would require a lot of AI) for the last few years. We're a small group, but I think our writing and work would be interesting for people in this area.
- Our post Prioritization Research for Advancing Wisdom and Intelligence from 2021 described much of this area as "Wisdom and Intelligence" interventions, and there I similarly came to the conclusion that AI+epistemics was likely the most exciting generic area there. I'm still excited for more prioritization work and direct work in this area.
- The FTX Future Fund made epistemics and AI+epistemics a priority. I'd be curious to see other funders research this area more. (Hat tip to the new OP forecasting team)
- "A forecasting bot made by the AI company FutureSearch is making profit on the forecasting platform Manifold. The y-axis shows profit. This suggests it’s better even than collective prediction of the existing human forecasters." -> I want to flag here that it's not too hard for a smart human to do as good or better. Strong human forecasters are expected to make a substantial profit. A more accurate statement here is, "This suggests that it's powerful for automation to add value to a forecasting platform, and to outperform some human forecasters", which is a lower bar. I expect it will be a long time until AIs beat Humans+AIs in forecasting, but I agree AIs will add value.

 



Comments on 2024-05-19

Habryka @ 2024-05-19T22:24 (+2) in response to Project idea: AI for epistemics

Me and the other people working on Lightcone + LW are pretty interested in working in this space (and LW + the AI Alignment Forum puts us IMO in a great positions to get ongoing feedback on users for our work in the space, and we've also collaborated a good amount with Manifold historically). However, we currently don't have funding for it, which is our biggest bottleneck for working on this.

AI engineering tends to be particularly expensive in terms of talent and capital expenditures. If anyone knows of funders interested in this kind of stuff, who might be interested in funding us for this kind of work, letting me know would be greatly appreciated. 

Geoffrey Miller @ 2024-05-17T19:21 (+5) in response to Why hasn't there been any significant AI protest

Good question. My hunch is that EA as a culture tends to prioritize epistemic and ethical sophistication and rigor, over direct 'political' action. And has traditionally avoided getting involved in issues that seem 'intractable' by virtue of being highly controversial and potentially partisan.

Against that background of EA's rather 'ivory tower' ethos, any direct protests may tend to be seen as rather simplistic, strident, and undignified -- even for issues such as animal agriculture where there's pretty strong EA consensus that factory farming is unethical. 

But I think it's time for EAs to climb down from our AI safety debates, recognize that the leading AI companies are not actually prioritizing safety, and start getting more involved in social media activism and in-person protests.

GideonF @ 2024-05-19T22:06 (+2)

I think this is untrue with regards to animal protests. My impression is a decently significant percentage of EA people working on animals have participated in protests

Lukas_Gloor @ 2024-05-19T21:33 (+4) in response to Project idea: AI for epistemics

Somewhat relatedly, what about using AI to improve not your own (or your project's) epistemics, but improve public discourse? Something like "improve news" or "improve where people get their info on controversial topics."

Edit: To give more context, I was picturing something like training LLMs to pass ideological turing tests and then create a summary of the strongest arguments for and against, as well as takedowns of common arguments by each side that are clearly bad. And maybe combine that with commenting on current events as they unfold (to gain traction), handling the tough balance of having to compete in the attention landscape while still adhering to high epistemic standards. The goal then being something like "trusted source of balanced reporting," which you can later direct to issues that matter the most (after gaining traction earlier by discussing all sorts of things).

Erich_Grunewald @ 2024-05-19T19:28 (+10) in response to The bullseye framework: My case against AI doom

Let me see if I can rephrase your argument, because I'm not sure I get it. As I understand it, you're saying:

  1. In humans, higher IQ means better performance across a variety of tasks. This is analogous to AI, where more compute/parameters/data etc. means better performance across a variety of tasks.
  2. AI systems tend to share a common underlying architecture, just as humans share the same basic biology.
  3. For humans, when IQ increases, there are improvements across the board, but still specialization, meaning no single human (the one with the most IQ) will be better than all other humans at all of those things.
  4. By analogy: For AIs, when they're scaled up, there are improvements across the board, but (likely) still specialization, meaning no single AI (the one with the most compute/parameters/data/etc.) will be better than all other AIs at all of those things.

Now I'm a bit unsure about whether you're saying that you find it extremely unlikely that any AI will be vastly better in the areas I mentioned than all humans, or that you find it extremely unlikely that any AI will be vastly better than all humans and all other AIs in those areas.

If you mean 1-4 to suggest that no AI is will be better than all humans and other AIs, I'm not sure about whether 4 follows from 1-3, but I think that seems plausible at least. But if this is what you mean, I'm not sure what you're original comment ("Note humans are also trained on all those abilities, but no single human is trained to be a specialist in all those areas. Likewise for AIs.") was meant to say in response to my original comment, which was meant as pushback against the view that AGI would be bad at taking over the planet since it wouldn't be intended for that purpose.

If you mean 1-4 to suggest that no AI will be better than all humans, I don't think the analogy holds, because the underlying factor (IQ versus AI scale/algorithms) is different. Like, it seems possible that even unspecialized AIs could just sweep past the most intelligent and specialized humans, given enough time.

Vasco Grilo @ 2024-05-19T21:11 (+4)

Thanks for the clarification, Erich! Strongly upvoted.

Let me see if I can rephrase your argument

I think your rephrasement was great.

Now I'm a bit unsure about whether you're saying that you find it extremely unlikely that any AI will be vastly better in the areas I mentioned than all humans, or that you find it extremely unlikely that any AI will be vastly better than all humans and all other AIs in those areas.

The latter.

If you mean 1-4 to suggest that no AI is will be better than all humans and other AIs, I'm not sure about whether 4 follows from 1-3, but I think that seems plausible at least. But if this is what you mean, I'm not sure what you're original comment ("Note humans are also trained on all those abilities, but no single human is trained to be a specialist in all those areas. Likewise for AIs.") was meant to say in response to my original comment, which was meant as pushback against the view that AGI would be bad at taking over the planet since it wouldn't be intended for that purpose.

I think a single AI agent would have to be better than the vast majority of agents (including both human and AI agents) to gain control over the world, which I consider extremely unlikely given gains from specialisation.

If you mean 1-4 to suggest that no AI will be better than all humans, I don't think the analogy holds, because the underlying factor (IQ versus AI scale/algorithms) is different. Like, it seems possible that even unspecialized AIs could just sweep past the most intelligent and specialized humans, given enough time.

I agree.

I'd be curious to hear if you have thoughts about which specific abilities you expect an AGI would need to have to take control over humanity that it's unlikely to actually possess?

I believe the probability of a rogue (human or AI) agent gaining control over the world mostly depends on its level of capabilities relative to those of the other agents, not on the absolute level of capabilities of the rogue agent. So I mostly worry about concentration of capabilities rather than increases in capabilities per se. In theory, the capabilities of a given group of (human or AI) agents could increase a lot in a short period of time such that capabilities become so concentrated that the group would be in a position to gain control over the world. However, I think this is very unlikely in practice. I guess the annual probability of human extinction over the next 10 years is around 10^-6.

Wei Dai @ 2024-05-19T18:02 (+4) in response to Don't panic: 90% of EAs are good people

Those low on the spectrum tend to shape the incentives around them proactively to create a culture that rewards what they don’t want to lose about their good qualities.

What percent of people do you think fall into this category? Any examples? Why are we so bad at distinguishing such people ahead of time and often handing power to the easily corrupted instead?

Lukas_Gloor @ 2024-05-19T20:48 (+4)

Off the cuff answers that may change as I reflect more:

  • Maybe around 25% of people in leadership positions in the EA ecosystem qualify? Somewhat lower for positions at orgs that are unusually "ambitious;" somewhat higher for positions that are more like "iterate on a proven system" or "have a slow-paced research org that doesn't involve itself too much in politics."
  • For the ambitious leaders, I unfortunately have no examples where I feel particularly confident, but can think of a few examples where I'm like "from a distance, it looks like they might be good leaders." I would count Holden in that category, even though I'd say the last couple of years seem suboptimal in terms of track record (and also want to flag that this is just a "from a distance" impression, so don't put much weight on it).
  • Why we're bad at identifying: This probably isn't the only reason, but the task is just hard. If you look at people who have ambitious visions and are willing to try hard to make them happen, they tend to be above-average on dark triad traits. You probably want someone who is very much not high on psychopathic traits, but still low enough on neuroticism that they won't be anxious all the time. Similarly, you want someone who isn't too high on narcissism, but they still need to have that ambitious vision and belief in being exceptional. You want someone who is humble and has inner warmth so they will uplift others along the way, so high on honesty-humility factor, but that correlates with agreeableness and neuroticism – which is a potential problem because you probably can't be too agreeable in the startup world or when running an ambitious org generally, and you can't be particularly neurotic.
    • (Edit) Another reason is, I think people often aren't "put into leadership positions" by others/some committee; instead, they put themselves there. Like, usually there isn't some committee with a great startup idea looking for a leader; instead, the leader comes with the vision and accumulates followers based on their conviction. And most people who aren't in leadership positions simply aren't vigilant or invested enough to care a lot about who becomes a leader. 

I think incentives matter, but I feel like if they're all that matters, then we're doomed anyway because "Who will step up as a leader to set good incentives?" In other words, the position "incentives are all that matters" seems self-defeating, because to change things, you can't just sit on the sidelines and criticize "the incentives" or "the system." It also seems too cynical: just because, e.g., lots of money is at stake, that doesn't mean people who were previously morally motivated and cautious about their motivations and trying to do the right thing, will suddenly go off the rails.

To be clear, I think there's probably a limit for everyone and no person is forever safe from corruption, but my point is that it matters where on the spectrum someone falls. Of the people that are low on corruptibility, even though most of them don't like power or would flail around helplessly and hopelessly if they had it, there are probably people who have the right mix of traits to create, maintain and grow pockets of sanity (well-run, well-functioning organizations, ecosystems, etc.). 

Brad West @ 2024-05-15T15:40 (+9) in response to Ilya Sutskever has officially left OpenAI

It seems as if the potential of the damages could make the vast majority of defendants "judgment-proof" (meaning they lack the assets to satisfy the judgment).

I wonder about the ethics of an organization that had the policy of financially supporting people (post-bankruptcy) who made potentially extremely high EV decisions that were personally financially ruinous.

Jason @ 2024-05-19T20:12 (+2)

I probably would be OK with that from an ethics standpoint. After all, I was not a party to the contracts in question. We celebrate (in appropriate circumstances) journalists who serve as conduits for actual classified information. Needless to say, I find the idea of being an enabler for the breach of contractual NDAs much less morally weighty than being an enabler for the breach of someone's oath to safeguard classified information.

Legally, such an organization would have to be careful to mitigate the risk of claims for tortious interference with contract and other theories that the AI company could come up with. Promising financial support prior to the leak might open the door for such claims; merely providing it (through a well-written trust) after the fact would probably be OK.

Vasco Grilo @ 2024-05-19T09:04 (+2) in response to The bullseye framework: My case against AI doom

I would expect improvements on these types of tasks to be highly correlated in general-purpose AIs.

Higher IQ in humans is correlated with better performance in all sorts of tasks too, but the probability of finding a single human performing better than 99.9 % of (human or AI) workers in each of the areas you mentioned is still astronomically low. So I do not expect a single AI system to become better than 99.9 % of (human or AI) workers in each of the areas you mentioned. It can still be the case that the AI systems share a baseline common architecture, in the same way that humans share the same underlying biology, but I predict the top performers in each area will still be specialised systems.

I think we've seen that with GPT-3 to GPT-4, for example: GPT-4 got better pretty much across the board (excluding the tasks that neither of them can do, and the tasks that GPT-3 could already do perfectly). That is not the case for a human who will typically improve in just one domain or a few domains from one year to the next, depending on where they focus their effort.

Going from GPT-3 to GPT-4 seems more analogous to a human going from 10 to 20 years old. There are improvements across the board during this phase, but specialisation still matters among adults. Likewise, I assume specialisation will matter among frontier AI systems (although I am quite open to a single future AI system being better than all humans at any task). GPT-4 is still far from being better than 99.9 % of (human or AI) workers in the areas you mentioned.

Erich_Grunewald @ 2024-05-19T19:28 (+10)

Let me see if I can rephrase your argument, because I'm not sure I get it. As I understand it, you're saying:

  1. In humans, higher IQ means better performance across a variety of tasks. This is analogous to AI, where more compute/parameters/data etc. means better performance across a variety of tasks.
  2. AI systems tend to share a common underlying architecture, just as humans share the same basic biology.
  3. For humans, when IQ increases, there are improvements across the board, but still specialization, meaning no single human (the one with the most IQ) will be better than all other humans at all of those things.
  4. By analogy: For AIs, when they're scaled up, there are improvements across the board, but (likely) still specialization, meaning no single AI (the one with the most compute/parameters/data/etc.) will be better than all other AIs at all of those things.

Now I'm a bit unsure about whether you're saying that you find it extremely unlikely that any AI will be vastly better in the areas I mentioned than all humans, or that you find it extremely unlikely that any AI will be vastly better than all humans and all other AIs in those areas.

If you mean 1-4 to suggest that no AI is will be better than all humans and other AIs, I'm not sure about whether 4 follows from 1-3, but I think that seems plausible at least. But if this is what you mean, I'm not sure what you're original comment ("Note humans are also trained on all those abilities, but no single human is trained to be a specialist in all those areas. Likewise for AIs.") was meant to say in response to my original comment, which was meant as pushback against the view that AGI would be bad at taking over the planet since it wouldn't be intended for that purpose.

If you mean 1-4 to suggest that no AI will be better than all humans, I don't think the analogy holds, because the underlying factor (IQ versus AI scale/algorithms) is different. Like, it seems possible that even unspecialized AIs could just sweep past the most intelligent and specialized humans, given enough time.

Ofer @ 2024-05-19T13:04 (+2) in response to Don't panic: 90% of EAs are good people

(haven't read the entire post)

I think the "good people'' label is not useful here. The problem is that humans tend to act as power maximizers, and they often deceive themselves into thinking that they should do [something that will bring them more power] because of [pro-social reason].

I'm not concerned that Dario Amodei will consciously think to himself: "I'll go ahead and press this astronomically net-negative button over here because it will make me more powerful". But he can easily end up pressing such a button anyway.

Closed Limelike Curves @ 2024-05-19T19:15 (+7)

The usefulness of the "bad people" label is exactly my point here. The fact of the matter is some people are bad, no matter what excuses they come up with. For example, Adolf Hitler was clearly a bad person, regardless of his belief that he was the single greatest and most ethical human being who had ever lived. The argument that all people have an equally strong moral compass is not tenable.

More than that, when I say "Sam Altman is a bad person", I don't mean "Sam Altman's internal monologue is just him thinking over and over again 'I want to destroy the world'". It means "Sam Altman's internal monologue is really good at coming up with excuses for unethical behavior".

Like:

I'm not concerned that Dario Amodei will consciously think to himself: "I'll go ahead and press this astronomically net-negative button over here because it will make me more powerful". But he can easily end up pressing such a button anyway.

I would like to state, for the record, that if Sam Altman pushes a "50% chance of making humans extinct" button, this makes him a bad person, no matter what he's thinking to himself. Personally I would just not press that button.

Wei Dai @ 2024-05-19T18:02 (+4) in response to Don't panic: 90% of EAs are good people

Those low on the spectrum tend to shape the incentives around them proactively to create a culture that rewards what they don’t want to lose about their good qualities.

What percent of people do you think fall into this category? Any examples? Why are we so bad at distinguishing such people ahead of time and often handing power to the easily corrupted instead?

Closed Limelike Curves @ 2024-05-19T19:08 (+1)

If I had to guess, the EA community is probably a bit worse at this than most communities because A) bad social skills and B) high trust.

This seems like a good tradeoff in general. I don't think we should be putting more emphasis on smooth-talking CEOs—which is what got us into the OpenAI mess in the first place. 

But at some point, defending Sam Altman is just charlie_brown_football.jpg

Dicentra @ 2024-05-19T05:19 (+9) in response to Don't panic: 90% of EAs are good people

I'm confused by this post. Sam Altman isn't an EA, afaik, and hasn't claimed to be, afaik, and afaik no relatively in-the-know EAs thought he was, or even in recent years thought he was particularly trustworthy, though I'd agree that many have updated negative over the last year or two.

But a substantial number of EAs spent the next couple of weeks or months making excuses not to call a spade a spade, or an amoral serial liar an amoral serial liar. This continued even after we knew he'd A) committed massive fraud, B) used that money to buy himself a $222 million house, and C) referred to ethics as a "dumb reputation game" in an interview with Kelsey Piper.

This wasn't because they thought the fraud was good; everyone was clear that SBF was very bad. It's because a surprisingly big number of people can't identify a psychopath. I'd like to offer a lesson on how to tell. If someone walks up to you and says "I'm a psychopath", they're probably a psychopath.

Very few EAs that I know did that (I'd like to see stats, of the dozens of EAs I know, none publicly/to my knowledge did such a thing except if I remember right Austin Chen in an article I now can't find). And for people who did defend Sam, I don't know why you'd assume that the issue is them not being able to identify psychopaths, as opposed to being confused about the crimes SBF committed and believing they were the result of a misunderstanding or something like that

Closed Limelike Curves @ 2024-05-19T19:07 (+5)

In the conversations I had with them, they very clearly understood the charges against him and what he'd done. The issue was they were completely unable to pass judgment on him as a person.

This is a good trait 95% of the time. Most people are too quick to pass judgment. This is especially true because 95% of people pass judgment based on vibes like "Bob seems weird and creepy" instead of concrete actions like "Bob has been fired from 3 of his last 4 jobs for theft".

However, the fact of the matter is some people are bad. For example, Adolf Hitler was clearly a bad person. Bob probably isn't very honest. Sam Altman's behavior is mostly motivated by a desire for money and power. This is true even if Sam Altman has somehow tricked himself into thinking his actions are good. Regardless of his internal monologue he's still acting to maximize his money and power.

EAs often have trouble going "Yup, that's a bad person" when they see someone who's very blatantly a bad person.

Mitchell Laughlin @ 2024-05-19T04:59 (+6) in response to Don't panic: 90% of EAs are good people

tldr, thanks for writing.

I would consider myself to be a very trusting and forgiving person who always starts with the mentality of "Yes, but most people are inherently good". I definitely went through a long period of "Ah well I mean surely Sam (SBF) isn't actually a 'bad' guy right". This felt like a productive slap in the face.

I would still prefer to be more than less trusting, but maybe I should dial it back a bit. Thanks. 

Closed Limelike Curves @ 2024-05-19T18:54 (+1)

"Trust but verify" is Reagan's famous line on this.

Most EAs would agree with "90% of people are basically trying to do the right thing". But most of them have a very difficult time acting as though there's a 10% chance anyone they're talking to is an asshole. You shouldn't be expecting people to be assholes, but you should be considering the 10% chance they are and updating that probability based on evidence. Maya Angelou wrote "If someone shows you who they are, believe them the first time".

As a Bayesian who recognizes the importance of not updating too quickly away from your prior, I'd like to amend this to "If someone shows you who they are, believe them the 2nd or 3rd time they release a model that substantially increases the probability we're all going to die".

Lukas_Gloor @ 2024-05-19T14:54 (+9) in response to Don't panic: 90% of EAs are good people

If you're saying "it takes more than good intentions to not get corrupted," I agree.

But then the question is, "Is Dario someone who's unusually unlikely to get corrupted?"

If you're saying "it doesn't matter who you put in power; bad incentives will corrupt everything," then I don't agree.

I think people differ a lot wrt how easily they get corrupted by power (or other bad incentive structures). Those low on the spectrum tend to shape the incentives around them proactively to create a culture that rewards what they don't want to lose about their good qualities.

Wei Dai @ 2024-05-19T18:02 (+4)

Those low on the spectrum tend to shape the incentives around them proactively to create a culture that rewards what they don’t want to lose about their good qualities.

What percent of people do you think fall into this category? Any examples? Why are we so bad at distinguishing such people ahead of time and often handing power to the easily corrupted instead?

DLMRBN @ 2024-05-19T15:23 (+1) in response to Shareholder Activism

Great article, thanks! I wonder if the work by Follow This fits the definition: https://www.follow-this.org/

Frank_R @ 2024-05-19T09:18 (+1) in response to Questioning assumptions: Why the EA community should lead in the debate on brain preservation

To answer your question I have to describe some scenarios how a non-aligned AI would act. This is slightly cringe since we do not know what an unaligned AI would do and this sounds very sci-fi like. In the case of something like a robot uprising or a nuclear war started by an AI many people would die under circumstances such that uploading is impossible, but brain banks could be still intact. If an unaligned AI really has the aim to upload and torture everyone, there will probably be better ways. [Insert something with nanobots here.]

In my personal, very subjective opinion there is a 10% chance of extinction by AI and a 1% chance for s-risks or something like Roko's basilisk. You may have different subjective probabilities and even if we agree on the possibilities, it depends very much on your preferred ethical theory what to do.       

AndyMcKenzie @ 2024-05-19T15:06 (+1)

I'm not disagreeing with you that there is a possibility, however small, of s-risk scenarios. I agree with this point of yours, although I'm thinking of things more like superhuman persuasion, deception, pinning human tribes against one another, etc., rather than nanobots necessarily: 

If an unaligned AI really has the aim to upload and torture everyone, there will probably be better ways. [Insert something with nanobots here.]

People often bring up this in the context of brain preservation. But it just seems to me that this possibility is mostly a generalized argument against life extension, medicine, pronatalism, etc in general. 

Ofer @ 2024-05-19T13:04 (+2) in response to Don't panic: 90% of EAs are good people

(haven't read the entire post)

I think the "good people'' label is not useful here. The problem is that humans tend to act as power maximizers, and they often deceive themselves into thinking that they should do [something that will bring them more power] because of [pro-social reason].

I'm not concerned that Dario Amodei will consciously think to himself: "I'll go ahead and press this astronomically net-negative button over here because it will make me more powerful". But he can easily end up pressing such a button anyway.

Lukas_Gloor @ 2024-05-19T14:54 (+9)

If you're saying "it takes more than good intentions to not get corrupted," I agree.

But then the question is, "Is Dario someone who's unusually unlikely to get corrupted?"

If you're saying "it doesn't matter who you put in power; bad incentives will corrupt everything," then I don't agree.

I think people differ a lot wrt how easily they get corrupted by power (or other bad incentive structures). Those low on the spectrum tend to shape the incentives around them proactively to create a culture that rewards what they don't want to lose about their good qualities.

Vaidehi Agarwalla @ 2024-05-19T14:27 (+20) in response to Marisa, the Co-Founder of EA Anywhere, Has Passed Away 

Marisa was a wonderfully kind, thoughtful and supportive person. She was one of the first friends I made in EA many years ago, and I felt I'd found a kindred spirit in her - we exchanged many ideas and about sociology, ops, community building. She gave wonderfully helpful feedback and I always enjoyed learning about what she was up to (always something new!).

I'm so saddened and heartbroken by this news. She deserved so much more.

CHL @ 2024-05-19T09:32 (+2) in response to Joseph Lemien's Quick takes

Why not start taking those steps today?

Joseph Lemien @ 2024-05-19T13:37 (+7)

Because my best estimate is that there are different steps toward different paths that would be better than trying to rewind life back to college age and start over. Like the famous Sylvia Plath quote about life branching like a fig tree, unchosen paths tend to wither away. I think that becoming a software developer wouldn't be the best path for me at this point: cost of tuition, competitiveness of the job market for entry-level developers, age discrimination, etc.

Being a 22-year old fresh grad with a bachelor's degree in computer science in 2010 is quite a different scenario than being a 40-year old who is newly self-taught through Free Code Camp in 202X. I predict that the former would tend to have a lot of good options (with wide variance, of course), while the latter would have fewer good options. If there was some sort of 'guarantee' regarding a good job offer or if a wealthy benefactor offered to cover tuition and cost of living while I learn then I would give training/education very serious consideration, but my understanding is that the 2010s were an abnormally good decade to work in tech, and there is now a glut of entry-level software developers.

Ofer @ 2024-05-19T13:04 (+2) in response to Don't panic: 90% of EAs are good people

(haven't read the entire post)

I think the "good people'' label is not useful here. The problem is that humans tend to act as power maximizers, and they often deceive themselves into thinking that they should do [something that will bring them more power] because of [pro-social reason].

I'm not concerned that Dario Amodei will consciously think to himself: "I'll go ahead and press this astronomically net-negative button over here because it will make me more powerful". But he can easily end up pressing such a button anyway.

Mjreard @ 2024-05-15T19:23 (+1) in response to Hiring non-EA Talent: Pros & Cons

Things downstream of OpenPhil are in the 90th+ percentile of charity pay, yes, but why do people work in the charity sector? Either because they believe in the specific thing (i.e. they are EAs) or because they want the warm glow of working for a charity. Non-EA charities offer more warm glow, but maybe there's a corner of "is a charity" and "pays well for a charity even though people in my circles don't get it" that appeals to some. I claim it's not many and EA jobs are hard to discover for the even smaller population of people who have preferences like these and are high competence. 

Junior EA roles sometimes pay better than market alternatives in the short run, but I believe high potential folks will disproportionately track lifetime earnings vs the short run and do something that's better career capital.

Tatiana K. Nesic Skuratova @ 2024-05-19T11:24 (+1)

I claim it's not many and EA jobs are hard to discover for the even smaller population of people who have preferences like these and are high competence.

On this note, a proactive recruitment approach emphasizes the importance of actively reaching out to potential candidates rather than passively waiting for the right individuals to encounter the job description on the right platform at the right time. I plan to write a follow-up post regarding the recrtuiment "craft" particular in EA field. 

Vasco Grilo @ 2024-05-19T11:17 (+3) in response to [Draft] The humble cosmologist's P(doom) paradox

Hi titotal,

I think it makes sense to assess the annual risk of simulation shutdown based on the mean annual probability of simulation shutdown. However, I also believe the risk of simulation shutdown is much lower than the one guessed by the humble cosmologist.

The mean of a loguniform distribution ranging from a to 1 is -1/ln(a). If a = 10^-100, the risk is 0.434 % (= -1/ln(10^-100)). However, I assume there is no reason to set the minimum risk to 10^-100, so the cosmologist may actually have been overconfident. Since there is no obvious natural lower bound for the risk, because more or less by definition we do not have evidence about the simulators, I guess the lower bound can be arbitrarily close to 0. In this case, the mean of the loguniform distribution goes to 0 (= -1/(ln(0))), so it looks like the humblest view corresponds to 0 risk of simulation shutdown.

In addition, the probability of surviving an annual risk of simulation shutdown of 0.434 % (= 10^-5.44) over the estimated age of the universe of 13.8 billion years is only 10^-75,072,000,000 (= (10^-5.44)^(13.8*10^9)), which is basically 0. So the universe would need to be super super lucky in order to have survived for so long with such high risk. One can try to counter this argument saying there are selection effects. However, it would be super strange to have an annual risk of simulation shutdown of 0.434 % without any partial shutdowns, given that tail risk usually follows something like a power law[1] without severe jumps in severity.

  1. ^

    Although I think tail risk often decays faster than suggested by a power law.

James Herbert @ 2024-05-14T11:14 (+19) in response to Hiring non-EA Talent: Pros & Cons

Agreed. 

Quick case study. We hired a production officer for EAGxUtrecht who is a professional event manager but wasn't even aware of EA as a thing. She's amazing. 

Tatiana K. Nesic Skuratova @ 2024-05-19T11:11 (+1)

Thank you for sharing and it's great to hear about another successful case! 

Vasco Grilo @ 2024-05-19T10:04 (+2) in response to The Leeroy Jenkins principle: How faulty AI could guarantee "warning shots"

Nice post, titotal!

This could be a whole post in itself, and in fact I’ve already explored it a bit here.

The link is private.

Elizabeth @ 2024-05-15T18:11 (+15) in response to 5 things you’ve got wrong about the Giving What We Can Pledge

Let’s say only one other person in your network hears that you took the pledge and is inspired to do the same. That would be doubling your impact. If two people in your network were inspired to pledge based on your decision, that would be tripling your impact

 

This math seems off on several levels. 

Will Howard @ 2024-05-19T09:55 (+7)

I didn't find this paragraph to be off or particularly misleading fwiw.

It is roughly true (minus what they would have donated otherwise) when thinking in terms of counterfactual impact, and assuming you are an average pledger and would be inspiring other average pledgers (no expected difference in income or attrition, or effectiveness of charities donated to).

I think the caveats are sufficiently obvious that the reader could be expected to understand them on their own. For instance if you convince someone to donate $1000 it seems obvious that they should get most of the credit, but it still might be true that you were counterfactual in their decision.

Joseph Lemien @ 2024-05-18T17:28 (+14) in response to Joseph Lemien's Quick takes

I just looked at [ANONYMOUS PERSON]'s donations. The amount that this person has donated in their life is more than double the amount that I have ever earned in my life. This person appears to be roughly the same age as I am (we graduated from college ± one year of each other). Oof. It makes me wish that I had taken steps to become a software developer back when I was 15 or 18 or 22.

Oh, well. As they say, comparison is the thief of joy. I'll try to focus on doing the best I can with the hand I'm dealt.

CHL @ 2024-05-19T09:32 (+2)

Why not start taking those steps today?

AndyMcKenzie @ 2024-05-18T14:24 (+3) in response to Questioning assumptions: Why the EA community should lead in the debate on brain preservation

I'm not sure I understand the scenario you are discussing. In your scenario, it sounds like you're positing a malevolent non-aligned AI that would forcibly upload and create suffering copies of people. Obviously, this is an almost unfathomably horrific hypothetical scenario which we should all try to prevent if we can. One thing I don't understand about the scenario you are describing is why this forcible uploading would only happen to people who are legally dead and preserved at the time, but not anyone living at the time. 

Frank_R @ 2024-05-19T09:18 (+1)

To answer your question I have to describe some scenarios how a non-aligned AI would act. This is slightly cringe since we do not know what an unaligned AI would do and this sounds very sci-fi like. In the case of something like a robot uprising or a nuclear war started by an AI many people would die under circumstances such that uploading is impossible, but brain banks could be still intact. If an unaligned AI really has the aim to upload and torture everyone, there will probably be better ways. [Insert something with nanobots here.]

In my personal, very subjective opinion there is a 10% chance of extinction by AI and a 1% chance for s-risks or something like Roko's basilisk. You may have different subjective probabilities and even if we agree on the possibilities, it depends very much on your preferred ethical theory what to do.       

Erich_Grunewald @ 2024-05-18T22:22 (+6) in response to The bullseye framework: My case against AI doom

For an agent to conquer to world, I think it would have to be close to the best across all those areas

That seems right.

I think this is super unlikely based on it being super unlikely for a human to be close to the best across all those areas

I'm not sure that follows? I would expect improvements on these types of tasks to be highly correlated in general-purpose AIs. I think we've seen that with GPT-3 to GPT-4, for example: GPT-4 got better pretty much across the board (excluding the tasks that neither of them can do, and the tasks that GPT-3 could already do perfectly). That is not the case for a human who will typically improve in just one domain or a few domains from one year to the next, depending on where they focus their effort.

Vasco Grilo @ 2024-05-19T09:04 (+2)

I would expect improvements on these types of tasks to be highly correlated in general-purpose AIs.

Higher IQ in humans is correlated with better performance in all sorts of tasks too, but the probability of finding a single human performing better than 99.9 % of (human or AI) workers in each of the areas you mentioned is still astronomically low. So I do not expect a single AI system to become better than 99.9 % of (human or AI) workers in each of the areas you mentioned. It can still be the case that the AI systems share a baseline common architecture, in the same way that humans share the same underlying biology, but I predict the top performers in each area will still be specialised systems.

I think we've seen that with GPT-3 to GPT-4, for example: GPT-4 got better pretty much across the board (excluding the tasks that neither of them can do, and the tasks that GPT-3 could already do perfectly). That is not the case for a human who will typically improve in just one domain or a few domains from one year to the next, depending on where they focus their effort.

Going from GPT-3 to GPT-4 seems more analogous to a human going from 10 to 20 years old. There are improvements across the board during this phase, but specialisation still matters among adults. Likewise, I assume specialisation will matter among frontier AI systems (although I am quite open to a single future AI system being better than all humans at any task). GPT-4 is still far from being better than 99.9 % of (human or AI) workers in the areas you mentioned.

Dicentra @ 2024-05-19T05:19 (+9) in response to Don't panic: 90% of EAs are good people

I'm confused by this post. Sam Altman isn't an EA, afaik, and hasn't claimed to be, afaik, and afaik no relatively in-the-know EAs thought he was, or even in recent years thought he was particularly trustworthy, though I'd agree that many have updated negative over the last year or two.

But a substantial number of EAs spent the next couple of weeks or months making excuses not to call a spade a spade, or an amoral serial liar an amoral serial liar. This continued even after we knew he'd A) committed massive fraud, B) used that money to buy himself a $222 million house, and C) referred to ethics as a "dumb reputation game" in an interview with Kelsey Piper.

This wasn't because they thought the fraud was good; everyone was clear that SBF was very bad. It's because a surprisingly big number of people can't identify a psychopath. I'd like to offer a lesson on how to tell. If someone walks up to you and says "I'm a psychopath", they're probably a psychopath.

Very few EAs that I know did that (I'd like to see stats, of the dozens of EAs I know, none publicly/to my knowledge did such a thing except if I remember right Austin Chen in an article I now can't find). And for people who did defend Sam, I don't know why you'd assume that the issue is them not being able to identify psychopaths, as opposed to being confused about the crimes SBF committed and believing they were the result of a misunderstanding or something like that

Mitchell Laughlin @ 2024-05-19T04:59 (+6) in response to Don't panic: 90% of EAs are good people

tldr, thanks for writing.

I would consider myself to be a very trusting and forgiving person who always starts with the mentality of "Yes, but most people are inherently good". I definitely went through a long period of "Ah well I mean surely Sam (SBF) isn't actually a 'bad' guy right". This felt like a productive slap in the face.

I would still prefer to be more than less trusting, but maybe I should dial it back a bit. Thanks. 

Jeffrey Kursonis @ 2024-05-19T04:52 (+1) in response to Why not socialism?

In the USA anybody with center leaning tendencies thinks of "socialism" as all the bad things that happened under communism, but democratic socialism is more palatable since it has private property and voting. I understand this is partly a problem of the right using those words to instill fear based on remaining cold war feelings, but still the challenge here is to educate to show how democratic socialists governments are not Soviet style communists. I personally prefer the hybrid of democratic and some capitalistic aspects being mixed in. I don't think communism can ever work based on the massive experiment we've already had with it that failed. If China is doing well it's clearly because of the capitalism mixed in. 

If I were to evangelize for socialism within EA I would definitely stay on using "democratic socialist" language, or it'll get slammed harder. And I'd educate on the private property and voting aspects. 

I'd personally like to see EA be less Utilitarian and be able to meet more in the middle with social movement interventions rather than just individuals...that might help democratic socialism to more accept EA also, and for EA to accept it. 

John S Dodds @ 2024-05-16T12:31 (–41) in response to Why not socialism?

A better name for socialism? Ineffective Altruism. 

Nowhere does this post explain which elements of socialism are intrinsically more efficient at delivering a better outcome for more people than Effective Altruism. And any that are proven to do so would then become elements of both EA and Socialism. 

Either way, Socialism remains focused on economic equity, community ownership of industrial hubs, and general egalitarianism. Ideological priorities, of which, none have been proven to lead to better outcomes for more people, in fact, which, so far, have had major negative effects on the nations which attempt to implement them. 

The reverse of this post could be written, explaining that training the entire population to be effective capitalist agents would be one of the highest RoI policies that EAs could push for, and it would almost certainly be correct. 

All in all, bad ideas, advocated by the intellectually weak, appealing mostly to the genetically subpar. 

Will Aldred @ 2024-05-19T03:11 (+2)

The closing sentence of this comment, “All in all, bad ideas, advocated by the intellectually weak, appealing mostly to the genetically subpar,” breaks our Forum norm against unnecessary rudeness or offensiveness.

The “genetically subpar” part is especially problematic. At best, it would appear that the commenter, John, is claiming that the post mainly appeals to the less intelligent—an unnecessarily rude and most likely false claim. A worse interpretation is that John is making a racist remark, which we view as strongly unacceptable.

Overall, we see this as an unpromising start to John’s Forum engagement—this is their first comment—and we have issued a one-month ban. If they return to the Forum then we’ll expect to see a higher standard of discourse.

As a reminder, bans affect the user, not the account.

If anyone has questions or concerns, feel free to reach out: if you think we made a mistake here, you can appeal the decision.

Ryan Greenblatt @ 2024-05-19T00:55 (+8) in response to "If we go extinct due to misaligned AI, at least nature will continue, right? ... right?"

I think literal extinction is unlikely even conditional on misaligned AI takeover due to:

  • The potential for the AI to be at least a tiny bit "kind" (same as humans probably wouldn't kill all aliens).[1]
  • Decision theory/trade reasons

This is discussed in more detail here and here.

Insofar as humans and/or aliens care about nature, similar arguments apply there too, though this is mostly beside the point: if humans survive and have (even a tiny bit of) resources they can preserve some natural easily.

I find it annoying how confident this article is without really bother to engage with the relevant arguments here.

(Same goes for many other posts asserting that AIs will disassemble humans for their atoms.)

(This comment echos Owen's to some extent.)

  1. ^

    This includes the potential for the AI to have preferences that are morally valueable from a typical human perspective.

Ben Millwood @ 2024-05-18T23:00 (+4) in response to Call for Attorneys for OpenAI Employees and Ex-Employees

Does anyone have guesses about how much it would cost to pay a California-licensed employment lawyer to form an opinion on this?

Jason @ 2024-05-19T00:02 (+14)

low thousands? Obviously haven't seen the documents, but a few hours times few hundred dollars per billable hours should be in the ballpark. Obviously someone might want a more in-depth study before taking actions that could expose them to massive liability, though...



Comments on 2024-05-18

Ben Millwood @ 2024-05-18T23:05 (+4) in response to Marisa, the Co-Founder of EA Anywhere, Has Passed Away 

do you know if Marisa exclusively used they/them pronouns, or she/they, or what? I remember hearing something about this but I'm not certain and can't find any online profiles anymore :(

IanDavidMoss @ 2024-05-18T23:19 (+6)

Marisa told me last fall that she'd settled on she/her, so that's what I've been using.

siobhanbrenton @ 2024-05-18T22:59 (+38) in response to Marisa, the Co-Founder of EA Anywhere, Has Passed Away 

Marisa was a beautiful soul who found great joy in helping others. As an exceptional generalist, I was confident that they would become a great operational leader once they were able to find inner peace.

Marisa began their journey with Rethink Charity as a volunteer and quickly became a fundamental part of the team. The two years they spent working on this project before starting grad school were transformational for RC. Marisa made me a better leader, and I am eternally grateful for the friendship we built.

The lockdowns were especially tough on Marisa, and I had hoped that moving to DC would offer the fresh start they needed. I always looked forward to our dinner catch-ups during my visits, and it deeply saddens me to know we will never get the chance to do that again.

My heart goes out to Marisa’s family, friends, and everyone in our community who is mourning this loss. They will always be remembered.

Ben Millwood @ 2024-05-18T23:05 (+4)

do you know if Marisa exclusively used they/them pronouns, or she/they, or what? I remember hearing something about this but I'm not certain and can't find any online profiles anymore :(

Ben Millwood @ 2024-05-18T23:00 (+4) in response to Call for Attorneys for OpenAI Employees and Ex-Employees

Does anyone have guesses about how much it would cost to pay a California-licensed employment lawyer to form an opinion on this?

siobhanbrenton @ 2024-05-18T22:59 (+38) in response to Marisa, the Co-Founder of EA Anywhere, Has Passed Away 

Marisa was a beautiful soul who found great joy in helping others. As an exceptional generalist, I was confident that they would become a great operational leader once they were able to find inner peace.

Marisa began their journey with Rethink Charity as a volunteer and quickly became a fundamental part of the team. The two years they spent working on this project before starting grad school were transformational for RC. Marisa made me a better leader, and I am eternally grateful for the friendship we built.

The lockdowns were especially tough on Marisa, and I had hoped that moving to DC would offer the fresh start they needed. I always looked forward to our dinner catch-ups during my visits, and it deeply saddens me to know we will never get the chance to do that again.

My heart goes out to Marisa’s family, friends, and everyone in our community who is mourning this loss. They will always be remembered.

Vasco Grilo @ 2024-05-18T20:56 (+2) in response to The bullseye framework: My case against AI doom

For an agent to conquer to world, I think it would have to be close to the best across all those areas, but I think this is super unlikely based on it being super unlikely for a human to be close to the best across all those areas.

Erich_Grunewald @ 2024-05-18T22:22 (+6)

For an agent to conquer to world, I think it would have to be close to the best across all those areas

That seems right.

I think this is super unlikely based on it being super unlikely for a human to be close to the best across all those areas

I'm not sure that follows? I would expect improvements on these types of tasks to be highly correlated in general-purpose AIs. I think we've seen that with GPT-3 to GPT-4, for example: GPT-4 got better pretty much across the board (excluding the tasks that neither of them can do, and the tasks that GPT-3 could already do perfectly). That is not the case for a human who will typically improve in just one domain or a few domains from one year to the next, depending on where they focus their effort.

Larks @ 2024-05-18T05:14 (+35) in response to Articles about recent OpenAI departures

Kelsey suggests that OpenAI may be admitting defeat here:

OpenAI also says that going forward, they *won't* strip anyone of their equity for not signing the secret NDA, which is a bigger deal. I asked if this was a change of policy. ... "This statement reflects reality", replied OpenAI's spokesperson. To be fair it's a Friday night and I'm sure she's sick of me. But I have multiple ex-employees confirming this, if true, would be a big change of policy, presumably in response to backlash from current employees.

https://twitter.com/KelseyTuoc/status/1791691267941990764

Neel Nanda @ 2024-05-18T22:18 (+18)

Damage control, not defeat IMO. It's not defeat until they free previous leavers from unfair non disparagements/otherwise make it right to them

Larks @ 2024-05-18T01:34 (+29) in response to Articles about recent OpenAI departures

Sounds like it is time for someone to report them to the NLRB.

Linch @ 2024-05-18T22:17 (+2)

I'm not sure if you need standing to complain, but here's the relevant link.

harfe @ 2024-05-18T18:53 (+5) in response to Yanni Kyriacos's Quick takes

Is this about the safety teams at capabilities labs?

If so, I consider it a non-obvious issue, whether pushing a talented people into an AI safety role at, e.g., DeepMind is a bad thing. If you think that is a bad thing, consider providing a more detailed argument, and writing a top-level post explaining your view.

If, instead, this is about EA institutions pushing people into capabilities roles, consider naming these concrete examples. As an example, 80k has a job advertising a role as a prompt engineer at Scale AI. That does not seem to be a very safety-focused role, and it is not clear how 80k wants to help prevent human extinction with that job ad.

yanni kyriacos @ 2024-05-18T22:03 (–2)

Thanks for the comment :) I think expecting this detail is normal from someone on the EA Forum. If you think that’s the way public relations are going to work, which is my main point, then we’re disagreeing.

Owen Cotton-Barratt @ 2024-05-18T17:34 (+2) in response to "If we go extinct due to misaligned AI, at least nature will continue, right? ... right?"

I think this is a plausible consequence, but not a clear one.

Many people put significant value on conservation. It is plausible that some version of this would survive in an AI which was somewhat misaligned (especially since conservation might be a reasonably simple goal to point towards), such that it would spend some fraction of its resources towards preserving nature -- and one planet is a tiny fraction of the resources it could expect to end up with.

The most straightforward argument against this is that such an AI maybe wouldn't wipe out all humans. I tend to agree, and a good amount of my probability mass on "existential catastrophe from misaligned AI" does not involve human extinction. But I think there's some possible middle ground where an AI was not capable of reliably seizing power without driving humans extinct, but was capable if it allowed itself to do so, could wipe them out without eliminating nature (which would presumably pose much less threat to its ascendancy).

Owen Cotton-Barratt @ 2024-05-18T21:35 (+2)

I thought about where the logic in the post seemed to be going wrong, and it led me to write this quick take on why most possible goals of AI systems are partially concerned with process and not just outcomes.

Owen Cotton-Barratt @ 2024-05-18T21:32 (+4) in response to Owen Cotton-Barratt's Quick takes

Most possible goals for AI systems are concerned with process as well as outcomes.

People talking about possible AI goals sometimes seem to assume something like "most goals are basically about outcomes, not how you get there". I'm not entirely sure where this idea comes from, and I think it's wrong. The space of goals which are allowed to be concerned with process is much higher-dimensional than the space of goals which are just about outcomes, so I'd expect that on most reasonable sense of "most" process can have a look-in.

What's the interaction with instrumental convergence? (I'm asking because vibe-wise it seems like instrumental convergence is associated with an assumption that goals won't be concerned with process.)

  • Process-concerned goals could undermine instrumental convergence (since some process-concerned goals could be fundamentally opposed to some of the things that would otherwise get converged-to), but many process-concerned goals won't
  • Since instrumental convergence is basically about power-seeking, there's an evolutionary argument that you should expect the systems which end up with most power to have the power-seeking behaviours
    • I actually think there are a couple of ways for this argument to fail:
      1. If at some point you get a singleton, there's now no evolutionary pressure on its goals (beyond some minimum required to stay a singleton)
      2. A social environment can punish power-seeking, so that power-seeking behaviour is not the most effective way to arrive at power
        • (There are some complications to this I won't get into here)
    • But even if it doesn't fail, it pushes towards things which have Omuhundro's basic AI drives (and so pushes away from process-concerned goals which could preclude those), but it doesn't push all the way to purely outcome-concerned goals

In general I strongly expect humans to try to instil goals that are concerned with process as well as outcomes. Even if that goes wrong, I mostly expect them to end up something which has incorrect preferences about process, not something that doesn't care about process.

How could you get to purely outcome-concerned goals? I basically think this should be expected just if someone makes a deliberate choice to aim for that (though that might be possible via self-modification; the set of goals that would choose to self-modify to be purely outcome-concerned may be significantly bigger than the set of purely outcome-concerned goals). Overall I think purely outcome-concerned goals (or almost purely outcome-concerned goals) are a concern, and worth further consideration, but I really don't think they should be treated as a default.

Erich_Grunewald @ 2024-05-18T20:12 (+4) in response to The bullseye framework: My case against AI doom

Yes, that's true. Can you spell out for me what you think that implies in a little more detail?

Vasco Grilo @ 2024-05-18T20:56 (+2)

For an agent to conquer to world, I think it would have to be close to the best across all those areas, but I think this is super unlikely based on it being super unlikely for a human to be close to the best across all those areas.

Vasco Grilo @ 2024-05-18T20:00 (+2) in response to The bullseye framework: My case against AI doom

Hi Erich,

Note humans are also trained on all those abilities, but no single human is trained to be a specialist in all those areas. Likewise for AIs.

Erich_Grunewald @ 2024-05-18T20:12 (+4)

Yes, that's true. Can you spell out for me what you think that implies in a little more detail?

EffectiveAdvocate @ 2024-05-18T20:04 (+1) in response to EffectiveAdvocate's Quick takes

In the past few weeks, I spoke with several people interested in EA and wondered: What do others recommend in this situation in terms of media to consume first (books, blog posts, podcasts)?

Isn't it time we had a comprehensive guide on which introductory EA books or media to recommend to different people, backed by data?

Such a resource could consider factors like background, interests, and learning preferences, ensuring the most impactful material is suggested for each individual. Wouldn’t this tailored approach make promoting EA among friends and acquaintances more effective and engaging?

Erich_Grunewald @ 2023-05-30T14:26 (+9) in response to The bullseye framework: My case against AI doom

The flaws and bugs that are most relevant to an AI’s performance in it’s domain of focus will be weeded out, but flaws outside of it’s relevant domain will not be. Bobby Fischer’s insane conspiracism had no effect on his chess playing ability. The same principle applies to stockfish. “Idiot savant” AI’s are entirely plausible, even likely.

[...]

For these reasons, I expect AGI to be flawed, and especially flawed when doing things it was not originally meant to do, like conquer the entire planet.

We might actually expect an AGI to be trained to conquer the entire planet, or rather to be trained in many of the abilities needed to do so. For example, we may train it to be good at things like:

  • Strategic planning
  • Getting humans to do what it wants effectively
  • Controlling physical systems
  • Cybersecurity
  • Researching new, powerful technologies
  • Engineering
  • Running large organizations
  • Communicating with humans and other AIs

Put differently, I think "taking control over humans" and "running a multinational corporation" (which seems like the sort of thing people will want AIs to be able to do) have lots more overlap than "playing chess" and "having true beliefs about subjects of conspiracies". I'd be curious to hear if you have thoughts about which specific abilities you expect an AGI would need to have to take control over humanity that it's unlikely to actually possess?

Vasco Grilo @ 2024-05-18T20:00 (+2)

Hi Erich,

Note humans are also trained on all those abilities, but no single human is trained to be a specialist in all those areas. Likewise for AIs.

Vasco Grilo @ 2024-05-18T19:54 (+2) in response to The bullseye framework: My case against AI doom

Great post, titotal!

Point 1: Early AI will be buggy as hell 

Full article:

https://forum.effectivealtruism.org/posts/pXjpZep49M6GGxFQF/the-first-agi-will-be-a-buggy-mess

The link is broken.

yanni kyriacos @ 2024-05-17T23:09 (+8) in response to Yanni Kyriacos's Quick takes

Remember: EA institutions actively push talented people into the companies making the world changing tech the public have said THEY DONT WANT. This is where the next big EA PR crisis will come from (50%). Except this time it won’t just be the tech bubble.

harfe @ 2024-05-18T18:53 (+5)

Is this about the safety teams at capabilities labs?

If so, I consider it a non-obvious issue, whether pushing a talented people into an AI safety role at, e.g., DeepMind is a bad thing. If you think that is a bad thing, consider providing a more detailed argument, and writing a top-level post explaining your view.

If, instead, this is about EA institutions pushing people into capabilities roles, consider naming these concrete examples. As an example, 80k has a job advertising a role as a prompt engineer at Scale AI. That does not seem to be a very safety-focused role, and it is not clear how 80k wants to help prevent human extinction with that job ad.

jimrandomh @ 2024-05-18T04:28 (+67) in response to Articles about recent OpenAI departures

The language shown in this tweet says:

If the Grantee becomes a Withdrawn Limited Partner, then unless, within 60 days following its applicable Withdrawal Event, the Grantee ... duly executes and delivers to the Partnership a general release of claims against the Partnership and the other Partners with regard to all matters relating to the Partnership up to and including the time of such Withdrawal Event, such Grantee's Units shall be cancelled and reduced to zero ...

It's a trick!

Departing OpenAI employees are then offered a general release which meets the requirements of this section and also contains additional terms. What a departing OpenAI employee needs to do is have their own lawyer draft, execute, and deliver a general release which meets the requirements set forth. Signing the separation agreement is a mistake, and rejecting the separation agreement without providing your own general release is a mistake.

I could be misunderstanding this; I'm not a lawyer, just a person reading carefully. And there's a lot more agreement text that I don't have screenshots of. Still, I think the practical upshot is that departing OpenAI employees may be being tricked, and this particular trick seems defeatable to me. Anyone leaving OpenAI really needs a good lawyer.

Greg_Colbourn @ 2024-05-18T18:46 (+15)

See also: Call for Attorneys for OpenAI Employees and Ex-Employees

BrownHairedEevee @ 2024-05-18T18:29 (+8) in response to BrownHairedEevee's Quick takes

Are there currently any safety-conscious people on the OpenAI Board?

harfe @ 2024-05-18T18:25 (+2) in response to "If we go extinct due to misaligned AI, at least nature will continue, right? ... right?"

I just wanted to say that the new aisafety.info website looks great! I have not looked at everything in detail, just clicking around a bit, but the article seem of good quality to me.

I will probably mainly recommend aisafety as an introductory resource.

titotal @ 2024-05-17T23:32 (+7) in response to Why hasn't there been any significant AI protest

EA is a fairly small and weird social movement in the grand scheme of things. A protest movements consisting only of EAers will produce pathetically small protests, which might get some curious media writeups but will be unlikely to scare or influence anybody. 

If you actually want a big protest movement, you have to be willing to form coalitions with other groups. And that means playing nice with people like AI ethicists, rather than mocking and attacking them as has been unfortunately common here.  

sammyboiz @ 2024-05-18T18:15 (+1)

Thank you for your response. My impression is that big or small, every individuals additional contribution to a protest is roughly proportional to the impact of the protest. This meaning that its just as impactful for people to have small scale protests.

Geoffrey Miller @ 2024-05-17T19:21 (+5) in response to Why hasn't there been any significant AI protest

Good question. My hunch is that EA as a culture tends to prioritize epistemic and ethical sophistication and rigor, over direct 'political' action. And has traditionally avoided getting involved in issues that seem 'intractable' by virtue of being highly controversial and potentially partisan.

Against that background of EA's rather 'ivory tower' ethos, any direct protests may tend to be seen as rather simplistic, strident, and undignified -- even for issues such as animal agriculture where there's pretty strong EA consensus that factory farming is unethical. 

But I think it's time for EAs to climb down from our AI safety debates, recognize that the leading AI companies are not actually prioritizing safety, and start getting more involved in social media activism and in-person protests.

sammyboiz @ 2024-05-18T18:13 (+1)

Thank you for your response! Along with the EA community, I too am scared of doing activism for something controversial and bizarre like AI safety

Owen Cotton-Barratt @ 2024-05-18T17:34 (+2) in response to "If we go extinct due to misaligned AI, at least nature will continue, right? ... right?"

I think this is a plausible consequence, but not a clear one.

Many people put significant value on conservation. It is plausible that some version of this would survive in an AI which was somewhat misaligned (especially since conservation might be a reasonably simple goal to point towards), such that it would spend some fraction of its resources towards preserving nature -- and one planet is a tiny fraction of the resources it could expect to end up with.

The most straightforward argument against this is that such an AI maybe wouldn't wipe out all humans. I tend to agree, and a good amount of my probability mass on "existential catastrophe from misaligned AI" does not involve human extinction. But I think there's some possible middle ground where an AI was not capable of reliably seizing power without driving humans extinct, but was capable if it allowed itself to do so, could wipe them out without eliminating nature (which would presumably pose much less threat to its ascendancy).

Joseph Lemien @ 2024-05-18T17:28 (+14) in response to Joseph Lemien's Quick takes

I just looked at [ANONYMOUS PERSON]'s donations. The amount that this person has donated in their life is more than double the amount that I have ever earned in my life. This person appears to be roughly the same age as I am (we graduated from college ± one year of each other). Oof. It makes me wish that I had taken steps to become a software developer back when I was 15 or 18 or 22.

Oh, well. As they say, comparison is the thief of joy. I'll try to focus on doing the best I can with the hand I'm dealt.

huw @ 2024-05-18T00:38 (+1) in response to Why not socialism?

Here is a very long list of large, organised groups failing to engineer transitions to socialism within individual countries, because the United Stated were larger, more organised, and better-funded.

freedomandutility @ 2024-05-18T16:04 (+2)

Even more reason to think that transitioning to socialism is not tractable - some people will fight against it like hell!

Tessa @ 2024-05-18T15:13 (+2) in response to Biosecurity newsletters you should subscribe to

A new biosecurity-relevant newsletter (which me and Anemone put together) is GCBR Organization Updates. Every few months, we’ll ask organizations who are doing impactful work to reduce GCBRs to share their current projects, recent publications, and any opportunities for collaboration.

Frank_R @ 2024-05-18T05:38 (+3) in response to Questioning assumptions: Why the EA community should lead in the debate on brain preservation

The difference is that if you are biologically dead, there is nothing you can do to prevent a malevolant actor to upload your mind. If you are terminally ill and are pessimistic about the future, you can at least choose cremation.

I am not saying that there should be no funding for brain preservation, but personally I am not very enthusiastic since there is the danger that we will not solve the alignment problem.

AndyMcKenzie @ 2024-05-18T14:24 (+3)

I'm not sure I understand the scenario you are discussing. In your scenario, it sounds like you're positing a malevolent non-aligned AI that would forcibly upload and create suffering copies of people. Obviously, this is an almost unfathomably horrific hypothetical scenario which we should all try to prevent if we can. One thing I don't understand about the scenario you are describing is why this forcible uploading would only happen to people who are legally dead and preserved at the time, but not anyone living at the time. 

Ben Millwood @ 2024-05-18T13:35 (+17) in response to Marisa, the Co-Founder of EA Anywhere, Has Passed Away 

I cried when I read this. What an absolutely miserable thing to have happened.

NikhilVenkatesh @ 2024-05-16T16:29 (+3) in response to Why not socialism?

On the second point again: it seems like almost 30 people have commented on this post so far. Only 2, as far as I can tell, are sympathetic to socialism. The same number have made eugenicist arguments against me. This is kind of hard to reconcile with the survey numbers.

mhendric @ 2024-05-18T12:40 (+1)

I am similarly unenthused about the weird geneticism. 

Insofar as somewhat more altruism in the economy is the aim, sure, why not! I'm not opposed to that, and you may think that e.g. giving pledges or founders pledge are already steps in that direction. But that seems different from what most people think of when you say socialism, which they associate with ownership of means of production, or very heavy state interventionism and planned economy! It feels a tiny bit bailey and motte ish.

To give a bit of a hooray for the survey numbers - at the German unconference, I organized a fishbowl-style debate on economic systems. I was pretty much the only person defending a free market economy, with maybe 3-5 people silently supportive and a good 25 or so folks arguing for strong interventionism and socialism. I think this is pretty representative of the German EA community at least, so there may be country differences. 

Vasco Grilo @ 2024-05-18T11:31 (+2) in response to EA Forum feature suggestion thread

Hi JP,

It would be nice to have the possibility of filtering the posts of a user by a given tag. As of now, it is not possible.

Larks @ 2024-05-18T05:14 (+35) in response to Articles about recent OpenAI departures

Kelsey suggests that OpenAI may be admitting defeat here:

OpenAI also says that going forward, they *won't* strip anyone of their equity for not signing the secret NDA, which is a bigger deal. I asked if this was a change of policy. ... "This statement reflects reality", replied OpenAI's spokesperson. To be fair it's a Friday night and I'm sure she's sick of me. But I have multiple ex-employees confirming this, if true, would be a big change of policy, presumably in response to backlash from current employees.

https://twitter.com/KelseyTuoc/status/1791691267941990764

Rebecca @ 2024-05-18T09:48 (+12)

What about for people who’ve already resigned?

Linch @ 2022-09-22T22:05 (+12) in response to AGI Battle Royale: Why “slow takeover” scenarios devolve into a chaotic multi-AGI fight to the death

Related: Fearon 1995 from the IR literature. Basically, rational actors should only go to war against each other in a fairly limited set of scenarios. 

Vasco Grilo @ 2024-05-18T09:05 (+2)

For reference, here is a seemingly nice summary of Fearon's "Rationalist explanations for war" by David Patel.

Rasool @ 2024-05-18T08:35 (+2) in response to Rasool's Quick takes

Swapcard tips:

  1. The mobile browser is more reliable than the app

You can use Firefox/Safari/Chrome etc. on your phone, go to swapcard.com and use that instead of downloading the Swapcard app from your app store. As far as I know, the only thing the app has that the mobile site does not, is the QR code that you need when signing in when you first get to the venue and pick up your badge

  1. Only what you put in the 'Biography' section in the 'About Me' section of your profile is searchable when searching in Swapcard

The other fields, like 'How can I help others' and 'How can others help me' appear when you view someone's profile, but will not be used when searching using Swapcard search. This is another reason to use the Swapcard Attendee Google sheet that is linked-to in Swapcard to search

  1. You can use a (local!) LLM to find people to connect with

People might not want their data uploaded to a commercial large language model, but if you can run an open-source LLM locally, you can upload the Attendee Google sheet and use it to help you find useful contacts

Agustín Covarrubias @ 2024-05-17T21:31 (+2) in response to My Lament to EA

Could you clarify what you mean by margin-based reasoning in this context? 

Guy Raveh @ 2024-05-18T08:11 (+2)

Treating each new person as a separate investment and trying to optimize for their marginal utility for EA, instead of looking at the aggregate effect on the movement of all the community building efforts.

Specifically in your comment, justifying diversifying investment in groups by saying "high quality group members" are the goal but top universities have bottlenecks which can't be easily solved by just pouring more money into them - instead of arguing that it's better to have a new group in Chile than a new group in Harvard, even if hypothetically people there were less qualified for existing EA jobs.

AndyMcKenzie @ 2024-05-17T13:06 (+2) in response to Questioning assumptions: Why the EA community should lead in the debate on brain preservation

Good point. This is true for those who believe this, but it applies to any form of medicine or life extension, right? Not just brain preservation. So for someone who holds this view, theoretically it might also apply to the antimalarial medication case as well?

Frank_R @ 2024-05-18T05:38 (+3)

The difference is that if you are biologically dead, there is nothing you can do to prevent a malevolant actor to upload your mind. If you are terminally ill and are pessimistic about the future, you can at least choose cremation.

I am not saying that there should be no funding for brain preservation, but personally I am not very enthusiastic since there is the danger that we will not solve the alignment problem.

Larks @ 2024-05-18T05:14 (+35) in response to Articles about recent OpenAI departures

Kelsey suggests that OpenAI may be admitting defeat here:

OpenAI also says that going forward, they *won't* strip anyone of their equity for not signing the secret NDA, which is a bigger deal. I asked if this was a change of policy. ... "This statement reflects reality", replied OpenAI's spokesperson. To be fair it's a Friday night and I'm sure she's sick of me. But I have multiple ex-employees confirming this, if true, would be a big change of policy, presumably in response to backlash from current employees.

https://twitter.com/KelseyTuoc/status/1791691267941990764

jimrandomh @ 2024-05-18T04:28 (+67) in response to Articles about recent OpenAI departures

The language shown in this tweet says:

If the Grantee becomes a Withdrawn Limited Partner, then unless, within 60 days following its applicable Withdrawal Event, the Grantee ... duly executes and delivers to the Partnership a general release of claims against the Partnership and the other Partners with regard to all matters relating to the Partnership up to and including the time of such Withdrawal Event, such Grantee's Units shall be cancelled and reduced to zero ...

It's a trick!

Departing OpenAI employees are then offered a general release which meets the requirements of this section and also contains additional terms. What a departing OpenAI employee needs to do is have their own lawyer draft, execute, and deliver a general release which meets the requirements set forth. Signing the separation agreement is a mistake, and rejecting the separation agreement without providing your own general release is a mistake.

I could be misunderstanding this; I'm not a lawyer, just a person reading carefully. And there's a lot more agreement text that I don't have screenshots of. Still, I think the practical upshot is that departing OpenAI employees may be being tricked, and this particular trick seems defeatable to me. Anyone leaving OpenAI really needs a good lawyer.

andrewpei @ 2024-05-18T04:26 (+8) in response to Marisa, the Co-Founder of EA Anywhere, Has Passed Away 

I am shocked and saddened. I did not know Marisa well but we were in the same EA Anywhere discussion group for several months. As you said she was quite funny and I enjoyed talking with her and hearing her ideas. 

jimrandomh @ 2024-05-18T04:20 (+38) in response to Articles about recent OpenAI departures

According to Kelsey's article, OpenAI employees are coerced into signing lifelong nondisparagement agreements, which also forbid discussion of the nondisparagement agreements themselves, under threat of losing all of their equity.

This is intensely contrary to the public interest, and possibly illegal. Enormous kudos for bringing it to light.

In a legal dispute initiated by an OpenAI employee, the most important thing would probably be what representations were previously made about the equity. That's hard for me to evaluate, but if it's true that they were presented as compensation and the nondisparagement wasn't disclosed, then rescinding those benefits could be a breach of contract. However, I'm not sure if this would apply if this was threatened but the threat wasn't actually executed.

CA GOV § 12964.5 and 372 NLRB No. 58 also offer some angles by which former OpenAI employees might fight this in court.

CA GOV § 12964.5 talks specifically about disclosure of "conduct that you have reason to believe is unlawful." Generically criticizing OpenAI as pursuing unsafe research would not qualify unless (the speaker believes) it rises to the level of criminal endangerment, or similar. Copyright issues would *probably* qualify. Workplace harrassment would definitely qualify.

(No OpenAI employees have alleged any of these things publicly, to my knowledge)

372 NLRB No. 58 nominally invalidates separation agreements that contain nondisparagement clauses, and that restrict discussion of the terms of the separation agreement itself. However, it's specifically focused on the effect on collective bargaining rights under the National Labor Relations Act, which could make it inapplicable.

JackM @ 2024-05-17T22:33 (+5) in response to Ilya Sutskever has officially left OpenAI

Hmm, I don't see why ensuring the best people go to Anthropic necessarily means they will take safety less seriously. I can actually imagine the opposite effect as if Anthropic catches up or even overtakes OpenAI then their incentive to cut corners should actually decrease because it's more likely that they can win the race without cutting corners. Right now their only hope to win the race is to cut corners.

Ultimately what matters most is what the leadership's views are. I suspect that Sam Altman never really cared that much about safety, but my sense is that the Amodeis do.

Ryan Greenblatt @ 2024-05-18T04:15 (+6)

Ultimately what matters most is what the leadership's views are.

I'm skeptical this is true particularly as AI companies grow massively and require vast amounts of investment.

It does seem important, but unclear it matters most.

MichaelStJules @ 2024-05-17T03:53 (+10) in response to 35-150 billion fish are raised in captivity to be released into the wild every year

 

Appendix: huge numbers of juveniles raised for an unknown reason

 

I suspect they're raised as feed for other farmed fish (and maybe other farmed aquatic species). Maybe they could also be released into wild fisheries as feed for wild-caught aquatic animals.

From Li and Xia, 2018, first some updated numbers:

Aquaculture production was 28.02 million tonnes in 2013, and the corresponding production of artificially propagated fry was 1 914.3 billion.

And they have a figure:

 

They have a section "6.4.4.2 Fry Production of Prey Fish as Feed for Predatory Fish". They specifically discuss mud carp fry as feed for mandarin fish (Siniperca chuatsi). They write:

6.4.4.2.3 The Relationship Between the Production of Mud Carp Fry and Mandarin Fish

Mud carp, is the favorite prey fish of mandarin fish. The production of mandarin fish has increased in relation to the growth of mud carp culture. The production of mud carp per growth cycle is about 7500 kg/ha, while mandarin fish was about 6000–7500 kg/ha, and feed coefficient was about 1:3–4. When the feed coefficient is 3.5, the production of mandarin fish was 284 780 tonnes in 2013, and required a prey fish production of mud carp of about 996 730 tonnes. Almost all prey for mandarin fish is provided through artificial propagation. The production of mandarin fish has increased over the years, and is significantly positively correlated with fry availability (Figure 6.4.9) (Pearson correlation = 0.70, P < 0.01) (China Fishery Statistical Yearbook 1996–2014). As a high‐quality food for mandarin fish, the variation in production of mud carp is directly related to the aquaculture scale of mandarin fish, as shown from the example in Guangdong Province (Figure 6.4.10). Here again, there is a significant linear correlation between the production of mandarin fish and mud carp (y = 0.348x – 48057, R2 = 0.765, P < 0.05) (Yao 1999).

(Though looking at the figures 6.4.9 and 6.4.10, fry production and mandarin fish production don't look very closely related, and it could just be all aquaculture going up.)

 

https://thefishsite.com/articles/cultured-aquatic-species-mandarin-fish, https://www.fao.org/fishery/affris/species-profiles/mandarin-fish/faqs/en/, and https://www.fao.org/fishery/affris/species-profiles/mandarin-fish/natural-food-and-feeding-habits/en/ also discuss fish fry fed live to mandarin fish.

MichaelStJules @ 2024-05-18T03:28 (+2)

I'm writing a quick piece on the scale, in case you (or anyone else) is interested in giving feedback before I post it (probably next week).

Ozzie Gooen @ 2024-05-18T03:00 (+11) in response to Marisa, the Co-Founder of EA Anywhere, Has Passed Away 

I've known Marisa for a few years and had the privilege of briefly working with her. I was really impressed by her drive and excitement. She seemed deeply driven and was incredibly friendly to be around. 

This will take me some time to process. I'm so sorry it ended like this. 

She will be remembered.

Agustín Covarrubias @ 2024-05-17T21:26 (+4) in response to Branding AI Safety Groups: A Field Guide

I agree that it's important to consider both needs and interests. Ultimately, a branding strategy should be embedded in a larger theory of change and strategy for your group, and that should determine which audiences you reach out to.
 

Regarding the latter, I agree that an interest in, say, hacker culture, does not adequately describe all people interested in CS. It might actually leave out a bunch of people that you should care about joining our group. At the same time, branding is all about tradeoffs, and you have to pick which things you cater to. Spread too thin, and you risk making the content too unappealing.

Ulrik Horn @ 2024-05-18T02:40 (+2)

Yeah or maybe you could do like with toothpaste: one for white teeth, another for good breath. I think it's the same toothpaste in both tubes.

Julia_Wise @ 2024-05-18T02:23 (+65) in response to Marisa, the Co-Founder of EA Anywhere, Has Passed Away 

I was so sorry to learn this.

Some other resources:
5 steps to help someone who may be suicidal
Crisis resources around the world

Years ago Marisa was the first person to put in an application for several EA Globals, to where I was curious if she had some kind of notification set up. I asked her about it once, and she was surprised to hear that she’d been first; she was just very keen.

Linch @ 2024-05-18T00:19 (+25) in response to Articles about recent OpenAI departures

This feels really suss to me:

Many people at OpenAI get more of their compensation from PPUs than from base salary. PPUs can only be sold at tender offers hosted by the company. When you join OpenAI, you sign onboarding paperwork laying all of this out.

And that onboarding paperwork says you have to sign termination paperwork with a 'general release' within sixty days of departing the company. If you don't do it within 60 days, your units are cancelled. No one I spoke to at OpenAI gave this little line much thought.

And yes this is talking about vested units, because a separate clause clarifies that unvested units just transfer back to the control of OpenAI when an employee undergoes a termination event (which is normal).

There's a common legal definition of a general release, and it's just a waiver of claims against each other. Even someone who read the contract closely might be assuming they will only have to sign such a waiver of claims.

But when you actually quit, the 'general release'? It's a long, hardnosed, legally aggressive contract that includes a confidentiality agreement which covers the release itself, as well as arbitration, nonsolicitation and nondisparagement and broad 'noninterference' agreement.

And if you don't sign within sixty days your units are gone. And it gets worse - because OpenAI can also deny you access to the annual events that are the only way to sell your vested PPUs at their discretion, making ex-employees constantly worried they'll be shut out.

Larks @ 2024-05-18T01:34 (+29)

Sounds like it is time for someone to report them to the NLRB.

Thomas Kwa @ 2024-05-17T23:21 (+23) in response to Tyler Johnston's Quick takes

I want to slightly push back against this post in two ways:

  • I do not think longtermism is any sort of higher form of care or empathy. Many longtermist EAs are motivated by empathy, but they are also driven by a desire for philosophical consistency, beneficentrism and scope-sensitivity that is uncommon among the general public. Many are also not motivated by empathy-- I think empathy plays some role for me but is not the primary motivator? Cold utilitarianism is more important but not the primary motivator either [1]. I feel much more caring when I cook dinner for my friends than when I do CS research, and it is only because I internalize scope sensitivity more than >99% of people that I can turn empathy into any motivation whatsoever to work on longtermist projects. I think that for most longtermists, it is not more empathy, nor a better form of empathy, but the interaction of many normal (often non-empathy) altruistic motivators and other personality traits that makes them longtermists.
  • Longtermists make tradeoffs between other common values and helping vast future populations that most people disagree with, and without ideosyncratic EA values there is no reason that a caring person should make the same tradeoffs as longtermists. I think the EA value of "doing a lot more good matters a lot more" is really important, but it is still trading off against other values.
    • Helping people closer to you / in your community: many people think this has inherent value
    • Beneficentrism: most people think there is inherent value in being directly involved in helping people. Habitat for Humanity is extremely popular among caring and empathic people, and they would mostly not think it is better to make more of an overall difference by e.g. subsidizing eyeglasses in Bangladesh.
    • Justice: most people think it is more important to help one human trafficking victim than one tuberculosis victim or one victim of omnicidal AI if you create the same welfare, because they place inherent value on justice. Both longtermists and GiveWell think they're similarly good modulo secondary consequences and decision theory.
    • Discount rate, risk aversion, etc.: There is no reason that having a 10% chance of saving 100 lives in 6,000 years is better than a 40% chance of saving 5 lives tomorrow, if you don't already believe in zero-discount expected value as the metric to optimize. The reason to believe in zero-discount expected value is a thought experiment involving the veil of ignorance, or maybe the VNM theorem. It is not caring doing the work here because both can be very caring acts, it is your belief in the thought experiment connecting your caring to the expected value.

In conclusion, I think that while care and empathy can be an important motivator to longtermists, and it is valid for us to think of longtermist actions as the ultimate act of care, we are motivated by a conjunction of empathy/care and other attributes, and it is the other attributes that are by far more important. For someone who has empathy/care and values beneficentrism and scope-sensitivity, preventing an extinction-level pandemic is an important act of care; for someone like me or a utilitarian, pandemic prevention is also an important act. But for someone who values justice more, applying more care does not make them prioritize pandemic prevention over helping a sex trafficking victim, and in the larger altruistically-inclined population, I think a greater focus on care and empathy conflict with longtermist values more than they contribute.

[1] More important for me are: feeling moral obligation to make others' lives better rather than worse, wanting to do my best when it matters, wanting future glory and social status for producing so much utility.

Tyler Johnston @ 2024-05-18T01:21 (+6)

Thanks for this reply — it does resonate with me. It actually got me thinking back to Paul Bloom's Against Empathy book, and how when I read that I thought something like: "oh yeah empathy really isn't the best guide to acting morally," and whether that view contradicts what I was expressing in my quick take above.

I think I probably should have framed the post more as "longtermism need not be totally cold and utilitarian," and that there's an emotional, caring psychological relationship we can have to hypothetical future people because we can imaginatively put ourselves in their shoes. And that it might even incorporate elements of justice or fairness if we consider them a disenfranchised group without representation in today's decision making who we are potentially throwing under the bus for our own benefit, or something like that. So justice and empathy can easily be folded into longtermist thinking. This sounds like what you are saying here, except maybe I do want to stand by the fact that EA values aren't necessarily trading off against justice, depending on how you define it.

Garrison @ 2024-05-17T23:33 (+24) in response to Why not socialism?

(Writing this quickly and while very sleep deprived).

I really appreciate the OP for so clearly making the case for such a big idea and everyone’s engagement with it. That said, it’s a bummer that maybe the most common/upvoted reply on the EA forum to pro-left-wing arguments is something like this because it assumes that socialism is just about making the government bigger, but it’s not, at least not necessarily. There are lots of different definitions of socialism, but I think the common thread is: a system that aims to empower the working class to build an alternative to capitalism. The most compelling and practical vision of this to me is Jacobin founder Bhaskar Sunkara’s. His appearance on Lex Fridman is a (relatively) short articulation, and his book The Socialist Manifesto goes into more detail.

(An uncomfortable implication of the above commenter’s perspective is that we should redistribute more money from the poor to the rich, on the off chance they put it toward effective causes.)

I don’t blame people for thinking socialism = more government, because, at least in the US, education on the topic is extremely bad (we did have a whole Cold War and all). 

Some examples of policies that push in a more socialist direction that don’t necessarily involve growing the government:

  • Worker codetermination on corporate boards (common in Germany, which has a strong economy and a far more equal distribution of wealth than the US)
  • Worker cooperatives
  • Participatory budgeting

And there are plenty of socialist-y policies that would grow the public sector but in a directed way to improve welfare for lots of people, like:

  • Public banking
  • Green subsidies
  • Public options for natural monopolies like fiber optic internet
  • Single-payer or nationalized healthcare

If you look at rich countries, there is a strong positive association between left-wing policies and citizen wellbeing. I think it’s worth noting that the book linked is pretty clearly written with a serious pro-market slant (as is the comment). At a glance, the book doesn’t appear to get into examples of socialist/leftist movements in Europe, the US or Canada. But these movements and the results of their policies are far more relevant to any discussion of socialism in rich countries with strongly developed civil societies (where most EAs live). Ignoring Europe, and Scandinavia in particular is cherry-picking.

Further, almost no socialists I know are advocating for a command economy like the Soviet Union, but rather things like the above. 

In general on the forum, it feels like capitalism-sympathetic views are treated with far less scrutiny than left-wing views.

(If anyone's curious, I discussed EA and the left with Habiba Banu on my podcast a while back.)

Ebenezer Dukakis @ 2024-05-18T01:16 (+5)

Thanks for the response, upvoted.

socialism is just about making the government bigger

OP framed socialism in terms of resource reallocation. ("The global economy’s current mode of allocating resources is suboptimal" was a key point, which yes, sounded like advocacy for a command economy.) I'm trying to push back on millenarian thinking that 'socialism' is a magic wand which will improve resource allocation.

If your notion of 'socialism' is favorable tax treatment for worker-owned cooperatives or something, that could be a good thing if there's solid evidence that worker-owned cooperatives achieve better outcomes, but I doubt it would qualify as a top EA cause.

(An uncomfortable implication of the above commenter’s perspective is that we should redistribute more money from the poor to the rich, on the off chance they put it toward effective causes.)

Here in EA, GiveDirectly (cash transfers for the poor) is considered a top EA cause. It seems fairly plausible to me that if the government cut a bunch of non-evidence-backed school and work programs and did targeted, temporary direct cash transfers instead, that would be an improvement.

If you look at rich countries, there is a strong positive association between left-wing policies and citizen wellbeing.

I'm skimming the post you linked and it doesn't look especially persuasive. Inferring causation from correlation is notoriously difficult, and these relationships don't look particularly robust. (Interesting that r^2=0.29 appears to be the only correlation coefficient specified in the article -- that's not a strong association!)

As an American, I don't particularly want America to move in the direction of a Nordic-style social democracy, because Americans are already very well off. In 2023, the US had the world's second highest median income adjusted for cost of living, right after Luxembourg. From a poverty-reduction perspective, the US government should be focused on effective foreign aid and facilitating immigration.

Similarly, from a global poverty reduction perspective, we should be focused on helping poor countries. If "socialism" tends to be good for rich countries but bad for poor countries, that suggests it is the wrong tool to reduce global poverty.

freedomandutility @ 2024-05-17T18:27 (+3) in response to Why not socialism?

"Thinking in terms of group rather than individual agency makes transition from capitalism to socialism appear more tractable."

I disagree. There is a long history of large, organised, and well-funded groups failing to engineer transitions to socialism within individual countries, let alone a global transition to socialism.

huw @ 2024-05-18T00:38 (+1)

Here is a very long list of large, organised groups failing to engineer transitions to socialism within individual countries, because the United Stated were larger, more organised, and better-funded.

Linch @ 2024-05-18T00:19 (+25) in response to Articles about recent OpenAI departures

This feels really suss to me:

Many people at OpenAI get more of their compensation from PPUs than from base salary. PPUs can only be sold at tender offers hosted by the company. When you join OpenAI, you sign onboarding paperwork laying all of this out.

And that onboarding paperwork says you have to sign termination paperwork with a 'general release' within sixty days of departing the company. If you don't do it within 60 days, your units are cancelled. No one I spoke to at OpenAI gave this little line much thought.

And yes this is talking about vested units, because a separate clause clarifies that unvested units just transfer back to the control of OpenAI when an employee undergoes a termination event (which is normal).

There's a common legal definition of a general release, and it's just a waiver of claims against each other. Even someone who read the contract closely might be assuming they will only have to sign such a waiver of claims.

But when you actually quit, the 'general release'? It's a long, hardnosed, legally aggressive contract that includes a confidentiality agreement which covers the release itself, as well as arbitration, nonsolicitation and nondisparagement and broad 'noninterference' agreement.

And if you don't sign within sixty days your units are gone. And it gets worse - because OpenAI can also deny you access to the annual events that are the only way to sell your vested PPUs at their discretion, making ex-employees constantly worried they'll be shut out.

Christina McCaffrey @ 2024-05-18T00:00 (+1) in response to Suggested readings & videos for a new college course on 'Psychology and AI'?

There's a fabulous book called God, Human, Animal, Machine written by Meghan O'Gleblyn that tackled the idea of the rise of AI and whether or not we could consider consciousness. I am fascinated with AI and I'm obsessed with how humans think (psychology degree) and I found it such an interesting read because it takes what we know and uses it to question the impact of AI. The greatest thing about the book is that the author struggled with her family of origin's religious ideology and now she's facing the same big questions about the meaning of consciousness and life now with the invention of AI. Read the reviews for more insight but I think it may be a good introduction from a bigger picture that includes some risk assessment and also some questions on an even bigger scale about the role of humanity.



Comments on 2024-05-17

Ebenezer Dukakis @ 2024-05-16T01:36 (+37) in response to Why not socialism?
  1. The global economy’s current mode of allocating resources is suboptimal. (Otherwise, why would effective altruism be necessary?)

The US government spent about $6.1 trillion in 2023 alone. That's over 40x Bill Gates' current net worth. Very little of that $6.1 trillion went to top EA causes.

[Edit: Here is an interesting 2015 quote regarding US government spending, from Vox of all sources: "A couple of years ago, former Obama and Bush officials estimated that only 1 percent of government spending is backed by any evidence at all ... Perhaps unsurprisingly, then, evaluations of government-sponsored school and work programs have found that some three-quarters of those have no effect." Maybe I would be more enthusiastic about socialism if this were addressed, but fundamentally it seems like a tricky incentives problem.]

The strategy of "take money from rich capitalists and have citizens vote on how to allocate it" doesn't seem to result in anything like effective altruism. $6.1 trillion is already an incomprehensibly large amount. I don't see how increasing it would change things.

I don't favor increasing the government's budget unless the government is spending money well.

  1. Individuals and institutions can be motivated to change their behaviour for the better on the basis of concern for others. (Otherwise, how could effective altruism be possible?)

My sense is that most people who hear about effective altruism aren't going to become effective altruists. EA doesn't have some sort of magic pill to distribute that makes you want to help people or animals who exist far away in time or space. EA recruitment is more about identifying (fairly rare) individuals in the general population who are interested in that stuff.

If this sort of mass behavior change was somehow possible at the flip of a switch, socialism wouldn't be necessary anyways. People would voluntarily be altruistic. No need to make it compulsory.

Why not a socialist alternative, that is, one in which people are motivated to a greater extent by altruism and a lesser extent by self-interest?

I don't think socialism will change the rate of greed in the general population. It will just redirect the greed towards grabbing a bigger share of the redistribution pie. The virtue of capitalism is that it harnesses greed in a way that often has beneficial effects for society. ("It is not from the benevolence of the butcher, the brewer, or the baker that we expect our dinner, but from their regard to their own self-interest.")

And some socialist economies have had some successes (human development in Kerala, economic growth in China, the USSR’s role in space technology and smallpox eradication, Cuba’s healthcare system).

Historically speaking, socialists often endorse economic systems that end up failing, but after they fail socialists forget they originally endorsed them. I think it's important for those cases to be included in the dataset too. See this book.

EAs should be more willing to fund and conduct research into alternative economic systems, socialist ones included.

Yep, I favor voluntary charter cities to experiment with alternative economic systems on a small scale, and I support folks who are trying to think rigorously about alternative systems, such as radicalxchange. The big thing socialism lacks is a small-scale, working proof of concept. Without a compelling and robust proof of concept, advocating for radical changes to big developed countries which already function fairly well in the grand scheme of things seems irresponsible.

Garrison @ 2024-05-17T23:33 (+24)

(Writing this quickly and while very sleep deprived).

I really appreciate the OP for so clearly making the case for such a big idea and everyone’s engagement with it. That said, it’s a bummer that maybe the most common/upvoted reply on the EA forum to pro-left-wing arguments is something like this because it assumes that socialism is just about making the government bigger, but it’s not, at least not necessarily. There are lots of different definitions of socialism, but I think the common thread is: a system that aims to empower the working class to build an alternative to capitalism. The most compelling and practical vision of this to me is Jacobin founder Bhaskar Sunkara’s. His appearance on Lex Fridman is a (relatively) short articulation, and his book The Socialist Manifesto goes into more detail.

(An uncomfortable implication of the above commenter’s perspective is that we should redistribute more money from the poor to the rich, on the off chance they put it toward effective causes.)

I don’t blame people for thinking socialism = more government, because, at least in the US, education on the topic is extremely bad (we did have a whole Cold War and all). 

Some examples of policies that push in a more socialist direction that don’t necessarily involve growing the government:

  • Worker codetermination on corporate boards (common in Germany, which has a strong economy and a far more equal distribution of wealth than the US)
  • Worker cooperatives
  • Participatory budgeting

And there are plenty of socialist-y policies that would grow the public sector but in a directed way to improve welfare for lots of people, like:

  • Public banking
  • Green subsidies
  • Public options for natural monopolies like fiber optic internet
  • Single-payer or nationalized healthcare

If you look at rich countries, there is a strong positive association between left-wing policies and citizen wellbeing. I think it’s worth noting that the book linked is pretty clearly written with a serious pro-market slant (as is the comment). At a glance, the book doesn’t appear to get into examples of socialist/leftist movements in Europe, the US or Canada. But these movements and the results of their policies are far more relevant to any discussion of socialism in rich countries with strongly developed civil societies (where most EAs live). Ignoring Europe, and Scandinavia in particular is cherry-picking.

Further, almost no socialists I know are advocating for a command economy like the Soviet Union, but rather things like the above. 

In general on the forum, it feels like capitalism-sympathetic views are treated with far less scrutiny than left-wing views.

(If anyone's curious, I discussed EA and the left with Habiba Banu on my podcast a while back.)

MichaelStJules @ 2024-05-17T03:53 (+10) in response to 35-150 billion fish are raised in captivity to be released into the wild every year

 

Appendix: huge numbers of juveniles raised for an unknown reason

 

I suspect they're raised as feed for other farmed fish (and maybe other farmed aquatic species). Maybe they could also be released into wild fisheries as feed for wild-caught aquatic animals.

From Li and Xia, 2018, first some updated numbers:

Aquaculture production was 28.02 million tonnes in 2013, and the corresponding production of artificially propagated fry was 1 914.3 billion.

And they have a figure:

 

They have a section "6.4.4.2 Fry Production of Prey Fish as Feed for Predatory Fish". They specifically discuss mud carp fry as feed for mandarin fish (Siniperca chuatsi). They write:

6.4.4.2.3 The Relationship Between the Production of Mud Carp Fry and Mandarin Fish

Mud carp, is the favorite prey fish of mandarin fish. The production of mandarin fish has increased in relation to the growth of mud carp culture. The production of mud carp per growth cycle is about 7500 kg/ha, while mandarin fish was about 6000–7500 kg/ha, and feed coefficient was about 1:3–4. When the feed coefficient is 3.5, the production of mandarin fish was 284 780 tonnes in 2013, and required a prey fish production of mud carp of about 996 730 tonnes. Almost all prey for mandarin fish is provided through artificial propagation. The production of mandarin fish has increased over the years, and is significantly positively correlated with fry availability (Figure 6.4.9) (Pearson correlation = 0.70, P < 0.01) (China Fishery Statistical Yearbook 1996–2014). As a high‐quality food for mandarin fish, the variation in production of mud carp is directly related to the aquaculture scale of mandarin fish, as shown from the example in Guangdong Province (Figure 6.4.10). Here again, there is a significant linear correlation between the production of mandarin fish and mud carp (y = 0.348x – 48057, R2 = 0.765, P < 0.05) (Yao 1999).

(Though looking at the figures 6.4.9 and 6.4.10, fry production and mandarin fish production don't look very closely related, and it could just be all aquaculture going up.)

 

https://thefishsite.com/articles/cultured-aquatic-species-mandarin-fish, https://www.fao.org/fishery/affris/species-profiles/mandarin-fish/faqs/en/, and https://www.fao.org/fishery/affris/species-profiles/mandarin-fish/natural-food-and-feeding-habits/en/ also discuss fish fry fed live to mandarin fish.

MichaelStJules @ 2024-05-17T23:32 (+10)

Well fuck, I guess this probably explains it. Yao & Li, 2018:

Mandarin fish have unusual feeding habits. The fish only eat live fish and shrimps, and do not consume dead prey or artificial diets during all lifecycle stages (Chiang 1959; Li et al. 2014a; Yao and Liang 2015). In nature it is completely carnivorous, and has been found to capture live fry of other fish species from the first feeding stages (Chiang 1959).

Also makes substitutes for fish fry not very promising; they'd probably also have to be other animals. But maybe we could find some that matter much less per kg.

Otherwise, we'd probably just want to reduce mandarin fish production, which could be hard to target specifically, especially being in China.

 

Some different fry numbers in Hu et al., 2021:

According to data from the China Fishery Statistical Yearbook, the fry number of freshwater fish increased from 59.51 billion in 1981 to 1.252 trillion in 2019, while the fry number of marine fish increased from 167 million in 1996 to 11.44 billion in 2019 [6,7].

titotal @ 2024-05-17T23:32 (+7) in response to Why hasn't there been any significant AI protest

EA is a fairly small and weird social movement in the grand scheme of things. A protest movements consisting only of EAers will produce pathetically small protests, which might get some curious media writeups but will be unlikely to scare or influence anybody. 

If you actually want a big protest movement, you have to be willing to form coalitions with other groups. And that means playing nice with people like AI ethicists, rather than mocking and attacking them as has been unfortunately common here.  

Tyler Johnston @ 2024-05-16T03:00 (+68) in response to Tyler Johnston's Quick takes

This is a cold take that’s probably been said before, but I thought it bears repeating occasionally, if only for the reminder:

The longtermist viewpoint has gotten a lot of criticism for prioritizing “vast hypothetical future populations” over the needs of "real people," alive today. The mistake, so the critique goes, is the result of replacing ethics with math, or utilitarianism, or something cold and rigid like that. And so it’s flawed because it lacks the love or duty or "ethics of care" or concern for justice that lead people to alternatives like mutual aid and political activism.

My go-to reaction to this critique has become something like “well you don’t need to prioritize vast abstract future generations to care about pandemics or nuclear war, those are very real things that could, with non-trivial probability, face us in our lifetimes.” I think this response has taken hold in general among people who talk about X-risk. This probably makes sense for pragmatic reasons. It’s a very good rebuttal to the “cold and heartless utilitarianism/pascal's mugging” critique.

But I think it unfortunately neglects the critical point that longtermism, when taken really seriously — at least the sort of longtermism that MacAskill writes about in WWOTF, or Joe Carlsmith writes about in his essays — is full of care and love and duty. Reading the thought experiment that opens the book about living every human life in sequential order reminded me of this. I wish there were more people responding to the “longtermism is cold and heartless” critique by making the case that no, longtermism at face value is worth preserving because it's the polar opposite of heartless. Caring about the world we leave for the real people, with emotions and needs and experiences as real as our own, who very well may inherit our world but who we’ll never meet, is an extraordinary act of empathy and compassion — one that’s way harder to access than the empathy and warmth we might feel for our neighbors by default. It’s the ultimate act of care. And it’s definitely concerned with justice.

(I mean, you can also find longtermism worthy because of something something math and cold utilitarianism. That’s not out of the question. I just don’t think it’s the only way to reach that conclusion.)

Thomas Kwa @ 2024-05-17T23:21 (+23)

I want to slightly push back against this post in two ways:

  • I do not think longtermism is any sort of higher form of care or empathy. Many longtermist EAs are motivated by empathy, but they are also driven by a desire for philosophical consistency, beneficentrism and scope-sensitivity that is uncommon among the general public. Many are also not motivated by empathy-- I think empathy plays some role for me but is not the primary motivator? Cold utilitarianism is more important but not the primary motivator either [1]. I feel much more caring when I cook dinner for my friends than when I do CS research, and it is only because I internalize scope sensitivity more than >99% of people that I can turn empathy into any motivation whatsoever to work on longtermist projects. I think that for most longtermists, it is not more empathy, nor a better form of empathy, but the interaction of many normal (often non-empathy) altruistic motivators and other personality traits that makes them longtermists.
  • Longtermists make tradeoffs between other common values and helping vast future populations that most people disagree with, and without ideosyncratic EA values there is no reason that a caring person should make the same tradeoffs as longtermists. I think the EA value of "doing a lot more good matters a lot more" is really important, but it is still trading off against other values.
    • Helping people closer to you / in your community: many people think this has inherent value
    • Beneficentrism: most people think there is inherent value in being directly involved in helping people. Habitat for Humanity is extremely popular among caring and empathic people, and they would mostly not think it is better to make more of an overall difference by e.g. subsidizing eyeglasses in Bangladesh.
    • Justice: most people think it is more important to help one human trafficking victim than one tuberculosis victim or one victim of omnicidal AI if you create the same welfare, because they place inherent value on justice. Both longtermists and GiveWell think they're similarly good modulo secondary consequences and decision theory.
    • Discount rate, risk aversion, etc.: There is no reason that having a 10% chance of saving 100 lives in 6,000 years is better than a 40% chance of saving 5 lives tomorrow, if you don't already believe in zero-discount expected value as the metric to optimize. The reason to believe in zero-discount expected value is a thought experiment involving the veil of ignorance, or maybe the VNM theorem. It is not caring doing the work here because both can be very caring acts, it is your belief in the thought experiment connecting your caring to the expected value.

In conclusion, I think that while care and empathy can be an important motivator to longtermists, and it is valid for us to think of longtermist actions as the ultimate act of care, we are motivated by a conjunction of empathy/care and other attributes, and it is the other attributes that are by far more important. For someone who has empathy/care and values beneficentrism and scope-sensitivity, preventing an extinction-level pandemic is an important act of care; for someone like me or a utilitarian, pandemic prevention is also an important act. But for someone who values justice more, applying more care does not make them prioritize pandemic prevention over helping a sex trafficking victim, and in the larger altruistically-inclined population, I think a greater focus on care and empathy conflict with longtermist values more than they contribute.

[1] More important for me are: feeling moral obligation to make others' lives better rather than worse, wanting to do my best when it matters, wanting future glory and social status for producing so much utility.

Peter Wildeford @ 2024-05-17T23:20 (+36) in response to Marisa, the Co-Founder of EA Anywhere, Has Passed Away 

Deeply saddened to hear this. We worked together on Rethink Charity. This loss is incredibly painful.

yanni kyriacos @ 2024-05-17T23:09 (+8) in response to Yanni Kyriacos's Quick takes

Remember: EA institutions actively push talented people into the companies making the world changing tech the public have said THEY DONT WANT. This is where the next big EA PR crisis will come from (50%). Except this time it won’t just be the tech bubble.

Holden Karnofsky @ 2023-03-21T16:37 (+5) in response to My takes on the FTX situation will (mostly) be cold, not hot

I don’t believe #1 is correct. The Open Philanthropy grant is a small fraction of the funding OpenAI has received, and I don’t think it was crucial for OpenAI at any point.

I think #2 is fair insofar as running a scaling lab poses big risks to the world. I hope that OpenAI will avoid training or deploying directly dangerous systems; I think that even the deployments it’s done so far pose risks via hype and acceleration. (Considering the latter a risk to society is an unusual standard to hold a company to, but I think it’s appropriate here.)

#3 seems off to me - “regulatory capture” does not describe what’s at the link you gave (where’s the regulator?) At best it seems like a strained analogy, and even there it doesn’t seem right to me - I don’t know of any sense in which I or anyone else was “captured” by OpenAI.

I can’t comment on #4.

#5 seems off to me. I don’t know whether OpenAI uses nondisparagement agreements; I haven’t signed one. The reason I am careful with public statements about OpenAI is (a) it seems generally unproductive for me to talk carelessly in public about important organizations (likely to cause drama and drain the time and energy of me and others); (b) I am bound by confidentiality requirements, which are not the same as nondisparagement requirements. Information I have access to via having been on the board, or via being married to a former employee, is not mine to freely share.

Wei Dai @ 2024-05-17T22:54 (+2)

#5 seems off to me. I don’t know whether OpenAI uses nondisparagement agreements;

Details about OpenAI's nondisparagement agreements have come out.

Wei Dai @ 2022-11-19T14:48 (+105) in response to My takes on the FTX situation will (mostly) be cold, not hot

Would be interested in your (eventual) take on the following parallels between FTX and OpenAI:

  1. Inspired/funded by EA
  2. Taking big risks with other people's lives/money
  3. Attempt at regulatory capture
  4. Large employee exodus due to safety/ethics/governance concerns
  5. Lack of public details of concerns due in part to non-disparagement agreements
Wei Dai @ 2024-05-17T22:52 (+2)

Unlike FTX, OpenAI has now had a second wave of resignations in protest of insufficient safety focus.

JackM @ 2024-05-17T22:33 (+5) in response to Ilya Sutskever has officially left OpenAI

Hmm, I don't see why ensuring the best people go to Anthropic necessarily means they will take safety less seriously. I can actually imagine the opposite effect as if Anthropic catches up or even overtakes OpenAI then their incentive to cut corners should actually decrease because it's more likely that they can win the race without cutting corners. Right now their only hope to win the race is to cut corners.

Ultimately what matters most is what the leadership's views are. I suspect that Sam Altman never really cared that much about safety, but my sense is that the Amodeis do.

Habryka @ 2024-05-17T22:47 (+6)

Yeah, I don't think this is a crazy take. I disagree with it based on having thought about it for many years, but yeah, I agree that it could make things better (though I don't expect it would and would instead make things worse).

Agustín Covarrubias @ 2024-05-17T21:45 (+2) in response to My Lament to EA

I'll note that I think that choosing to prioritize resources (or support) for top universities doesn't imply that one thinks that EAs need to be any less altruistic or truth-seeking. One can prioritize groups at top universities while (for example) maintaining the same threshold for what a good organizer or group member looks like. These kinds of strategies might result in less diversity overall, but I don't think they result in having less altruistic people.

David M @ 2024-05-17T22:42 (+2)

Indeed not. I think that trying to appeal to those who chase prestige selects against truth-seeking and altruism, and I don’t think merely focusing on top unis has that effect. I’m responding to the part of the post about appealing to prestige chasers.

Habryka @ 2024-05-17T22:16 (+4) in response to Ilya Sutskever has officially left OpenAI

Yes, that if we send people to Anthropic with the aim of "winning an AI arms race" that this will make it more likely that Anthropic will start to cut corners. Indeed, that is very close to the reasoning that caused OpenAI to exist and what seems to have caused it to cut lots of corners.

JackM @ 2024-05-17T22:33 (+5)

Hmm, I don't see why ensuring the best people go to Anthropic necessarily means they will take safety less seriously. I can actually imagine the opposite effect as if Anthropic catches up or even overtakes OpenAI then their incentive to cut corners should actually decrease because it's more likely that they can win the race without cutting corners. Right now their only hope to win the race is to cut corners.

Ultimately what matters most is what the leadership's views are. I suspect that Sam Altman never really cared that much about safety, but my sense is that the Amodeis do.

JackM @ 2024-05-17T21:38 (+2) in response to Ilya Sutskever has officially left OpenAI

What are you suggesting? That if we direct safety conscious people to Anthropic that it will make it more likely that Anthropic will start to cut corners? Not sure what your point is.

Habryka @ 2024-05-17T22:16 (+4)

Yes, that if we send people to Anthropic with the aim of "winning an AI arms race" that this will make it more likely that Anthropic will start to cut corners. Indeed, that is very close to the reasoning that caused OpenAI to exist and what seems to have caused it to cut lots of corners.

Linch @ 2024-05-16T23:59 (+6) in response to Yanni Kyriacos's Quick takes

I would do it but my LTFF funding does not cover this

(Speaking as someone on LTFF, but not on behalf of LTFF) 

How large of a constraint is this for you? I don't have strong opinions on whether this work is better than what you're funded to do, but usually I think it's bad if LTFF funding causes people to do things that they think is less (positively) impactful! 

We probably can't fund people to do things that are lobbying or lobbying-adjacent, but I'm keen to figure out or otherwise brainstorm an arrangement that works for you.

yanni kyriacos @ 2024-05-17T22:07 (+1)

Hey Linch, thanks for reaching out! Maybe send me your email or HMU here yannikyriacos@gmail.com

Stephen Clare @ 2024-05-16T17:08 (+11) in response to Yanni Kyriacos's Quick takes

You should probably also blank their job title (which would make it easy to work out who they are) and their phone number (!)

yanni kyriacos @ 2024-05-17T22:05 (+1)

Why am I so bad at this Stephen. Send help.

David M @ 2024-05-17T09:28 (+2) in response to My Lament to EA

I’ve never affiliated with a university group. I’m sad to hear that at least some university groups seem to be trying to appeal to ambitious prestige-chasers, and I hope it’s not something that the CEA Groups team has applied generally. I wonder if it comes from a short-sighted strategy of trying to catch those who are most likely to end up in powerful positions in the future, which would be in line with the reasons there has been a focus on the most prestigious universities. I call it short-sighted because filling the next generation of your movement with people who are light on values and strong on politics seems like a certain way to kill what’s valuable about EA (such as commitments to altruism and truth-seeking).

Agustín Covarrubias @ 2024-05-17T21:45 (+2)

I'll note that I think that choosing to prioritize resources (or support) for top universities doesn't imply that one thinks that EAs need to be any less altruistic or truth-seeking. One can prioritize groups at top universities while (for example) maintaining the same threshold for what a good organizer or group member looks like. These kinds of strategies might result in less diversity overall, but I don't think they result in having less altruistic people.

Habryka @ 2024-05-17T20:40 (+2) in response to Ilya Sutskever has officially left OpenAI

That sounds like the way OpenAI got started.

JackM @ 2024-05-17T21:38 (+2)

What are you suggesting? That if we direct safety conscious people to Anthropic that it will make it more likely that Anthropic will start to cut corners? Not sure what your point is.

Guy Raveh @ 2024-05-11T14:27 (+7) in response to My Lament to EA

I'm on the one hand happy to hear that the groups team isn't as elite-focused as I had thought; on the other hand, I'm still troubled by the margin-based reasoning.

Agustín Covarrubias @ 2024-05-17T21:31 (+2)

Could you clarify what you mean by margin-based reasoning in this context?