Latest comments on the EA Forum

Comments on 2024-04-14

Jason @ 2024-04-14T13:23 (+4) in response to To what extent & how did EA indirectly contribute to financial crime - and what can be done now? One attempt at a review

Can you link to a discussion of the suit in question? I don't think that would be an accurate characterization of the suit I am aware of (complaint analyzed here). That suit is about roughly about aiding SBF in making specific transfers in breach of his fiduciary duty to Alameda. I wouldn't say it is about "endors[ing]" in the ordinary meaning of the word, or that it relied on allegations that Nick should have known about the criminal frauds going on at FTX.

That being said, I do agree more generally that "people who had a role at FTXFF" tend to be at the top of my mental lists of people who should worry most about the possibility of individual suits and people who have probably been advised by counsel to keep their mouths shut.

David Mathers @ 2024-04-14T13:28 (+2)

OK, I misremembered the exact nature of the suit. Sorry.

David Mathers @ 2024-04-14T11:58 (+4) in response to To what extent & how did EA indirectly contribute to financial crime - and what can be done now? One attempt at a review

I strongly suspect Will is trying to avoid being sued.

Even from a purely selfish point of view, explicitly apologising and saying "sorry I made a mistake in not trusting the people who told me Sam behaved badly at Alameda", since it would actually help restore his reputation a bit.

But Nick Beckstead has already been sued (unsuccessfully) on the grounds that he in some sense endorsed SBF whilst he "should have known" that he was a fraud. And this was on grounds far weaker than "people who worked Sam told him he was unethical and lies to clients." Being sued is not nice, even if you eventually win. Every time FTX has come up on the forum, EAs with legal experience have said "people and orgs probably can't say much without risking legal exposure". Seems plausible to me that is right.

Unfortunately this means we'll likely never have a full accounting of who did what when.

Jason @ 2024-04-14T13:23 (+4)

Can you link to a discussion of the suit in question? I don't think that would be an accurate characterization of the suit I am aware of (complaint analyzed here). That suit is about roughly about aiding SBF in making specific transfers in breach of his fiduciary duty to Alameda. I wouldn't say it is about "endors[ing]" in the ordinary meaning of the word, or that it relied on allegations that Nick should have known about the criminal frauds going on at FTX.

That being said, I do agree more generally that "people who had a role at FTXFF" tend to be at the top of my mental lists of people who should worry most about the possibility of individual suits and people who have probably been advised by counsel to keep their mouths shut.

ethai @ 2024-04-14T01:47 (+3) in response to a guy named josh's Quick takes

(I run hiring rounds with ~100-1000 applicants) agree with Jamie here. However, if someone was close to a cutoff, I do specifically include "encourage you to apply to future roles" in my rejection email. I also always respond when somebody asks for feedback proactively.

Is revealing scores useful to candidates for some other reason not covered by that? It seems to me the primary reason (since it sounds like you aren't asking for qualitative feedback to also be provided) would be to inform candidates as to whether applying for future similar roles is worth the effort.

a guy named josh @ 2024-04-14T12:56 (+1)

I think scores would be good in the potentially time-saving way you outlined. I also think that having a more nuanced sense of how well my applications - or specific parts of it - were perceived/scored would be helpful. 

My experience asking for qualitative feedback has been mixed - sometimes I have gotten just flat out ignored, at other times I have gotten the usual 'no can do due to lack of operational capacity' and some times I have actually gotten valuable personal feedback. 

My idea is that there has to be a way to make some feedback beyond yes/no automatically available to all applicants. Maybe simply being told one is a particularly strong applicant and should reapply or apply to similar roles is good (and kind) enough. 

Jamie_Harris @ 2024-04-13T12:27 (+11) in response to a guy named josh's Quick takes

I tried doing this a while back. Some things I think I worried about at the time:

(1) disheartening people excessively by sending them scores that seem very low/brutal, especially if you use an unusual scoring methodology (2) causing yourself more time costs than it seems like at first, because (a) you find yourself needing to add caveats or manually hide some info to make it less disheartening to people, (b) people ask you follow-up questions (3) exposing yourself to some sort of unknown legal risk by saying something not-legally-defensible about the candidate or your decision-making.

(1) turned out to be pretty justified I think, e.g. at least one person expressing upset/dissatisfaction at being told this info. (2) definitely happened too, although maybe not all that many hours in the grand scheme of things (3) we didn't get sued but who knows how much we increased the risk by

a guy named josh @ 2024-04-14T12:49 (+1)

Okay, I definitely see those concerns! Unknown legal risk - especially as it relates to in many cases hiring in a lot of different countries at the same time with potentially different laws seems like a good reason not to release scores. 

For me personally getting a rejection vs getting a rejection and being told I had the lowest score among all applicants, probably wouldn't make much of a difference - it might even save me time spent on future applications for similar positions. But on that maybe just releasing quarter percentiles would be a better less brutal alternative? 

I think a general, short explainer of the scoring methodology used for a hiring round could/should be released to the applicants, if only for transparency's sake. So, explainer + raw scores and no ranking might also be another alternative? 

Maybe I am misguided in my idea that 'this could be a low-time-cost way of making sure all applicants get a somewhat better sense of how good/bad their applications were.' I have after all only ever been on the applicant side of things and it does seem the current system is working fine at generating good hires. 

David Mathers @ 2024-04-14T11:58 (+4) in response to To what extent & how did EA indirectly contribute to financial crime - and what can be done now? One attempt at a review

I strongly suspect Will is trying to avoid being sued.

Even from a purely selfish point of view, explicitly apologising and saying "sorry I made a mistake in not trusting the people who told me Sam behaved badly at Alameda", since it would actually help restore his reputation a bit.

But Nick Beckstead has already been sued (unsuccessfully) on the grounds that he in some sense endorsed SBF whilst he "should have known" that he was a fraud. And this was on grounds far weaker than "people who worked Sam told him he was unethical and lies to clients." Being sued is not nice, even if you eventually win. Every time FTX has come up on the forum, EAs with legal experience have said "people and orgs probably can't say much without risking legal exposure". Seems plausible to me that is right.

Unfortunately this means we'll likely never have a full accounting of who did what when.

Radical Empath Ismam @ 2024-04-14T12:44 (+15)

If this is the case that MacAskill cannot be forthcoming for valid reasons (opening himself up to legal vulnerability), as a community it would still make sense for us to err on the side of caution and have other leaders for this community as Chris argues for.

Joseph Lemien @ 2024-04-13T19:07 (+2) in response to a guy named josh's Quick takes

Jamie, I've been contemplating writing up a couple of informal "case study"-type reports of different hiring practices. My intention/thought process would be to allow EA orgs to learn about how several different orgs do hiring, to highlight some best practices, and generally to allow/encourage organizations to improve their methods. How would you feel about writing up a summary or having a call with me to allow me to understand how you tried giving feedback and what specific aspects caused challenges?

a guy named josh @ 2024-04-14T12:34 (+1)

This is something I would be interested in seeing! A lot of EA orgs already have public info on their hiring process (at least in a structural sense). I'd be more curious about what happens under the hood, 'scoring methodologies' in particular. 

EJT @ 2024-04-06T11:44 (+1) in response to My favourite arguments against person-affecting views

Interesting, thanks! I hadn't come across this argument before.

Matthew Rendall @ 2024-04-14T12:17 (+1)

It's in his book Inequality, chapter 9. Ingmar Persson makes a similar argument about the priority view here: https://link.springer.com/article/10.1023/A:1011486120534.

Chriswaterguy @ 2024-04-14T11:11 (+18) in response to To what extent & how did EA indirectly contribute to financial crime - and what can be done now? One attempt at a review

Will MacAskill appears to be ignoring these questions. E.g. he was interviewed about FTX recently by Sam Harris¹ and made zero mention of any whistleblowing in his account. He also gave the impression that he barely knew SBF, describing only a few fairly shallow interactions (not at all the impression I'd received while SBF was still in favour).

The interview portion of the episode was 80 min, so it wasn't for lack of time.

I've been waiting for a response from Will – a full explanation and (if things are as they seem) a sincere mea culpa. I would expect no less of myself; and I expect more from someone who has held such responsibility.

Based on public information, it seems to me that Will exercised very poor judgement and a lack of moral leadership. And he now appears to be avoiding responsibility with a distorted retelling of events.

I hope and expect that his role in EA in future is restricted to that of a philosopher, and not in any sense that of a leader.

(These arguments might also apply in varying degrees to other leaders who were involved with SBF, or who ignored/hushed whistleblowers – however I'm less familiar with their roles.)

If this continues, and if Will's lack of spine is matched by that of the rest of the EA leadership, I'll sadly continue to drift away from EA.

¹Making Sense podcast, "#361 Sam Bankman-Fried & Effective Altruism", 2 April 2024.

David Mathers @ 2024-04-14T11:58 (+4)

I strongly suspect Will is trying to avoid being sued.

Even from a purely selfish point of view, explicitly apologising and saying "sorry I made a mistake in not trusting the people who told me Sam behaved badly at Alameda", since it would actually help restore his reputation a bit.

But Nick Beckstead has already been sued (unsuccessfully) on the grounds that he in some sense endorsed SBF whilst he "should have known" that he was a fraud. And this was on grounds far weaker than "people who worked Sam told him he was unethical and lies to clients." Being sued is not nice, even if you eventually win. Every time FTX has come up on the forum, EAs with legal experience have said "people and orgs probably can't say much without risking legal exposure". Seems plausible to me that is right.

Unfortunately this means we'll likely never have a full accounting of who did what when.

Yarrow Bouchard @ 2024-04-13T06:42 (+33) in response to To what extent & how did EA indirectly contribute to financial crime - and what can be done now? One attempt at a review

I do not follow everything that happens in the EA world. I don't use Twitter. I'm out of the loop. So, I don't know if any of the people named have responded to the claims made in this Time article from March 2023:

...[Will] MacAskill had long been aware of concerns around Bankman-Fried. He was personally cautioned about Bankman-Fried by at least three different people in a series of conversations in 2018 and 2019, according to interviews with four people familiar with those discussions and emails reviewed by TIME.

He wasn’t alone. Multiple EA leaders knew about the red flags surrounding Bankman-Fried by 2019, according to a TIME investigation based on contemporaneous documents and interviews with seven people familiar with the matter. Among the EA brain trust personally notified about Bankman-Fried’s questionable behavior and business ethics were Nick Beckstead, a moral philosopher who went on to lead Bankman-Fried’s philanthropic arm, the FTX Future Fund, and Holden Karnofsky, co-CEO of OpenPhilanthropy, a nonprofit organization that makes grants supporting EA causes. Some of the warnings were serious: sources say that MacAskill and Beckstead were repeatedly told that Bankman-Fried was untrustworthy, had inappropriate sexual relationships with subordinates, refused to implement standard business practices, and had been caught lying during his first months running Alameda, a crypto firm that was seeded by EA investors, staffed by EAs, and dedicating to making money that could be donated to EA causes.

 

Many of the emerging issues at Alameda that were reported to EA leaders beginning in 2018—including pervasive dishonesty, sloppy accounting, and rejection of corporate controls—presaged the scandal that unfolded at FTX four years later, according to sources who were granted anonymity to avoid professional retribution or becoming entangled in Bankman-Fried’s ongoing legal drama. “I was shocked at how much of what came out about FTX rhymed with the concerns we raised in the early days,” says one person who spoke directly with MacAskill and others about Bankman-Fried in 2018. “It was the same thing. All of the same problems.”

 

It’s not entirely clear how EA leaders reacted to the warnings. Sources familiar with the discussions told TIME that the concerns were downplayed, rationalized as typical startup squabbles, or dismissed as “he said-she said,” as two people put it. EA leaders declined or did not respond to multiple requests from TIME to explain their reaction to these warnings and what they did in response. But by the end of 2018, Bankman-Fried’s behavior was such an open secret that EA leaders were debating Bankman-Fried’s presence on the board of the Centre for Effective Altruism. In emails among senior EA leaders, which TIME reviewed, one person wrote that they had raised worries about Bankman-Fried’s trustworthiness directly with MacAskill, and that MacAskill had dismissed the concerns as “rumor.” In 2019, Bankman-Fried left CEA’s board.

MacAskill declined to answer a list of detailed questions from TIME for this story. “An independent investigation has been commissioned to look into these issues; I don’t want to front-run or undermine that process by discussing my own recollections publicly,” he wrote in an email. “I look forward to the results of the investigation and hope to be able to respond more fully after then.” Citing the same investigation, Beckstead also declined to answer detailed questions. Karnofsky did not respond to a list of questions from TIME. Through a lawyer, Bankman-Fried also declined to respond to a list of detailed written questions. The Centre for Effective Altruism (CEA) did not reply to multiple requests to explain why Bankman-Fried left the board in 2019. A spokesperson for Effective Ventures, the parent organization of CEA, cited the independent investigation, launched in Dec. 2022, and declined to comment while it was ongoing.

 

In the weeks leading up to that April 2018 confrontation with Bankman-Fried and in the months that followed, [Tara] Mac Aulay and others warned MacAskill, Beckstead and Karnofsky about her co-founder’s alleged duplicity and unscrupulous business ethics, according to four people with knowledge of those discussions. Mac Aulay specifically flagged her concerns about Bankman-Fried’s honesty and trustworthiness, his maneuvering to control 100% of the company despite promising otherwise, his pattern of unethical behavior, and his inappropriate relationships with subordinates, sources say.

[Naia] Bouscal recalled speaking to Mac Aulay immediately after one of Mac Aulay’s conversations with MacAskill in late 2018. “Will basically took Sam’s side,” said Bouscal, who recalls waiting with Mac Aulay in the Stockholm airport while she was on the phone. (Bouscal and Mac Aulay had once dated; though no longer romantically involved, they remain close friends.) “Will basically threatened Tara,” Bouscal recalls. “I remember my impression being that Will was taking a pretty hostile stance here and that he was just believing Sam’s side of the story, which made no sense to me.”

“He was treating it like a ‘he said-she said,’ even though every other long-time EA involved had left because of the same concerns,” Bouscal adds.

To me, this feels like the most important part of the whole story. 

Chriswaterguy @ 2024-04-14T11:11 (+18)

Will MacAskill appears to be ignoring these questions. E.g. he was interviewed about FTX recently by Sam Harris¹ and made zero mention of any whistleblowing in his account. He also gave the impression that he barely knew SBF, describing only a few fairly shallow interactions (not at all the impression I'd received while SBF was still in favour).

The interview portion of the episode was 80 min, so it wasn't for lack of time.

I've been waiting for a response from Will – a full explanation and (if things are as they seem) a sincere mea culpa. I would expect no less of myself; and I expect more from someone who has held such responsibility.

Based on public information, it seems to me that Will exercised very poor judgement and a lack of moral leadership. And he now appears to be avoiding responsibility with a distorted retelling of events.

I hope and expect that his role in EA in future is restricted to that of a philosopher, and not in any sense that of a leader.

(These arguments might also apply in varying degrees to other leaders who were involved with SBF, or who ignored/hushed whistleblowers – however I'm less familiar with their roles.)

If this continues, and if Will's lack of spine is matched by that of the rest of the EA leadership, I'll sadly continue to drift away from EA.

¹Making Sense podcast, "#361 Sam Bankman-Fried & Effective Altruism", 2 April 2024.

Linch @ 2024-04-13T22:24 (+4) in response to Should the main text of the write-ups of Open Philanthropy’s large grants be longer than 1 paragraph?

Grantees are obviously welcome to do this. That said, my guess is that this will make the forum less enjoyable/useful for the average reader, rather than more. 

Vasco Grilo @ 2024-04-14T09:12 (+2)

Grantees are obviously welcome to do this.

Right, but they have not been doing it. So I assume EA Funds would have to at least encourage applicants to do it, or even make it a requirement for most applications. There can be confidential information in some applications, but, as you said below, applicants do not have to share everything in their public version.

That said, my guess is that this will make the forum less enjoyable/useful for the average reader, rather than more.

I guess the opposite, but I do not know. I am mostly in favour of experimenting with a few applications, and then deciding whether to stop or scale up.

NickLaing @ 2024-04-14T08:04 (+4) in response to Dear EA, please be the reason people like me will actually see a better world. Help me make some small stride on extreme poverty where I live -- by the end of 2024.

After reading a bit more, one potential issue here is that most of the white sorghum and cassava processed by this project (Anthony can correct me if I'm wrong) will be used for making alcohol, which could cause negative externalities through increasing alcohol production or reducing price, although this is hard to measure.

There could also be more local brewing as well using these crops.

Anthony what do you think about this potential negative to selling sorghum to the alcohol companies?

Stuart Buck @ 2024-04-12T18:43 (+5) in response to Sam Harris and William MacAskill on SBF & EA

Is the consensus currently that the investment in Twitter has paid off or is ever likely to do so? 

AnonymousTurtle @ 2024-04-14T07:26 (+7)

No, but in expectation it wasn't very far from the stock market valuation. I think it's very possible that it was positive EV even if it didn't work out

EA Opportunity Board @ 2024-04-13T05:58 (+7) in response to Applications Open: Elevate Your Mental Resilience with Rethink Wellbeing's CBT Program

This seems like a great opportunity. It is now live on the EA Opportunity Board!

Inga @ 2024-04-14T07:23 (+2)

Thank you for spreading the word! We hope to help lots of ambitious altruists this year.

Peter Wildeford @ 2024-04-12T13:14 (+23) in response to Mediocre EAs: career paths and how do they engage with EA?

I still don't see why people aren't still excited about "earning to give" and donating.

Anyone who can donate $5000 via GiveWell can save a life.

Possibly you can do even better than that per dollar if you're okay accepting some premises around nonhuman animal welfare / sentience or risks from emerging technologies.

I think this is all very cool.

Moreover, while a $5000 donation is a big commitment, it is also achievable by a rather wide array of people. Many people are wealthy enough to do this donation, but do not do it, or donate far less effectively. You could have the same philanthropic power as a multi-millionaire philanthropist by allocating better.

If you earn a median income of ~$40K USD/yr[1] and spend $32,400[2], that gives you $7600 left over to donate each year, which could potentially save three lives every two years.


  1. As a single person. ↩︎

  2. Spend $6K/yr on taxes. Then spend $1K/mo on rent and $100/mo on utilities which is doable in most metropolitan areas in a small apartment or if you have roommates. Then maybe spend $300/mo on groceries and $300/mo on other things. Then save 15% of your income ($6K/yr), which is pretty standard financial advice. ↩︎

Dylan Richardson @ 2024-04-14T06:34 (+1)

This seems like a common group misperception to me, that (other) EAs have turned against earning to give. Take this comment for instance - zero disagrees. 

But maybe there's a vague unease as opposed to explicit beliefs? Like student clubs just not broaching the subject as much as they had before? Self-censoring? If so, it's not obviously represented in any forum activity I've seen, neither is it obvious on the EA survey, which finds "further de-emphasize ETG" in only 5% of responses. Maybe that's enough to be worried anyways?

Yarrow Bouchard @ 2024-04-14T04:17 (+1) in response to Yarrow Bouchard's Quick takes

Have Will MacAskill, Nick Beckstead, or Holden Karnofsky responded to the reporting by Time that they were warned about Sam Bankman-Fried's behaviour years before the FTX collapse?

Habryka @ 2024-04-14T02:24 (+2) in response to Understanding FTX's crimes

This link's hypothesis is about people just trying to fit in―but SBF seemed not to try to fit in to his peer group! He engaged in a series of reckless and fraudulent behaviors that none of his peers seemed to want.

(Author of the post) My model is that Sam had some initial tendencies for reckless behavior and bullet-biting, and those were then greatly exacerbated via evaporative cooling dynamics at FTX. 

It sounds like SBF drove away everyone who couldn't stand his methods until only people who tolerated him were left. That's a pretty different way of making an organization go insane.

Relatedly, this kind of evaporative cooling is exactly the dynamic I was trying to point to in my post. Quotes: 

People who don’t want to live up to the demanding standard leave, which causes evaporative cooling and this raises the standards for the people who remain. Frequently this also causes the group to lose critical mass. 

[...]

My current best model of what happened at an individual psychological level was many people being attracted to FTX/Alameda because of the potential resources, then many rounds of evaporative cooling as anyone who was not extremely hardcore according to the group standard was kicked out, with there being a constant sense of insecurity for everyone involved that came from the frequent purges of people who seemed to not be on board with the group standard.

DPiepgrass @ 2024-04-14T03:20 (+6)

Sorry if I sounded redundant. I'd always thought of "evaporative cooling of group beliefs" like "we start with a group with similar values/goals/beliefs; the least extreme members gradually get disengaged and leave; which cascades into a more extreme average that leads to others leaving"―very analogous to evaporation. I might've misunderstood, but SBF seemed to break the analogy by consistently being the most extreme, and actively and personally pushing others away (if, at times, accidentally). Edit: So... arguably one can still apply the evaporative cooling concept to FTX, but I don't see it as an explanation of SBF himself.

Joseph Lemien @ 2024-04-13T19:13 (+5) in response to ABishop's Quick takes

Haha. Well, I guess I would first ask effective at what? Effective at giving people additional years of healthy & fulfilling life? Effective at creating new friendships? Effective at making people smile?

I haven't studied it at all, but my hypothesis that it is the kind of intervention that is  similar to "awareness building," but it doesn't have any call to action (such as a donation). So it is probably effective in giving people a nice experience for a few seconds, and maybe improving their mood for a period of time, but it probably doesn't have longer-lasting effects. From a cursory glance at Google Scholar, it looks like there hasn't been much research on free hugs.

ABishop @ 2024-04-14T03:20 (+1)

Hmm, I'm a little confused. If I cook a meal for someone, it doesn't seem to mean much. But if no one is cooking for someone, it is a serious problem and we need to help. Of course, I'm not sure if we're suffering from that kind of "skinship hunger."

Chris Leong @ 2024-04-14T03:13 (+11) in response to Things EA Group Organisers Need To Hear

Thank you for your service.

David T @ 2024-04-13T22:59 (+3) in response to Should the main text of the write-ups of Open Philanthropy’s large grants be longer than 1 paragraph?

I think a dedicated area would minimise the negative impact on people that aren't interested whilst potentially adding value (to prospective applicants in understanding what did and didn't get accepted, and possibly also to grant assessors if there was occasional additional insight offered by commenters)

I 'd expect there would be some details of some applications that wouldn't be appropriate to share on a public forum though

Linch @ 2024-04-14T02:57 (+4)

I 'd expect there would be some details of some applications that wouldn't be appropriate to share on a public forum though

Hopefully grantees can opt-in/out as appropriate! They don't need so share everything. 

DPiepgrass @ 2024-04-13T23:20 (+9) in response to Understanding FTX's crimes

this almost confirms for me that FTX belongs on the list of ways EA and rationalist organizations can basically go insane in harmful ways,

I was confused by this until I read more carefully. This link's hypothesis is about people just trying to fit in―but SBF seemed not to try to fit in to his peer group! He engaged in a series of reckless and fraudulent behaviors that none of his peers seemed to want. From Going Infinite:

He had not been able to let Modelbot rip the way he’d liked—because just about every other human being inside Alameda Research was doing whatever they could to stop him. “It was entirely within the realm of possibility that we could lose all our money in an hour,” said one. One hundred seventy million dollars that might otherwise go to effective altruism could simply go poof. [...]

Tara argued heatedly with Sam until he caved and agreed to what she thought was a reasonable compromise: he could turn on Modelbot so long as he and at least one other person were present to watch it, but should turn it off if it started losing money. “I said, ‘Okay, I’m going home to go to sleep,’ and as soon as I left, Sam turned it on and fell asleep,” recalled Tara. From that moment the entire management team gave up on ever trusting Sam.

Example from Matt Levine:

There is an anecdote (which has been reported before) from the early days of Alameda Research, the crypto trading firm that Bankman-Fried started before his crypto exchange FTX, the firm whose trades with FTX customer money ultimately brought down the whole thing. At some point Alameda lost track of $4 million of investor money, and the rest of the management team was like “huh we should tell our investors that we lost their money,” and Bankman-Fried was like “nah it’s fine, we’ll probably find it again, let’s just tell them it’s still here.” The rest of the management team was horrified and quit in a huff, loudly telling the investors that Bankman-Fried was dishonest and reckless.

It sounds like SBF drove away everyone who couldn't stand his methods until only people who tolerated him were left. That's a pretty different way of making an organization go insane.

It doesn't seem like this shouldn't be an EA failure mode when the EA community is working well. Word should have gotten around about SBF's shadiness and recklessness, leading to some kind of investigation before FTX reached the point of collapse. The first person I heard making the case against SBF post-collapse was an EA (Rob Wiblin?), but we were way too slow. Of course it has been pointed out that many people who worked with / invested in FTX were fooled as well, so what I wonder about is: why weren't there any EA whistleblowers on the inside? Edit: was it that only four people plus SBF knew about FTX's worst behaviors, and the chance of any given person whistle-blowing in a situation like that is under 25%ish? But certainly more people than that knew he was shady. Edit 2: I just saw important details on who knew what. P.S. I will never get used to the EA/Rat tendency to downvote earnest comments, without leaving comments of their own...

Habryka @ 2024-04-14T02:24 (+2)

This link's hypothesis is about people just trying to fit in―but SBF seemed not to try to fit in to his peer group! He engaged in a series of reckless and fraudulent behaviors that none of his peers seemed to want.

(Author of the post) My model is that Sam had some initial tendencies for reckless behavior and bullet-biting, and those were then greatly exacerbated via evaporative cooling dynamics at FTX. 

It sounds like SBF drove away everyone who couldn't stand his methods until only people who tolerated him were left. That's a pretty different way of making an organization go insane.

Relatedly, this kind of evaporative cooling is exactly the dynamic I was trying to point to in my post. Quotes: 

People who don’t want to live up to the demanding standard leave, which causes evaporative cooling and this raises the standards for the people who remain. Frequently this also causes the group to lose critical mass. 

[...]

My current best model of what happened at an individual psychological level was many people being attracted to FTX/Alameda because of the potential resources, then many rounds of evaporative cooling as anyone who was not extremely hardcore according to the group standard was kicked out, with there being a constant sense of insecurity for everyone involved that came from the frequent purges of people who seemed to not be on board with the group standard.

a guy named josh @ 2024-04-12T11:13 (+1) in response to a guy named josh's Quick takes

Would love for orgs running large-scale hiring rounds (say 100+ applicants) to provide more feedback to their (rejected) applicants. Given that in most cases applicants are already being scored and ranked on their responses, maybe just tell them their scores, their overall ranking and what the next round cutoff would have been - say: prompt 1 = 15/20, prompt 2 = 17.5/20, rank = 156/900, cutoff for work test at 100.

Since this is already happening in the background (if my impression here is wrong please lmk), why not make the process more transparent and release scores - with what seems to be very little extra work required (beyond some initial automation). 

ethai @ 2024-04-14T01:47 (+3)

(I run hiring rounds with ~100-1000 applicants) agree with Jamie here. However, if someone was close to a cutoff, I do specifically include "encourage you to apply to future roles" in my rejection email. I also always respond when somebody asks for feedback proactively.

Is revealing scores useful to candidates for some other reason not covered by that? It seems to me the primary reason (since it sounds like you aren't asking for qualitative feedback to also be provided) would be to inform candidates as to whether applying for future similar roles is worth the effort.

MathiasKB @ 2024-03-31T12:04 (+136) in response to Should the main text of the write-ups of Open Philanthropy’s large grants be longer than 1 paragraph?

I think it's a travesty that so many valuable analyses are never publicly shared, but due to unreasonable external expectations it's currently hard for any single organization to become more transparent without occurring enormous costs.

If open phil actually were to start publishing their internal analyses behind each grant, I will bet you at good odds the the following scenario is going to play out on the EA Forum:

  1. Somebody digs deep into a specific analysis carried out. It turns out Open Phil’s analysis has several factual errors that any domain expert could have alerted them to, additionally they entirely failed to consider some important aspect which may change the conclusion.
  2. Somebody in the comments accuses Open Phil of shoddy and irresponsible work. That they are making such large donations decisions based on work filled with errors, proves their irresponsibility. Moreover, why have they still not responded to the criticism?
  3. A new meta-post argues that the EA movement needs reform, and uses the above as one of several examples showing the incompetence of ‘EA leadership’.

Several things would be true about the above hypothetical example:

  1. Open Phil’s analysis did, in fact, have errors.
  2. It would have been better for Open Phil’s work not to have those errors.
  3. The errors were only found because they chose to make the analysis public.
  4. The costs for Open Phil to reduce the error rate of analyses, would not be worth the benefits.
  5. These mistakes were found, and at no cost (outside of reputation) to the organization.

Criticism shouldn’t have to warrant a response if it takes time away from work which is more important. The internal analyses from open phil I’ve been privileged to see were pretty good. They were also made by humans, who make errors all the time.

In my ideal world, every one of these analyses would be open to the public. Like open-source programming people would be able to contribute to every analysis, fixing bugs, adding new insights, and updating old analyses as new evidence comes out.

But like an open-source programming project, there has to be an understanding that no repository is ever going to be bug-free or have every feature.

If open phil shared all their analyses and nobody was able to discover important omissions or errors, my main conclusion would be they are spending far too much time on each analysis.

Some EA organizations are held to impossibly high standards. Whenever somebody points this out, a common response is: “But the EA community should be held to a higher standard!”. I’m not so sure! The bar is where it’s at because it takes significant effort to higher it. EA organizations are subject to the same constraints the rest of the world is subject to.

More openness requires a lowering of expectations. We should strive for a culture that is high in criticism, but low in judgement.

DPiepgrass @ 2024-04-14T01:27 (+4)

What if, instead of releasing very long reports about decisions that were already made, there were a steady stream of small analyses on specific proposals, or even parts of proposals, to enlist others to aid error detection before each decision?

DPiepgrass @ 2024-04-13T23:20 (+9) in response to Understanding FTX's crimes

this almost confirms for me that FTX belongs on the list of ways EA and rationalist organizations can basically go insane in harmful ways,

I was confused by this until I read more carefully. This link's hypothesis is about people just trying to fit in―but SBF seemed not to try to fit in to his peer group! He engaged in a series of reckless and fraudulent behaviors that none of his peers seemed to want. From Going Infinite:

He had not been able to let Modelbot rip the way he’d liked—because just about every other human being inside Alameda Research was doing whatever they could to stop him. “It was entirely within the realm of possibility that we could lose all our money in an hour,” said one. One hundred seventy million dollars that might otherwise go to effective altruism could simply go poof. [...]

Tara argued heatedly with Sam until he caved and agreed to what she thought was a reasonable compromise: he could turn on Modelbot so long as he and at least one other person were present to watch it, but should turn it off if it started losing money. “I said, ‘Okay, I’m going home to go to sleep,’ and as soon as I left, Sam turned it on and fell asleep,” recalled Tara. From that moment the entire management team gave up on ever trusting Sam.

Example from Matt Levine:

There is an anecdote (which has been reported before) from the early days of Alameda Research, the crypto trading firm that Bankman-Fried started before his crypto exchange FTX, the firm whose trades with FTX customer money ultimately brought down the whole thing. At some point Alameda lost track of $4 million of investor money, and the rest of the management team was like “huh we should tell our investors that we lost their money,” and Bankman-Fried was like “nah it’s fine, we’ll probably find it again, let’s just tell them it’s still here.” The rest of the management team was horrified and quit in a huff, loudly telling the investors that Bankman-Fried was dishonest and reckless.

It sounds like SBF drove away everyone who couldn't stand his methods until only people who tolerated him were left. That's a pretty different way of making an organization go insane.

It doesn't seem like this shouldn't be an EA failure mode when the EA community is working well. Word should have gotten around about SBF's shadiness and recklessness, leading to some kind of investigation before FTX reached the point of collapse. The first person I heard making the case against SBF post-collapse was an EA (Rob Wiblin?), but we were way too slow. Of course it has been pointed out that many people who worked with / invested in FTX were fooled as well, so what I wonder about is: why weren't there any EA whistleblowers on the inside? Edit: was it that only four people plus SBF knew about FTX's worst behaviors, and the chance of any given person whistle-blowing in a situation like that is under 25%ish? But certainly more people than that knew he was shady. Edit 2: I just saw important details on who knew what. P.S. I will never get used to the EA/Rat tendency to downvote earnest comments, without leaving comments of their own...

DPiepgrass @ 2024-04-14T01:16 (+7)

You know what, I was reading Zvi's musings on Going Infinite...

Q: But it’s still illegal to mislead a bank about the purpose of a bank account.

Michael Lewis: But nobody would have cared about it.

He seems to not understand that this does not make it not a federal crime? That ‘we probably would not have otherwise gotten caught on this one’ is not a valid answer?

Similarly, Lewis clearly thinks ‘the money was still there and eventually people got paid back’ should be some sort of defense for fraud. It isn’t, and it shouldn’t be.

...

Nor was Sam a liar, in Lewis’s eyes. Michael Lewis continued to claim, on the Judging Sam podcast, that he could trust Sam completely. That Sam would never lie to him. True, Lewis said, Sam would not volunteer information and he would use exact words. But Sam’s exact words to Lewis, unlike the words he saw Sam constantly spewing to everyone else, could be trusted.

It’s so weird. How can the same person write a book, and yet not have read it?

And it occurred to me that all SBF had to do was find a few people who thought like Michael Lewis, and people like that don't seem rare. I mean, don't like 30% of Americans think that the election was stolen from Trump, or that the cases against Trump are a witch hunt, because Trump says so and my friends all agree he's a good guy (and they seek out pep talks to support such thoughts)? Generally the EA community isn't tricked this easily, but SBF was smarter than Trump and he only needed to find a handful of people willing to look the other way while trusting in his Brilliance and Goodness. And since he was smart (and overconfident) and did want to do good things, he needed no grand scheme to deceive people about that. He just needed people like Lewis who lacked a gag reflex at all the bad things he was doing.

Before FTX I would've simply assumed other EAs had a "moral gag reflex" already. Afterward, I think we need more preaching about that (and more "punchy" ways to hammer home the importance of things like virtues, rules, reputation and conscientiousness, even or especially in utilitarianism/consequentialism). Such preaching might not have affected SBF himself (since he cut so many corners in his thinking and listening), but someone in his orbit might have needed to hear it.



Comments on 2024-04-13

Clare_Diane @ 2024-04-11T16:25 (+20) in response to Reasons for optimism about measuring malevolence to tackle x- and s-risks

Thank you for sharing these insights. I am also pessimistic about using self-report methods to measure malevolent traits in the context of screening, and it’s very helpful to hear your insights on that front. However, I think that the vast majority of the value that would come from working on this problem would come from other approaches to it.

 

Instead of trying to use gameable measures as screening methods, I think that: 

(1) Manipulation-proof/non-gameable measures of specific malevolent traits are worth investigating further.[1] There are reasons to investigate both the technical feasibility and the perceived acceptability and political feasibility of these measures.[2]

(2) Gameable measures could be useful in low-stakes, anonymous research settings, despite being (worse than) useless as screening tools.

I explain those points later in this comment, under headings (1) and (2).


 

On the neglectedness point

I think the argument that research in this area is not neglected is very important to consider, but I think that it applies much more strongly in relation to the use of gameable self-report measures for screening than it does to other approaches such as the use of manipulation-proof measures. As I see it, when it comes to using non-gameable measures of malevolent traits, this topic is still neglected relative to its potential importance.  

 

The specific ways in which different levels of malevolence potentially interact with x-risks and s-risks (and risk factors) also seem to be relatively neglected by mainstream academia.

 

Manipulation-proof measures of malevolence also appear to be neglected in practice

  • Despite a number of high-stakes jobs requiring security clearances, ~none that I’m aware of use manipulation-proof measures of malevolence,[3] with the possible exception of well-designed manipulation-proof behavioral tests[4] and careful background checks.[5] 
  • There are also many jobs that (arguably) should, but currently do not, require at least the level of rigor and screening that goes into security clearances. For those roles, the overall goal of reducing the influence of malevolent actors seems to be relatively absent (or at least not actively pursued). Examples of such roles include leadership positions at key organizations working to reduce x-risks and s-risks, as well as leadership positions at organizations working to develop transformative artificial intelligence.
  • Many individuals and institutions don’t seem to have a good understanding of how to identify elevated malevolent traits in others and seem to fail to recognize the impacts of malevolent traits when they’re present.[6] It’s plausible that there’s a bidirectional relationship whereby low levels of recognition of situations where malevolence is contributing to organizational problems would plausibly reduce the degree to which people recognize the value of measuring malevolence in the first place (particularly in manipulation-proof ways), and vice versa.[7]


 

(1) Some reasons to investigate manipulation-proof/non-gameable measures of malevolent traits
 

I’ll assume anyone reading this is already familiar with the arguments made in this post. One approach to reducing risks from malevolent actors could be to develop objective, non-gameable/manipulation-proof measures of specific malevolent traits.

 

Below are some reasons why it might be valuable to investigate the possibility of developing such measures.

 

(A) Information value. 

 

Unlike self-report measures, objective measures of malevolent traits are still relatively neglected (further info on this below). It seems valuable to at least invest in reducing the (currently high) levels of uncertainty here as to the probability that such measures would be (1) technically and (2) socially and politically feasible to use for the purposes discussed in the OP.

 

(B) Work in this area might actually contribute to the development of non-gameable measures of malevolent traits that could then be used for the purposes most relevant to this discussion.

 

I think it would be helpful to develop a set of several manipulation-proof measures to use in combination with each other. To increase the informativeness[8] of the set of measures in combination, it would be helpful if we could find multiple measures whose errors were uncorrelated with each other (though I do not know whether this would be possible in practice, I think it’s worth aiming for). To give an example outside of measuring malevolence, the combination of electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) appears much more useful for diagnosing epilepsy than either modality alone.

 

In some non-research/real-life contexts, such as the police force, there are already indirect measures in place that are specifically designed to be manipulation-proof or non-gameable. These can include behavioral tests designed in such a way that those undertaking the test do not know (or at least cannot verify) whether they are undertaking a test and/or cannot easily deduce what the “correct” course of action is. They are designed to identify specific behaviors that are associated with underlying undesirable traits (such as behaviors that demonstrate a lack of integrity or that demonstrate dishonesty or selfishness). Conducting (covert) integrity testing on police officers is one example of this kind of test (see also: Klaas, 2021, ch. 12).

 

Behavioral tests such as these would likely require significant human resources and other costs in order to be truly unpredictable and manipulation-proof. Despite the costs, in the context of high-stakes decisions (e.g., regarding who should have or keep a key position in an organization or group that influences x-risks or s-risks), it seems worth considering the idea of using behavioral measures in combination with background checks (similar to those done as part of security clearances, as discussed earlier) plus a set of more objective measures (ideas for which are listed below).

 

Below is a tentative, non-exhaustive list of objective methods which seem worth investigating as future ways of measuring malevolent traits. Please note the following things about it, though: 

  • Listing a study or meta-analysis below doesn’t imply that I think it’s well done; this table was put together quickly. 
  • Many of the effect sizes in these studies are quite small, so using any one of these methods in isolation does not seem like it would be useful (based on the information I’ve seen so far).
     
Objective approaches to measuring malevolence that seem worth investigating

Approach


 

Brief commentsExamples of research applications of this approach
Electroencephalography (EEG) event-related potentials (ERPs)

This involves recording electrical activity (measured at the scalp) of the brain via an electrogram. Thanks to its high temporal resolution, EEG allows one to assess unconscious neural responses within milliseconds of a given stimulus being presented. 


 

Relatively portable and cheap.


 

Assessing deficits in neural responses to seeing fearful faces among people with high levels of the “meanness” factor of psychopathy (across multiple studies) (link to full text thesis version of the same study)
Functional magnetic resonance imaging (fMRI)

This involves inferring neural activity based on changes in blood flow to different parts of the brain (which is called blood-oxygen-level dependent [BOLD] imaging). In comparison to EEG, fMRI offers high spatial resolution but relatively poor temporal resolution.


 

Not portable or convenient. (More expensive than all the other methods in this table; fMRI is at least 10 times the cost of fNIRS!) However, a lack of (current) scalability does not (in my opinion) completely rule out the use of such a test in high-stakes hiring decisions.[9]

Assessing mirror neuron activity during emotional face processing across multiple fMRI studies


 

Functional near-infrared spectroscopy (fNIRS)

Measures changes in the concentrations of both oxygenated and deoxygenated hemoglobin near the surface of the cortex.


 

Relatively portable, but not as cheap as some of the other methods.

Assessing medial prefrontal cortex oxygenated hemoglobin activity in response to positive vs negative emotional films in individuals with high versus low levels of callous-unemotional (CU) traits


 

Measuring activity in the dorsolateral prefrontal cortex among subjects with different levels of impulsivity and forgiveness in an experimental game setting

Pupillometry, especially in the context of measuring pupil reactivity to positive, negative, and neutral images

This involves measuring variations in pupil size; such measurements are reflective of changes in sympathetic nervous system tone, but they have an added advantage of higher temporal resolution compared to other measures of sympathetic nervous system activation.


 

Relatively portable and cheap.

Screening for children at risk of future psychopathology Assessing the reduced pupil dilation in response to negative images among 4-7-year-old children and using that to predict later conduct problems and reduced prosocial behavior


 

Assessing pupil reactivity to emotional faces among male prisoners with histories of serious sexual or violent offenses 

Startle reactivity, especially the assessment of aversive startle potentiation (ASP) 

The startle response in humans can be measured by monitoring the movement of the muscle orbicularis oculi

surrounding the eyes. Aversive startle potentiation (ASP) involves presenting a stimulus designed to exaggerate the startle response, such as negative images or negative film clips.


 

Relatively portable and cheap.


 

Assessing startle reactivity in response to negative, positive, and neutral images among people with high levels of psychopathy and everyday sadism

 

 

Other approaches (not listed above)

 

There are also ~manipulation-proof measures that, despite their relatively low susceptibility to manipulation, seem to me to be non-starters due to being likely to have a low positive predictive value, including polygenic risk scores in adults and the use of structural neuroimaging in isolation (i.e., without any functional components), as I again predict that that would have a low positive predictive value. (That’s one of the reasons I did not list magnetoencephalograpy (MEG) above - relatively few functional [fMEG] studies appear to have been done on malevolent traits.)

 

Finally, there are multiple measures that I do *not* currently[10] think are promising due to being too susceptible to direct or indirect manipulation, including self-report surveys, implicit association tests (unless they are used in combination with pupillometry, in which case it’s possible that they would become less gameable), eye tracking, assessments of interpersonal distance and body language analysis, behavior in experimental game situations or in other contexts where it’s clear what the socially desirable course of action would be, measures of sympathetic nervous system activation that lack temporal resolution,[11] anonymous informant interviews (as mentioned earlier[5]), reference reports, and 360 reviews. In the case of the latter two approaches, I predict that individuals with high enough levels of social skills, intelligence, and charisma would be able to garner support through directly or indirectly manipulating (and/or threatening) the people responsible for rating them.

 

Thinking about implementation in the real-world: tentative examples of the social and political feasibility of using objective screening measures in high-stakes hiring and promotion decisions

 

Conditioned on accurate objective measures of malevolent traits being available and cost-effective (which would be a big achievement in itself), would such screening methods ever actually be taken up in the real world, and if they were taken up, could this be done ethically? 

 

It seems like surveys could address the question of perceived acceptability of such measures. But until such surveys are available, it seems reasonable to be pessimistic about this point. 

 

Having said this, there are some real-world examples of contexts in which it is already accepted that people should be selected based on specific traits. In positions requiring security clearance or other positions requiring very high degrees of trust in someone prior to hiring them, it is common to select candidates on the basis of their assessed character (despite the fact that there are not yet any objective measures for the character traits they are testing for). For example: 

 

In addition, there are some professional contexts in which EEG is already being used in decisions relating to allowing people to start or keep a job (in much lower-stakes settings than the ones of interest to longtermits). In these contexts, it is to try to rule out the possibility of epilepsy, which people tend to think about in a different way to personality traits. However, the examples still seem worth noting.

  • Before becoming a pilot, one typically has to undergo EEG screening for epileptiform activity, regardless of one’s medical history - and this is reportedly despite there not even being sufficient evidence for the benefits of this.
  • Driving regulations for epilepsy in Europe often stipulate that someone with epilepsy should show no evidence of epileptiform activity on EEG (and often the EEG has to be done while sleep deprived as well). 

 

Of course, taking on a leadership position in an organization that has the capacity to influence x-risks or s-risks would (arguably) be much a higher-stakes decision than taking on a role as a pilot or as a professional driver, for example. The fact that there is a precedent for using a form of neuroimaging as part of an assessment of whether someone should take on a professional role, even in the case of these much lower-stakes roles, is noteworthy. It suggests that it’s not unrealistic to expect people applying to much higher-stakes roles to undergo similarly manipulation-proof tests to assess their safety in that role.


 

(2) Why gameable/manipulable measures might be useful in research contexts (despite not being useful as screening methods)

 

Self-report surveys are easy to collect data on, and it seems that people are less likely to lie in low-stakes, anonymous settings. 

 

Administering self-report surveys could improve our understanding of how various traits of interest correlate with other traits and with specific beliefs (e.g., with fanatical beliefs) and behaviors of interest (including beliefs and behaviors that would be concerning from an x-risk or s-risk perspective, which do not appear to have been studied much to date in the context of malevolence research).[12]


 

Thank you to David Althaus for very helpful discussions and for quickly reading over this! Any mistakes or misunderstandings are mine.

  1. ^

    Just to be clear, I don’t think that these investigations need to be done by people within the effective altruism community. I also agree with William MacAuliffe’s comment above that there would be value in mainstream academics working on this topic, and I hope that more of them do so in the future. However, this topic seems neglected enough to me that there could be value in trying to accelerate progress in this area. And like the OP mentioned, there could be a role for EA orgs in testing the feasibility of using measures of malevolent traits, if or once they are developed.

  2. ^

    Of course, anyone pursuing this line of investigation would need to carefully consider the downsides and ethical implications at each stage. Hopefully this goes without saying.

  3. ^

    I’d be very happy to be proven wrong about this.

  4. ^

    You could say: of course manipulation-proof measures of malevolence are neglected in practice - they don’t exist yet! However, well-designed manipulation-proof behavioral tests do exist in some places, and they do tend to test things that are correlated with malevolence, such as dishonesty. So it seems at least possible to track malevolence in some manipulation-proof ways, but this appears to be done quite rarely. 

  5. ^

    One could argue that security clearance processes also tend to include other, indirect measures of malevolence (or of traits that [anti]correlate with malevolent traits), and some of these indirect measures are less susceptible to manipulation than others. When it comes to background checks and similar checks as part of security clearances, these  seem difficult to game, but not impossible (for example, if someone was a very “skilled” fraud, or if someone hypothetically stole the identity of someone who already had security clearance). Other things to note about background checks are that they can be costly and can still miss “red flags” (for example, this paper claims that this happened in the case of Edward J. Snowden’s security clearance). However, if done well, such investigations could identify instances where the person of interest has been dishonest, has displayed indirect evidence of past conflicts, or has a record of more overtly malevolent behaviors. In addition to background checks, interviews with people who know the person of interest could also be useful. However, I would argue that these are somewhat gameable, because sufficiently malevolent individuals might cause some interviewees/informants to be fearful about giving honest feedback (even if they were giving feedback anonymously, and even if they were actively reassured of their anonymity). Having said this, there could be value in combining information from anonymous informant reports with a set of truly manipulation-proof measures of malevolence (if or when some measures are found to be informative enough to be used in this context). 

  6. ^

    It appears there is also a relative paucity of psychoeducation about malevolent traits. A lack of understanding of malevolent traits has arguably also affected the EA community. For example, many people didn’t recognise how concerning some of SBF’s traits are for someone in a position as powerful as he was in. This lack of awareness, and the resulting lack of tracking of the probability that someone has high levels of malevolence, could plausibly contribute to institutional failures (such as a failure to take whistleblowers seriously and a general failure to identify and respond appropriately when someone shows a pattern of conflict, manipulation, deception, or other behaviors associated with malevolence). In the case of SBF, Spencer Greenberg found that some people close to SBF had not considered his likely deficient affective experience (DAE) as an explanation for his behavior within the set of hypotheses they’d been considering (disclosure: I work for Spencer, but I’m writing this comment in my personal capacity). Spencer spoke with four people who knew SBF well and presented them with the possibility that SBF had DAE while also believing in EA principles. He said that, after being presented with this hypothesis, all of their reactions “fell somewhere on the spectrum from “that seems plausible” to “that seems likely,” though it was hard to tell exactly where in that range they each landed.” Surprisingly, however, before their conversations with Spencer, the four people “seemed not to have considered [that possibility] before.”  

  7. ^

    In the absence of accurate methods of identifying people with high levels of malevolent traits, organizations may fail to identify situations where someone’s malevolent traits are contributing to poor outcomes. In turn, this lack of recognition of the presence and impact of malevolent traits would plausibly reduce the willingness of such organizations to develop or use measures of malevolence in the first place. On the other hand, if organizations either began to improve their ability to detect malevolent traits, or became more aware of the impact of malevolent traits on key outcomes, this could plausibly contribute to a positive feedback loop where improved ability to detect malevolence and an appreciation of the importance of detecting malevolence positively reinforce each other. Such improvements might be useful not only for hiring decisions but also for decisions as to whether to promote someone or keep them in a position of power. For example, it seems reasonable to expect that whistleblowers within organizations (or others who try to speak out against someone with high levels of malevolent traits) would be more likely to be listened to (and responded to appropriately) if there was greater awareness of the behavioral patterns of malevolent actors.

  8. ^

    In this context, I’m talking about the degree to which measures could more sensitively detect (but not overestimate) when an individual’s levels of specific malevolent traits (such as callousness or sadism) would be high enough that we’d expect them to substantially increase x-risks or s-risks if given a specific influential position (that they’ve applied to, for example). Due to the dimensionalcontinuous nature of malevolent traits, the use of labels such as “malevolent” or “not malevolent” (which could be used if we wanted to calculate the sensitivity and specificity of measures of interest) would be artificial and would need to be decided carefully. Deciding whether and to what extent someone’s levels of malevolent traits would increase x-risks or s-risks would be a probabilistic assessment, but I think it would be an important part of assessing the risks involved in making high-stakes decisions about who to involve (or keep involved) in the key roles of high-impact organizations, groups, or projects.

  9. ^

    The comment above (by William McAuliffe) mentioned that it would be difficult to implement non-gameable measures at scale in a practical context. I think this concern should definitely be investigated in more depth, but I also think that the measures would not necessarily need to be implementable on a large scale to provide value. The measures would, of course, need to be studied well enough to have a good understanding of their specificity and sensitivity with respect to identifying actors most likely to increase x-risks or s-risks if given specific positions of power, but the relatively low numbers of these positions may mean that it could be justifiable to implement non-gameable tests of malevolent traits even if they are expensive and difficult to scale. The higher-stakes the position of interest, the more it would seem to be justified to invest significant resources into preventing malevolent actors from taking up that position.

  10. ^

    Some of the currently-gameable measures might become less gameable in the future. For example, using statistical or machine learning methods to model and correct for impression management and socially desirable responding in self-report surveys would at least increase the utility of self-report methods (i.e., it seems possible that such developments could elevate them from being worse than useless to being one component in a set of tests as part of a screening method). However, in view of William McAuliffe’s well-evidenced pessimism on this topic, I’m also less optimistic about this than I would otherwise have been. 

  11. ^

    There are several measures of sympathetic nervous system activation that I did not list above. This is mainly because sympathetic activation is arguably under some degree of voluntary control (e.g., one could hyperventilate to increase their level of sympathetic activation, and one can train oneself to vary one’s heart rate variability), and given that there are also multiple other factors (other than malevolent traits) that can contribute to variations in sympathetic activation, I did not list them as potential measures at this stage. 


    However, for completion, I’ll list a couple of specific examples in this category here. Heart rate (HR) orienting responses to images designed to induce threat and distress have been investigated among people with high versus low levels of callous-unemotional (CU) traits. Similarly, HR variability (HRV) appears to be altered among people with high levels of the boldness factor of psychopathy. In addition to heart rate variability, another indirect measure of sympathetic nervous system activation is skin conductance (SC) or electrodermal activity (EDA), which capitalizes on the fact that sweat changes the electrical conductance of the skin. Notwithstanding the noisiness and downsides of such measures, changes in electrodermal activity have been observed in psychopathy, antisocial personality disorder, and conduct disorder (across multiple studies). Both HRV and EDA measurements lack the temporal resolution of pupillometry, though, so if a measure of sympathetic activation was going to be investigated in the context of screening for malevolent traits, I predict that it would be more useful to use pupillometry in combination with specific emotional stimuli.

  12. ^

    This is extremely speculative and uncertain (sorry), but if it turns out that a better understanding of the correlations between different traits and outcomes of interest within humans could translate into a better ability to predict personality-like traits and behaviors of large language models (for example, if positive correlations between one trait and another among humans translated into positive correlations between those two traits in corpuses of training data and in LLM outputs), then research in this area could be relevant to evaluations of those models. However, this is just a very speculative additional benefit beyond the main sources of value discussed above.

trevor1 @ 2024-04-13T23:32 (+1)

Why are you pessimistic about eyetracking and body language? Although those might not be as helpful in experimental contexts, they're much less invasive per unit time, and people in high-risk environments can agree to have specific delineated periods of eyetracking and body language data collected while in the high-performance environments themselves such as working with actual models and code (ie not OOD environments like a testing room).

AFAIK analysts might find uses for this data later on, e.g. observing differences in patterns of change over time based on based on the ultimate emergence of high risk traits, or comparing people to others who later developed high risk traits (comparing people to large amounts of data from other people could also be used to detect positive traits from a distance), spotting the exact period where high risk traits developed and cross-referencing that data with the testimony of a high risk person who voluntarily wants other high risk people to be easier to detect, or depending on advances in data analysis, using that data to help refine controlled environment approaches like pupillometry data or even potentially extrapolating it to high-performance environments. Conditional on this working and being helpful, high-impact people in high-stakes situations should have all the resources desired to create high-trust environments.

Thomas Kwa @ 2024-04-11T21:54 (+17) in response to Understanding FTX's crimes

Thanks. In addition to lots of general information about FTX, this helps answer some of my questions about FTX: it seems likely that FTX/Alameda were never massively profitable except for large bets on unsellable assets (anyone have better information on this?); even though they had large revenues maybe much of it was spent dubiously by SBF. And the various actions needed to maintain a web of lies indicate that Caroline Ellison and Nishad Singh, and very likely Gary Wang and Sam Trabucco (who dropped off the face of the earth at the time of the bankruptcy [1]) were definitely complicit in fraud severe and obvious enough that any moral person, (possibly even a hardcore utilitarian, if it was true that FTX was consistently losing money), should have quit or leaked evidence of said fraud.

Four or five people is very different from a single bad actor, and this almost confirms for me that FTX belongs on the list of ways EA and rationalist organizations can basically go insane in harmful ways, alongside Leverage, Zizians and possibly others. It is not clear that FTX experienced a specifically EA failure mode, rather than the very common one in which power corrupts.

DPiepgrass @ 2024-04-13T23:20 (+9)

this almost confirms for me that FTX belongs on the list of ways EA and rationalist organizations can basically go insane in harmful ways,

I was confused by this until I read more carefully. This link's hypothesis is about people just trying to fit in―but SBF seemed not to try to fit in to his peer group! He engaged in a series of reckless and fraudulent behaviors that none of his peers seemed to want. From Going Infinite:

He had not been able to let Modelbot rip the way he’d liked—because just about every other human being inside Alameda Research was doing whatever they could to stop him. “It was entirely within the realm of possibility that we could lose all our money in an hour,” said one. One hundred seventy million dollars that might otherwise go to effective altruism could simply go poof. [...]

Tara argued heatedly with Sam until he caved and agreed to what she thought was a reasonable compromise: he could turn on Modelbot so long as he and at least one other person were present to watch it, but should turn it off if it started losing money. “I said, ‘Okay, I’m going home to go to sleep,’ and as soon as I left, Sam turned it on and fell asleep,” recalled Tara. From that moment the entire management team gave up on ever trusting Sam.

Example from Matt Levine:

There is an anecdote (which has been reported before) from the early days of Alameda Research, the crypto trading firm that Bankman-Fried started before his crypto exchange FTX, the firm whose trades with FTX customer money ultimately brought down the whole thing. At some point Alameda lost track of $4 million of investor money, and the rest of the management team was like “huh we should tell our investors that we lost their money,” and Bankman-Fried was like “nah it’s fine, we’ll probably find it again, let’s just tell them it’s still here.” The rest of the management team was horrified and quit in a huff, loudly telling the investors that Bankman-Fried was dishonest and reckless.

It sounds like SBF drove away everyone who couldn't stand his methods until only people who tolerated him were left. That's a pretty different way of making an organization go insane.

It doesn't seem like this shouldn't be an EA failure mode when the EA community is working well. Word should have gotten around about SBF's shadiness and recklessness, leading to some kind of investigation before FTX reached the point of collapse. The first person I heard making the case against SBF post-collapse was an EA (Rob Wiblin?), but we were way too slow. Of course it has been pointed out that many people who worked with / invested in FTX were fooled as well, so what I wonder about is: why weren't there any EA whistleblowers on the inside? Edit: was it that only four people plus SBF knew about FTX's worst behaviors, and the chance of any given person whistle-blowing in a situation like that is under 25%ish? But certainly more people than that knew he was shady. Edit 2: I just saw important details on who knew what. P.S. I will never get used to the EA/Rat tendency to downvote earnest comments, without leaving comments of their own...

Linch @ 2024-04-13T22:24 (+4) in response to Should the main text of the write-ups of Open Philanthropy’s large grants be longer than 1 paragraph?

Grantees are obviously welcome to do this. That said, my guess is that this will make the forum less enjoyable/useful for the average reader, rather than more. 

David T @ 2024-04-13T22:59 (+3)

I think a dedicated area would minimise the negative impact on people that aren't interested whilst potentially adding value (to prospective applicants in understanding what did and didn't get accepted, and possibly also to grant assessors if there was occasional additional insight offered by commenters)

I 'd expect there would be some details of some applications that wouldn't be appropriate to share on a public forum though

Elham Sahal @ 2024-04-13T22:34 (+1) in response to Introducing EA in Arabic

I am so happy to see this :D The translation and the website are amazing. Have you considered sharing the experiences of people from the MENA region who are open about EA ideas within their community? (It's something I am personally struggling with)

Vasco Grilo @ 2024-04-13T14:13 (+2) in response to Should the main text of the write-ups of Open Philanthropy’s large grants be longer than 1 paragraph?

We've started working on this [making some application public], but no promises. My guess is that making public the rejected applications is more valuable than accepted ones, eg on Manifund. Note that grantees also have the option to upload their applications as well (and there are less privacy concerns if grantees choose to reveal this information).

Manifund already has quite a good infrastructure for sharing grants. However, have you considered asking applicants to post a public version of their applications on EA Forum? People who prefer to remain anonymous could use an anonymous account, and anonymise the public version of their grant. At a higher cost, there would be a new class of posts[1] which would mimic some of the features of Manifund, but this is not strictly necessary. The posts with the applications could simply be tagged appropriately (with new tags created for the purpose), and include a standardised section with some key information, like the requested amount of funding, and the status of the grant (which could be changed over time editing the post).

The idea above is inspired by some thoughts from Hauke Hillebrandt.

  1. ^

    As of now, there are 3 types, normal posts, question posts and linkposts/crossposts.

Linch @ 2024-04-13T22:24 (+4)

Grantees are obviously welcome to do this. That said, my guess is that this will make the forum less enjoyable/useful for the average reader, rather than more. 

Joseph Lemien @ 2024-04-13T19:04 (+2) in response to a guy named josh's Quick takes

That actually seems like a really strong signal of something important: can people improve, if given a modest amount of guidance/support. I'd certainly be interested in hiring someone who does rather than someone who doesn't.

But I'm also impressed that you provide feedback to candidates consistently. I've always thought that it would be something fairly time-consuming, even if you set up a system to provide feedback in a fairly standardized way. Would you be willing to share a bit about how you/your team does feedback for rejected job applicants?

John Salter @ 2024-04-13T22:22 (+2)

I view our hiring process as a constant work in progress, and we look back at the application process of everyone after their time with us, potatoes and gems alike, and try figure out how we could have told ahead of time. Part of that is writing up notes. We use chatgpt to make the notes more sensitive and send them to the applicant. 

Caveat: We only do this for people who show some promise of future admission. 

Peter Wildeford @ 2024-04-13T18:26 (+15) in response to Peter Wildeford's Quick takes

The TV show Loot, in Season 2 Episode 1, introduces a SBF-type character named Noah Hope DeVore, who is a billionaire wonderkid who invents "analytic altruism", which uses an algorithm to determine "the most statistically optimal ways" of saving lives and naturally comes up with malaria nets. However, Noah is later arrested by the FBI for wire fraud and various other financial offenses.

Jason @ 2024-04-13T21:56 (+2)

I wonder if anyone else will getting a thinly veiled counterpart -- given that the lead character of the show seems somewhat based on MacKenzie Scott, this seems to be maybe a thing for the show.

Zar Amareis @ 2024-04-10T15:25 (+1) in response to AI things that are perhaps as important as human-controlled AI (Chi version)

Hello Chi, thanks for sharing the interesting list and discussion questioning the focus on “human control of AGI”.  For readers, a friend shared this post with me, so the 'you' refers to this friend :-).

I wrote a short post in line with the one you shared on “Aligning the Aligners”: “Solving The AI Control Problem Amplifies The Human Control Problem”.  The animal advocacy inclusive AI post makes a good point, too.  I’ve also written about how much “AI Safety” lore is [rather unethical](https://gardenofminds.art/bubbles/why-much-ai-ethics-lore-is-highly-unethical/) by species-agnostic standards.  Need we mention how “child safety” refers to the safety of children, so the term we use is a misnomer?  It should be “Human Safety from AI”.

I believe these other concerns can be **more important** than aiming to “keep AI under human control”.  How about increasing peace on Earth and protopic support for the thriving of her sentient beings?  Avoiding total extinction of all organic life on Earth is one of the purported reasons why “it’s so important to control AI”, right?  If beneficial outcomes for humans and animals could be attained without controlling AI, why would it still be important to keep them under control?  Especially this community should be mindful not to lose sight of the *primary goals* for the *proxy goals*.  I’d argue that *control* is a proxy goal in this case.

While the topic of *which* humans get to “control AI” is introduced, curiously “democracy” doesn’t show up!  Nuclear weapons can entrench the difference between those who do and don’t have nuclear weapons.  Couldn’t control over AGI by a few major world powers further solidify their rule in the “nuclear and artificial intelligence order”?  There are huge differences between treaties on the use of nuclear materials and AI, too.  AGI will be incredibly useful all across the socio-economic spectrum, which poses an additional challenge for channeling all AGI R&D through a few international research laboratories.

Some of the ideas seem to fall under the general header of “making better AGI”.  We’d like to create AGIs that can effectively reason about complex philosophical topics, investigate epistemology, and integrate the results into itseslf.  [Question: do such capacities point in the direction of open ended intelligence, which runs counter to *control*?]  Ideally, compared with humans, neuro-symbolic AGI will be in better situations for meta-cognition, self-reflective reasoning, truth-preserving reasoning, and explicitly following decision-theoretic approaches.  As we approach *strong AGI*, many smaller problems could begin to weaken or fall away.  Take *algorithmic bias* and how susceptible the systems are to training data (— which is not dissimilar from human susceptibility to influence —): metacognition should allow the AGI to identify *morally positive and negative* examples in the training data so that instead of parroting *poor behavior*, the negative examples are clearly understood as what they are.  Even where the data distribution is *discriminatorily biased*, the system can be aware of higher principles of equality across certain properties.  “Constitutional AI” seems to be heading in this direction already 🙂🤓.

The “human safety from AI” topics often seem to, imo, strongly focus on ensuring that there are no “rogue” AIs with less attention given to the question of what to do about the likely fact that there will be some.  Warfare will probably not be solved by the time AGI is developed, too.  Hopefully, we can work with AI to help move toward peace and increase our wisdom.  Do we wish for human wisdom-enhancing AI to be centralized and opaque or decentralized and open so that people are *‘democratically’* capable of choosing how to develop their wisdom and compassion?

A cool feature of approaches such as *constitutional AI* and *scalable oversight* is that they lean in the direction of fostering an *ecosystem of AI entities* that keep each other in check with reciprocal accountability.  I highly recommend David Brin’s talk at BGI24 on this topic.  AI personality profiling falls under this header, too.  Some of the best approaches to “AI control” actually share an affinity with the “non-control-based” approaches.  They may further benefit from the increased diversity of AGI entities so that we’re less likely to suffer any particular AGI system undergoing some perversities in its development?

A question I ponder about is: to what extent are some of the control and non-control-based approaches to “harmonious human-AI co-existence” mutually incompatible?  A focus on open-ended intelligence and motivations with decentralized compute infrastructures and open source code, even with self-owned robot/AGI entities, so that no single group can control the force of the intelligence explosion on Earth is antithetical to some of the attempts at wielding human control over AGI systems.  These *liberal* approaches can also aim to help with sociopolitical power struggles on Earth, too, aiming to avoid the blossoming of AGI further solidifying the current power structures.  I believe there is some intersection of approaches, too.

The topic of which value systems bias us toward beneficial and harmful outcomes is also important (and is probably best discussed w/o worrying about whether they provide safety guarantees).  In the other comment, I mentioned the idea that “careless” goals will likely lead to the manifestation of “dark factor traits”.  Some goal formulations are more compatible with satisficing and decreased marginal returns, too, which would help with the fears that “AGIs wipe all humans out to maximize their own likelihood of surviving” (which, imo, seems to assume some *stupidity* on the part of the *superintelligent* AGI 🤷‍♂️).  Working with increasingly *unfussy* preferences is probably wise, too.  Human requests of AGI could be *fussy* — whereas allowing AGis to refactor their own motivations to be less fussy and more internally coherent leads away from *ease of controllability*

A big pink elephant in the room is the incentive structures into which we’re introducing proto-AGI systems.  At present, helping people via APIs and chat services is far better than I may have feared it could be.  “*Controlled AI” used for a misaligned corporate entity* may do massive harm nonetheless.  Let’s have AI nurses, scientists, peace negotiators, etc.

David Wood summarized the “reducing risks of AI catastrophe” sessions at BGI24.  Suggestion #8 is to change mental dispositions around the world, which is similar to reducing human malevolence.  Such interventions done in a top-down manner can seem very, very creepy — and even more so if advanced proto-AGI is used for this purpose, directed by some international committee.  The opacity of the process could make things worse.  Decentralized, open-source AGI “personal life coaches” could come across very differently!

Transparency as to how AGI projects are going is one “safety mechanism” most of us may agree on?  There may be more.  Should such points of agreement receive more attention and energy?

During our discussions, you said that “AI Safety is helpful” (or something similar).  I might question the extent to which it’s helpful.  

For example, let’s say that “ASI is probably theoretically uncontrollable” and “the kinds of guarantees desired are probably unattainable”.  If so, how valuable was the “AI Safety” work spent on trying to find a way to guarantee human safety?  Many attendees of the AGI conference would probably tell you that it’s “obviously not likely to work”, so much time was spent confirming the obvious.  Are all efforts hopeless?  No.  Yet stuff like “scalable oversight” would fall under the category of “general safety schemes for multi-agent systems”, not so specific to “AGI.

What if we conclude that it’s insufficient to rely on control-centric techniques, especially given the bellicose state of present human society power dynamics?  An additional swathe of “AI Safety” thought may fall by the wayside.  Open, liberal approaches will require different strategies.  How important is it to delve deep into thought experiments about possible sign flips, as if we’re unleashing one super AGI and someone got the *moral guidance* wrong at the last second? — whoops, game over!  

Last week I was curious what EA folk thought about the Israel-Hamas war and found one discussion about how a fresh soldier realized that most of the “rationality optimization” techniques he’s practiced are irrelevant, approaches to measuring suffering he’d taken appear off, attempts to help can backfire, etc: “models of complex situations can be overly simplistic and harmful”.  How do we know a lot of “AI x-risk” discussions aren’t naively Pascal-mugging people?  Simple example: discussing the precautions we should take to slow down and ensure the wise development and deployment of AGI assuming idealistic governance and geopolitical models without adequately dealing with the significant question of “*which humans get to influence AGIs and how”.*

How confident are we that p(doom|pause) is significantly different from p(doom|carry on)?  That it’s *necessarily lower*?  How confident should we be that international deliberation will go close enough to ideally?  If making such rosy assumptions, why not assume people will responsibly proceed as is?  Advocating *pausing* until we’re sure it’s sufficiently low is a choice with yet more difficult-to-predict consequences?  What if I think that p(doom|centralized AGI) is significantly higher than p(doom|decentralized AGI)?  Although the likelihood of ‘smaller scale’ catastrophes may be higher?  And p(centralized AGI|pause) is also significantly higher?  Clearly, we need some good simulation models to play with this stuff :- D.  To allow our budding proto-AGI systems to play with!  :D.  The point is that fudging numbers for overly simplistic estimates of *doom* could easily lead to Pascal-mugging people, all while sweeping many relevant real-world concerns under the rug.  Could we find ourselves in some weird scenarios where most of the "AI Safety" thought thus far turns out to be "mostly irrelevant"?

A common theme among my AGI dev friends is that “humans in control of highly advanced yet not fully general AI may be far, far more dangerous than self-directed full AGI”.  Actually enslaved “AGI systems” could be even worse.  Thinking in this direction could lead to the conclusion that p(doom|pause) is not necessarily lower.

As for concluding remarks, it seems that much of this work focuses on “building better AGI”.  Then there’s “working with AI to better humanity”.  My hunch is that any work improving *peace on Earth* will likely enhance p(BGI).  Heck, if we could solve the *misaligned corporation problem*, that would be fabulous!  

One cool feature of the non-control-based approaches is that they may be more worthwhile investments even if only partial progress is made.  Increasing the capacity for deep philosophical reasoning and decreasing the fussiness of the goals of *some* AGI systems may already pay off and increase p(BGI) substantially.  With control-centric approaches, I often see the attitude that we “must nail it or we’re doomed”, as if there’s no resilience for failure.  If a system breaks out, then we’re doomed (especially because we only focused on securing control without improving the core properties of the AGI’s mind.

I’ll add that simple stuff like developing “artificial bodhisattvas” embracing “universal loving care” as suggested in Care as the Driver of Intelligence is worthwhile and not control-based.  Stuart Russell and David Hanson both advocate (via different routes) the development of AGI systems that enter into reciprocal, empathic relationships with humans to *learn to care for us in practice*, querying us for feedback as to their success.  I personally think these approaches should receive much more attention (and, afaict, RLHF loosely points in this direction).

Daniel_Friedrich @ 2024-04-13T20:57 (+1)

Let me dox myself as the addressee. :) Many thanks for the response. I really value that you take seriously the possible overlap of policies and research agendas covered by AI safety and your own approach.

I totally agree that "control is a proxy goal" and I believe the AI safety mainstream does as well, as it's the logical consequence of Bostrom's principle of epistemic deference. Once we have an AI that reliably performs tasks in the way they were intended, the goal should be to let it shape the world according to the wisest interpretation of morality it will find. If you tried to formalize this framing, as well as the proposal to inject it with "universal loving care", I find it very likely that you would build the same AI.

So I think our crux doesn't concern values, which is a great sign of a tractable disagreement.
I also suppose we could agree on a simple framework of factors that would be harmful on the path to this goal from the perspectives of:

a) safety (AI self-evolves to harm)
b) power / misuse (humans do harm with AI)
c) sentience (AI is harmed)
d) waste (we fail to prevent harm)

Here's my guess on how the risks compare. I'd be most curious whether you'd be able to say if the model I've sketched out seems to track your most important considerations, when evaluating the value of AI safety efforts - and if so, which number would you dispute with the most certainty.

One disclaimer: I think it's more helpful to think about specific efforts, rather than comparing the AI safety movement on net. Policy entails a lot of disagreement even within AI safety and a lot of forces clashed at the negotiations around the existing policies. I mentioned that I like the general, value-uncertain framework of the EU AI act but the resulting stock of papers isn't representative of typical AI safety work.

In slight contrast, the community widely agrees that technical AI safety research would be good if successful. I'd argue that would manifest in a robust a decrease of risk in all of the highlighted perspectives (a-d). Interpretability, evals and scaling all enable us to resolve the disagreements in our predictions regarding the morality of emergent goals and of course, and of course, work on "de-confusion" about the very relationship between goals, intelligence and morality seems beneficial regardless of our predictions and to also quite precisely match your own focus. :)

So far, my guess is that we mostly disagree on

1) Do the political AI safety efforts lead to the kind of centralization of power that could halt our cosmic potential?

  • I'd argue the emerging regulation reduces misuse / power risks in general. Both US and EU regulations combine monitoring of tech giants with subsidies, which is a system that should accelerate beneficial models, while decelerating harmful ones. This system, in combination with compute governance, should also be effective in the misuse risks posed by terrorists and random corporations letting superhuman AIs with random utility functions evolve with zero precautions.

2) Would [a deeply misaligned] AGI be "stupid" to wipe out humans, in its own interest?

  • I don't see a good reason to but I don't think this is the important question. We should really be asking: Would a misaligned AGI let us fulfill the ambition of longtermism (of optimally populating cosmos with flourishing settlements)?

3) Is it "simple stuff" to actually put something like "optimal morality" or "universal loving care" into code of a vastly more intelligent entity, which is so robust that we can entrust it our cosmic potential?

Arpit @ 2024-04-13T20:40 (+1) in response to This is why we can’t have nice laws

To me, the subsidies that farmers receive is yet another example of how government taking taxpayer money and giving it to people/businesses can lead to outcomes which a majority of people don't approve of. I hope this example makes people more likely to entertain a deontological objection to any kind of government subsidy.

yanni kyriacos @ 2024-04-12T10:14 (+1) in response to Yanni Kyriacos's Quick takes

What are some historical examples of a group (like AI Safety folk) getting something incredibly wrong about an incoming Technology? Bonus question: what led to that group getting it so wrong? Maybe there is something to learn here.

Ives Parr @ 2024-04-13T20:19 (+9)

This is probably a good exercise. I do want to point out a common bias about getting existential risks wrong. If someone was right about doomsday, we would not be here to discuss it. That is a huge survivorship bias. Even catestrophic events which lessen the number of people are going to be systemically underestimated. This phenomenon is the anthropic shadow which is relevant to an analysis like this. 

Ian Turner @ 2024-04-13T19:51 (+3) in response to Jeroen De Ryck's Quick takes

The posts do have the “April Fool’s Day” tag right at the beginning?

Ives Parr @ 2024-04-13T20:15 (+6)

I think [April Fools] added to the title might be a good addition since the tag is hard to see. 

Jeroen De Ryck @ 2024-04-13T16:43 (+53) in response to Jeroen De Ryck's Quick takes

Why are April Fools jokes still on the front page? On April 1st, you expect to see April Fools' posts and know you have to be extra cautious when reading strange things online. However, April 1st was 13 days ago and there are still two posts that are April Fools posts on the front page. I think it should be clarified that they are April Fools jokes so people can differentiate EA weird stuff from EA weird stuff that's a joke more easily. Sure, if you check the details you'll see that things don't add up, but we all know most people just read the title or first few paragraphs.

Ian Turner @ 2024-04-13T19:51 (+3)

The posts do have the “April Fool’s Day” tag right at the beginning?

ABishop @ 2024-04-13T01:59 (+3) in response to ABishop's Quick takes

I would like to estimate how effective free hugs are. Can anyone help me?

Joseph Lemien @ 2024-04-13T19:13 (+5)

Haha. Well, I guess I would first ask effective at what? Effective at giving people additional years of healthy & fulfilling life? Effective at creating new friendships? Effective at making people smile?

I haven't studied it at all, but my hypothesis that it is the kind of intervention that is  similar to "awareness building," but it doesn't have any call to action (such as a donation). So it is probably effective in giving people a nice experience for a few seconds, and maybe improving their mood for a period of time, but it probably doesn't have longer-lasting effects. From a cursory glance at Google Scholar, it looks like there hasn't been much research on free hugs.

Jamie_Harris @ 2024-04-13T12:27 (+11) in response to a guy named josh's Quick takes

I tried doing this a while back. Some things I think I worried about at the time:

(1) disheartening people excessively by sending them scores that seem very low/brutal, especially if you use an unusual scoring methodology (2) causing yourself more time costs than it seems like at first, because (a) you find yourself needing to add caveats or manually hide some info to make it less disheartening to people, (b) people ask you follow-up questions (3) exposing yourself to some sort of unknown legal risk by saying something not-legally-defensible about the candidate or your decision-making.

(1) turned out to be pretty justified I think, e.g. at least one person expressing upset/dissatisfaction at being told this info. (2) definitely happened too, although maybe not all that many hours in the grand scheme of things (3) we didn't get sued but who knows how much we increased the risk by

Joseph Lemien @ 2024-04-13T19:07 (+2)

Jamie, I've been contemplating writing up a couple of informal "case study"-type reports of different hiring practices. My intention/thought process would be to allow EA orgs to learn about how several different orgs do hiring, to highlight some best practices, and generally to allow/encourage organizations to improve their methods. How would you feel about writing up a summary or having a call with me to allow me to understand how you tried giving feedback and what specific aspects caused challenges?

John Salter @ 2024-04-13T15:09 (+2) in response to a guy named josh's Quick takes

Have also tried this, although most our applicants aren't EAs. People who reapply given detailed feedback usually don't hit the bar.

We still do it, in part because we think it's good for the applicants, and in part because people who make a huge improvement attempt 2 usually make strong long-term hires

Joseph Lemien @ 2024-04-13T19:04 (+2)

That actually seems like a really strong signal of something important: can people improve, if given a modest amount of guidance/support. I'd certainly be interested in hiring someone who does rather than someone who doesn't.

But I'm also impressed that you provide feedback to candidates consistently. I've always thought that it would be something fairly time-consuming, even if you set up a system to provide feedback in a fairly standardized way. Would you be willing to share a bit about how you/your team does feedback for rejected job applicants?

Stephen Clare @ 2024-04-12T14:47 (+20) in response to Stephen Clare's Quick takes

Many organizations I respect are very risk-averse when hiring, and for good reasons. Making a bad hiring decision is extremely costly, as it means running another hiring round, paying for work that isn't useful, and diverting organisational time and resources towards trouble-shooting and away from other projects. This leads many organisations to scale very slowly.

However, there may be an imbalance between false positives (bad hires) and false negatives (passing over great candidates). In hiring as in many other fields, reducing false positives often means raising false negatives. Many successful people have stories of being passed over early in their careers. The costs of a bad hire are obvious, while the costs of passing over a great hire are counterfactual and never observed.

I wonder  whether, in my past hiring decisions, I've properly balanced the risk of rejecting a potentially great hire against the risk of making a bad hire. One reason to think we may be too risk-averse, in addition to the salience of the costs, is that the benefits of a great hire could grow to be very large, while the costs of a bad hire are somewhat bounded, as they can eventually be let go.

Joseph Lemien @ 2024-04-13T18:37 (+2)

It looks like there are two people who voted disagree with this. I'm curious as to what they disagree with. Do they disagree with the claim that some organizations are "very risk-averse when hiring"? Do they disagree with the claim that "reducing false positives often means raising false negatives"? That this has a causal effect with organisations scale slowly? Or perhaps that "the costs of a bad hire are somewhat bounded"? I would love for people who disagree vote to share information regarding what it is they disagree with.

Peter Wildeford @ 2024-04-13T18:26 (+15) in response to Peter Wildeford's Quick takes

The TV show Loot, in Season 2 Episode 1, introduces a SBF-type character named Noah Hope DeVore, who is a billionaire wonderkid who invents "analytic altruism", which uses an algorithm to determine "the most statistically optimal ways" of saving lives and naturally comes up with malaria nets. However, Noah is later arrested by the FBI for wire fraud and various other financial offenses.

Kaspar Brandner @ 2024-04-13T08:10 (+1) in response to Repugnance and replacement

Okay, having an initial start world (call it S), that is assumed to be current, makes it possible to treat the other worlds (futures) as choices. So S has 1 million people, but how much utility points do they have each? Something like 10? Then A and A+ would be an improvement for them, and Z would be worse (for them).

I don't intend for you to be able to switch from A+ or Z to A by killing people. A is defined so that the extra people never exist. It's the way things could turn out. Creating extra people and then killing them would be a different future.

But if we can't switch worlds in the future that does seem like an unrealistic restriction? Future people have just as much control over their future as we have over ours. Not being able to switch worlds in the future (change the future of the future) would mean we couldn't, once we were at A+, switch from A+ to a more "fair" future (like Z). Since not-can implies not-ought, there would then be no basis in calling A+ unfair, insofar "unfair" means that we ought to switch to a more fair future.

The fairness consideration assumes utility can be redistributed, like money. Otherwise utility would presumably be some inherent property of the brains of people, and it wouldn't be unfair to anyone to not having been born with a different brain (assuming brains can't be altered).

MichaelStJules @ 2024-04-13T17:32 (+2)

Does it matter to you what the starting welfare levels of the 1 million people are? Would your intuitions about which outcome is best be different?

There are a few different perspectives you could take on the welfare levels in the outcomes. I intended them to be aggregate whole life welfare, including the past, present and future. Not just future welfare, and not welfare per future moment, day or year or whatever. But this difference often doesn't matter.

Z already seems more fair than A+ before you decide which comes about; you're deciding between them ahead of time, not (necessarily just) entering one (whatever that would mean) and then switching.

I think, depending on the details, e.g. certain kinds of value lock-in, say because the extra people will become unreachable, it can be realistic to be unable to switch worlds in the future. Maybe the extra people are sent out into space, and we're deciding how many of the limited resources they'll be sent off with, which will decide welfare levels. But the original million people are better off in A+, because the extra people will eliminate some threat to the current people, or the original people at least have some desire for the extra people to exist, or the extra people will return with some resources.

Or, it could be something like catastrophic climate change, and the extra people are future generations. We can decide not to have children (A), go with business as usual (A+) or make serious sacrifices now (Z) to slightly better their lives.

No matter how the thought experiment is made more concrete, if you take the welfare levels to be aggregate whole lifetime welfare, then it's definitely not possible to switch from Z or A+ to A after the extra people have already come to exist. A describes a world in which they never existed. If you wanted to allow switching later on, then you could allow switching every way except to A.

If you want an option where all the extra people are killed early, that could look like A+, but worse than A+ and A for the original million people, because they had to incur the costs of bringing about all the extra people and then the costs of killing them. It would also be no better than A+ for the extra people (but we could make it worse, or equal for them).

titotal @ 2024-04-12T15:54 (+12) in response to Yanni Kyriacos's Quick takes

In the 90's and 2000's, many people such as Eric Drexler were extremely worried about nanotechnology and viewed it as an existential threat through the "gray goo" scenario. Yudkowsky predicted drexler style nanotech would occur by 2010, using very similar language to what he is currently saying about AGI. 

It turned out they were all being absurdly overoptimistic about how soon the technology would arrive, and the whole drexlerite nanotech project flamed out by the end of the 2000's and has pretty much not progressed since. I think a similar dynamic playing out with AGI is less likely, but still very plausible. 

Habryka @ 2024-04-13T17:10 (+2)

Do you have links to people being very worried about gray goo stuff?

(Also, the post you link to makes this clear, but this was a prediction from when Eliezer was a teenager, or just turned 20, which does not make for a particularly good comparison, IMO)

Kaspar Brandner @ 2024-04-13T08:17 (+1) in response to Repugnance and replacement

But it doesn't seem to matter whether X or Y (or neither) is actually current, everyone should be able to agree whether, e.g., "switching from X to Y is bad" is true or not. The switch choices always hypothetically assume that the first world (in this case X) is current, because that's where the potential choice to switch is made.

MichaelStJules @ 2024-04-13T16:47 (+2)

Between A and Z, the people in A are much better off in A, and the extra people in Z are much better off in Z (they get to exist, with positive lives). It seems like they'd disagree about switching, if everyone only considers the impact on their own welfare.

(Their welfare levels could also be the degree of satisfaction of their impartial or partially other-regarding preferences, but say they have different impartial preferences.)

Stephen Clare @ 2024-04-12T14:47 (+20) in response to Stephen Clare's Quick takes

Many organizations I respect are very risk-averse when hiring, and for good reasons. Making a bad hiring decision is extremely costly, as it means running another hiring round, paying for work that isn't useful, and diverting organisational time and resources towards trouble-shooting and away from other projects. This leads many organisations to scale very slowly.

However, there may be an imbalance between false positives (bad hires) and false negatives (passing over great candidates). In hiring as in many other fields, reducing false positives often means raising false negatives. Many successful people have stories of being passed over early in their careers. The costs of a bad hire are obvious, while the costs of passing over a great hire are counterfactual and never observed.

I wonder  whether, in my past hiring decisions, I've properly balanced the risk of rejecting a potentially great hire against the risk of making a bad hire. One reason to think we may be too risk-averse, in addition to the salience of the costs, is that the benefits of a great hire could grow to be very large, while the costs of a bad hire are somewhat bounded, as they can eventually be let go.

Joseph Lemien @ 2024-04-13T16:44 (+2)

Forgive my rambling. I don't have much to contribute here, but I generally want to say A)I am glad to see other people thinking about this, and B) I sympathize with the difficulty

The "reducing false positives often means raising false negatives" is one of the core challenges in hiring. Even the researchers who investigate the validity of various methods and criteria in hiring don't have a great way to deal with it this problem. Theoretically we could randomly hire 50% of the applicants and reject 50% of them, and then look at how the new hires perform compared to the rejects one year later. But this is (of course) infeasible. And of course, so much of what we care about is situationally specific: If John Doe thrives in Organizational Culture A performing Role X, that doesn't necessarily mean he will thrive in Organizational Culture B performing Role Y.

I do have one suggestion, although it isn't as good of a suggestion as I would like. Ways to "try out" new staff (such as 6-month contacts, 12-month contracts, internships, part-time engagements, and so on) can let you assess with greater confidence how the person will perform in your organization in that particular role much better than a structured interview, a 2-hour work trial test, or a carefully filled out application form. But if you want to have a conversation with some people that are more expert in this stuff I could probably put you in touch with some Industrial Organizational Psychologists who specialize in selection methods. Maybe a 1-hour consultation session would provide some good directions to explore?

I've shared this image[1] with many people, as I think it is a fairly good description of the issue. I generally think of one of the goals of hiring to be "squeezing" this shape to get as much off the area as possible in the upper right and lower left, and to have as little as possible in the upper left and lower right. We can't squeeze it infinitely thin, and there is a cost to any squeezing, but that is the general idea.

.

  1. ^
Jeroen De Ryck @ 2024-04-13T16:43 (+53) in response to Jeroen De Ryck's Quick takes

Why are April Fools jokes still on the front page? On April 1st, you expect to see April Fools' posts and know you have to be extra cautious when reading strange things online. However, April 1st was 13 days ago and there are still two posts that are April Fools posts on the front page. I think it should be clarified that they are April Fools jokes so people can differentiate EA weird stuff from EA weird stuff that's a joke more easily. Sure, if you check the details you'll see that things don't add up, but we all know most people just read the title or first few paragraphs.

dotsam @ 2024-03-27T12:09 (+27) in response to Nick Bostrom’s new book, “Deep Utopia”, is out today

Anyone know if there'll be an audiobook?

dotsam @ 2024-04-13T16:29 (+1)

Audiobook version is "in the works", coming "probably in a few months": https://youtu.be/KOHO_MKUjhg?feature=shared&t=2997

Aperson @ 2024-04-13T15:10 (+1) in response to Is capitalism the root of all evil?

This is a fair question to ask, and an important conversation to be had. Both the title and the question under it. It is rather obvious that the "all" in the title question shouldn't be taken literally, just like you wouldn't take in its absolutely literal sense the "everyone" in "everyone knows that Earth revolves around the Sun".

a guy named josh @ 2024-04-12T11:13 (+1) in response to a guy named josh's Quick takes

Would love for orgs running large-scale hiring rounds (say 100+ applicants) to provide more feedback to their (rejected) applicants. Given that in most cases applicants are already being scored and ranked on their responses, maybe just tell them their scores, their overall ranking and what the next round cutoff would have been - say: prompt 1 = 15/20, prompt 2 = 17.5/20, rank = 156/900, cutoff for work test at 100.

Since this is already happening in the background (if my impression here is wrong please lmk), why not make the process more transparent and release scores - with what seems to be very little extra work required (beyond some initial automation). 

John Salter @ 2024-04-13T15:09 (+2)

Have also tried this, although most our applicants aren't EAs. People who reapply given detailed feedback usually don't hit the bar.

We still do it, in part because we think it's good for the applicants, and in part because people who make a huge improvement attempt 2 usually make strong long-term hires

trevor1 @ 2024-04-13T15:00 (+11) in response to To what extent & how did EA indirectly contribute to financial crime - and what can be done now? One attempt at a review

The crypto section here didn't seem to adequately cover a likely root cause of the problem. 

The "dark side" of crypto is a dynamic called information asymmetry; in the case of Crypto, it's that wealthier traders are vastly superior at buying low and selling high, and the vast majority of traders are left unaware of how profoundly disadvantaged they are in what is increasingly a zero-sum game. Michael Lewis covered this concept extensively in Going Infinite, the Sam Bankman-Fried book.

This dynamic is highly visible to those in the crypto space (and quant/econ/logic people in general who catch so much as a glimpse), and many elites in the industry like Vitalik and Altman saw it coming from a mile away and tried to find/fund technical solutions e.g. to fix the zero-sum problem e.g. Vitalik's d/acc concept

It was very clear that SBF also appeared to be trying to find technical solutions, rather than just short-term profiteering, but his decision to commit theft points towards the hypothesis that this was superficial.

I can't tell if there's any hope for crypto (I only have verified information on the bad parts, not the good parts if there are any left), but if there is, it would have to come from elite reformers, who are these types of people (races to the bottom to get reputation and outcompete rivals) and who each come with the risk of being only superficially committed.

Hence why the popular idea of "cultural reform" seems like a roundaboutly weak plan. EA needs to get better at doing the impossible on a hostile planet, including successfully sorting/sifting through accusationspace/powerplays/deception, and evaluating the motives of powerful people in order to determine safe levels of involvement and reciprocity. Not massive untested one-shot social revolutions with unpredictable and irreversible results.

Linch @ 2024-04-09T23:22 (+6) in response to Should the main text of the write-ups of Open Philanthropy’s large grants be longer than 1 paragraph?

(Appreciate the upvote!)

At a high level, l I'm of the opinion that we practice better reasoning transparency than ~all EA funding sources outside of global health, e.g. a) I'm responding to your thread here and other people have not, b) (I think) people can have a decent model of what we actually do rather than just an amorphous positive impression, and c) I make an effort of politely delivering messages that most grantmakers are aware of but don't say because they're worried about flack. 

It's really not obvious that this is the best use of limited resources compared to e.g. engaging with large donors directly or having very polished outwards-facing content, but I do think criticizing our lack of public output is odd given that we invest more in it than almost anybody else.

(I do wonder if there's an effect where because we communicate our overall views so much, we become a more obvious/noticeable target to criticize.) 

This illustrates CE shares much more information about the interventions they support than EA Funds' shares about the grants for which there are longer write-ups. So it is possible to have a better picture of CE's work than EA Funds'. This is not to say CE's donors actually have a better picture of CE's work than EA Funds' donors have of EA Funds' work. I do not know how whether CE's donors look into their reports.

Well, I haven't read CE's reports. Have you?

I think you have a procedure-focused view where the important thing is that articles are written, regardless of whether they're read. I mostly don't personally think it's valuable to write things people don't read. (though again for all I know CE's reports are widely read, in which case I'd update!) And it's actually harder to write things people want to read than to just write things.

(To be clear, I think there are exceptions. Eg all else equal, writing up your thoughts/cruxes/BOTECs are good even if nobody else reads them because it helps with improving quality of thinking). 

How about just making some applications public, as Austin suggested? I actually think it would be good to make public the applications of all grants EA Funds makes, and maybe even rejected applications.

We've started working on this, but no promises. My guess is that making public the rejected applications is more valuable than accepted ones, eg on Manifund. Note that grantees also have the option to upload their applications as well (and there are less privacy concerns if grantees choose to reveal this information).

Vasco Grilo @ 2024-04-13T14:13 (+2)

We've started working on this [making some application public], but no promises. My guess is that making public the rejected applications is more valuable than accepted ones, eg on Manifund. Note that grantees also have the option to upload their applications as well (and there are less privacy concerns if grantees choose to reveal this information).

Manifund already has quite a good infrastructure for sharing grants. However, have you considered asking applicants to post a public version of their applications on EA Forum? People who prefer to remain anonymous could use an anonymous account, and anonymise the public version of their grant. At a higher cost, there would be a new class of posts[1] which would mimic some of the features of Manifund, but this is not strictly necessary. The posts with the applications could simply be tagged appropriately (with new tags created for the purpose), and include a standardised section with some key information, like the requested amount of funding, and the status of the grant (which could be changed over time editing the post).

The idea above is inspired by some thoughts from Hauke Hillebrandt.

  1. ^

    As of now, there are 3 types, normal posts, question posts and linkposts/crossposts.

a guy named josh @ 2024-04-12T11:13 (+1) in response to a guy named josh's Quick takes

Would love for orgs running large-scale hiring rounds (say 100+ applicants) to provide more feedback to their (rejected) applicants. Given that in most cases applicants are already being scored and ranked on their responses, maybe just tell them their scores, their overall ranking and what the next round cutoff would have been - say: prompt 1 = 15/20, prompt 2 = 17.5/20, rank = 156/900, cutoff for work test at 100.

Since this is already happening in the background (if my impression here is wrong please lmk), why not make the process more transparent and release scores - with what seems to be very little extra work required (beyond some initial automation). 

Jamie_Harris @ 2024-04-13T12:27 (+11)

I tried doing this a while back. Some things I think I worried about at the time:

(1) disheartening people excessively by sending them scores that seem very low/brutal, especially if you use an unusual scoring methodology (2) causing yourself more time costs than it seems like at first, because (a) you find yourself needing to add caveats or manually hide some info to make it less disheartening to people, (b) people ask you follow-up questions (3) exposing yourself to some sort of unknown legal risk by saying something not-legally-defensible about the candidate or your decision-making.

(1) turned out to be pretty justified I think, e.g. at least one person expressing upset/dissatisfaction at being told this info. (2) definitely happened too, although maybe not all that many hours in the grand scheme of things (3) we didn't get sued but who knows how much we increased the risk by

MichaelStJules @ 2024-04-13T03:48 (+2) in response to Repugnance and replacement

Arguably we can only say a world state X is "better" than a world state Y iff both

  1. switching from X to Y is bad and
  2. switching from Y to X is good.

FWIW, people with person-affecting views would ask "better for whom?". Each set of people could have their own betterness order. Person-affecting views basically try to navigate these different possible betterness orders.

Kaspar Brandner @ 2024-04-13T08:17 (+1)

But it doesn't seem to matter whether X or Y (or neither) is actually current, everyone should be able to agree whether, e.g., "switching from X to Y is bad" is true or not. The switch choices always hypothetically assume that the first world (in this case X) is current, because that's where the potential choice to switch is made.

MichaelStJules @ 2024-04-13T03:26 (+2) in response to Repugnance and replacement

Thanks for the feedback! I've edited the post with some clarification.

I think standard decision theory (e.g. expected utility theory) is actually often framed as deciding between (or ranking) outcomes, or prospects more generally, not between actions. But actions have consequences, so we just need actions with the above outcomes as consequences. Maybe it's pressing buttons, pulling levers or deciding government policy. Either way, this doesn't seem very important, and I doubt most people will be confused about this point.

On the issue of switching between worlds, for the sake of the thought experiment, assume the current world has 1 million people, the same people common to all three outcomes, but it’s not yet decided whether the world will end up like A, A+ or Z. That's what you're deciding. Choosing between possible futures (or world histories, past, present and future, but ignoring the common past).

I don't intend for you to be able to switch from A+ or Z to A by killing people. A is defined so that the extra people never exist. It's the way things could turn out. Creating extra people and then killing them would be a different future.

We could make one of the three options the "default future", and then we have the option to pick one of the others. If we’re consequentialists, we (probably) shouldn’t care about which future is default.

Or, maybe I add an uncontroversially horrible future, a 4th option, the 1 million people being tortured forever, as the default future. So, this hopefully removes any default bias.

Kaspar Brandner @ 2024-04-13T08:10 (+1)

Okay, having an initial start world (call it S), that is assumed to be current, makes it possible to treat the other worlds (futures) as choices. So S has 1 million people, but how much utility points do they have each? Something like 10? Then A and A+ would be an improvement for them, and Z would be worse (for them).

I don't intend for you to be able to switch from A+ or Z to A by killing people. A is defined so that the extra people never exist. It's the way things could turn out. Creating extra people and then killing them would be a different future.

But if we can't switch worlds in the future that does seem like an unrealistic restriction? Future people have just as much control over their future as we have over ours. Not being able to switch worlds in the future (change the future of the future) would mean we couldn't, once we were at A+, switch from A+ to a more "fair" future (like Z). Since not-can implies not-ought, there would then be no basis in calling A+ unfair, insofar "unfair" means that we ought to switch to a more fair future.

The fairness consideration assumes utility can be redistributed, like money. Otherwise utility would presumably be some inherent property of the brains of people, and it wouldn't be unfair to anyone to not having been born with a different brain (assuming brains can't be altered).

Thomas Kwa @ 2024-04-12T23:25 (+38) in response to Dear EA, please be the reason people like me will actually see a better world. Help me make some small stride on extreme poverty where I live -- by the end of 2024.

[Warning: long comment] Thanks for the pushback. I think converting to lives is good in other cases, especially if it's (a) useful for judging effectiveness, and (b) not used as a misleading rhetorical device [1].

The basic point I want to make is that all interventions have to pencil out. When donating, we are trying to maximize the good we create, not decide which superficially sounds better between the different strategies "empower beneficiaries to invest in their communities' infrastructure" and "use RCTs to choose lifesaving interventions" [2]. Lives are at stake, and I don't think those lives are less important simply because it's harder to put names and faces to the ~60 lives that were saved from a 0.04% chance of reduction of malaria deaths from a malaria net. Of course this applies equally to the Wytham Abbey purchase or anything else. But to point (a), we actually can compare the welfare gain from 61 lives saved to the economic security produced by this project. GiveWell has weights for doubling of consumption, partly based on interviews from Africans [3]. With other projects, this might be intractable due to entirely different cause areas or different moral preferences e.g. longtermism.

Imagine that we have a cost-effectiveness analysis made by a person with knowledge of local conditions and local moral preferences, domain expertise in East African agricultural markets, and the quantitative expertise of GiveWell analysts. If it comes out that one intervention is 5 or 10 times better than the other, as is very common, we need a very compelling reason why some consideration was missed to justify funding the other one. Compare this to our currently almost complete state of ignorance as to the value of building this plant, and you see the value of numbers. We might not get a CEA this good, but we should get close as we have all the pieces.

As to point (b), I am largely pro making these comparisons in most cases just to remind people of the value of our resources. But I feel like the Wytham and HPMOR cases, depending on phrasing, could exploit peoples' tendency to think of projects that save lives in emotionally salient ways as better than projects that save lives via less direct methods. It will always sound bad to say that intervention A is funded rather than saving X lives, and we should generally not shut down discussion of A by creating indignation. This kind of misleading rhetoric is not at all my intention; we all understand that allowing a large enough number of farmers access to sorghum markets can produce more welfare than preventing 61 deaths from malaria. We have the choice between saving 61 of someones' sons and daughters, and allowing X extremely poor people to perhaps buy metal roofs, send their children to school, and generally have some chance of escaping a millennia-long poverty trap. We should think: "I really want to know how large X is".

[1] and maybe (c) not bad for your mental health?

[2] Unless you believe empowering people is inherently better regardless of the relative cost, which I strongly disagree with.

[3] This is important-- Westerners may be biased here because we place different values on life compared to doubling consumption. But these interviews were from Kenya and Ghana, so maybe Uganda's weights slightly differ.

NickLaing @ 2024-04-13T07:58 (+1)

@Thomas Kwa in my eyes this is a hugely insightful (perhaps even spectacular) response, thanks for taking the time to think about it and write it. Perhaps consder writing a full post with these kinds of insights about benefits of CEAs.

That is If you can stomach spending more time away from your real job making sure that we still exist in 50 years to even talk about GHD ;).

saulius @ 2024-04-12T12:01 (+5) in response to Yanni Kyriacos's Quick takes

There were many predictions about AI and AGI in the past (maybe mostly last century) that were very wrong. I think I read about it in Superintelligence. A quick Google search shows this article which probably talks about that.

yanni kyriacos @ 2024-04-13T07:28 (0)

Thanks!

titotal @ 2024-04-12T15:54 (+12) in response to Yanni Kyriacos's Quick takes

In the 90's and 2000's, many people such as Eric Drexler were extremely worried about nanotechnology and viewed it as an existential threat through the "gray goo" scenario. Yudkowsky predicted drexler style nanotech would occur by 2010, using very similar language to what he is currently saying about AGI. 

It turned out they were all being absurdly overoptimistic about how soon the technology would arrive, and the whole drexlerite nanotech project flamed out by the end of the 2000's and has pretty much not progressed since. I think a similar dynamic playing out with AGI is less likely, but still very plausible. 

yanni kyriacos @ 2024-04-13T07:28 (+1)

I hope you're right. Thanks for the example, it seems like a good one.

Yarrow Bouchard @ 2024-04-13T06:42 (+33) in response to To what extent & how did EA indirectly contribute to financial crime - and what can be done now? One attempt at a review

I do not follow everything that happens in the EA world. I don't use Twitter. I'm out of the loop. So, I don't know if any of the people named have responded to the claims made in this Time article from March 2023:

...[Will] MacAskill had long been aware of concerns around Bankman-Fried. He was personally cautioned about Bankman-Fried by at least three different people in a series of conversations in 2018 and 2019, according to interviews with four people familiar with those discussions and emails reviewed by TIME.

He wasn’t alone. Multiple EA leaders knew about the red flags surrounding Bankman-Fried by 2019, according to a TIME investigation based on contemporaneous documents and interviews with seven people familiar with the matter. Among the EA brain trust personally notified about Bankman-Fried’s questionable behavior and business ethics were Nick Beckstead, a moral philosopher who went on to lead Bankman-Fried’s philanthropic arm, the FTX Future Fund, and Holden Karnofsky, co-CEO of OpenPhilanthropy, a nonprofit organization that makes grants supporting EA causes. Some of the warnings were serious: sources say that MacAskill and Beckstead were repeatedly told that Bankman-Fried was untrustworthy, had inappropriate sexual relationships with subordinates, refused to implement standard business practices, and had been caught lying during his first months running Alameda, a crypto firm that was seeded by EA investors, staffed by EAs, and dedicating to making money that could be donated to EA causes.

 

Many of the emerging issues at Alameda that were reported to EA leaders beginning in 2018—including pervasive dishonesty, sloppy accounting, and rejection of corporate controls—presaged the scandal that unfolded at FTX four years later, according to sources who were granted anonymity to avoid professional retribution or becoming entangled in Bankman-Fried’s ongoing legal drama. “I was shocked at how much of what came out about FTX rhymed with the concerns we raised in the early days,” says one person who spoke directly with MacAskill and others about Bankman-Fried in 2018. “It was the same thing. All of the same problems.”

 

It’s not entirely clear how EA leaders reacted to the warnings. Sources familiar with the discussions told TIME that the concerns were downplayed, rationalized as typical startup squabbles, or dismissed as “he said-she said,” as two people put it. EA leaders declined or did not respond to multiple requests from TIME to explain their reaction to these warnings and what they did in response. But by the end of 2018, Bankman-Fried’s behavior was such an open secret that EA leaders were debating Bankman-Fried’s presence on the board of the Centre for Effective Altruism. In emails among senior EA leaders, which TIME reviewed, one person wrote that they had raised worries about Bankman-Fried’s trustworthiness directly with MacAskill, and that MacAskill had dismissed the concerns as “rumor.” In 2019, Bankman-Fried left CEA’s board.

MacAskill declined to answer a list of detailed questions from TIME for this story. “An independent investigation has been commissioned to look into these issues; I don’t want to front-run or undermine that process by discussing my own recollections publicly,” he wrote in an email. “I look forward to the results of the investigation and hope to be able to respond more fully after then.” Citing the same investigation, Beckstead also declined to answer detailed questions. Karnofsky did not respond to a list of questions from TIME. Through a lawyer, Bankman-Fried also declined to respond to a list of detailed written questions. The Centre for Effective Altruism (CEA) did not reply to multiple requests to explain why Bankman-Fried left the board in 2019. A spokesperson for Effective Ventures, the parent organization of CEA, cited the independent investigation, launched in Dec. 2022, and declined to comment while it was ongoing.

 

In the weeks leading up to that April 2018 confrontation with Bankman-Fried and in the months that followed, [Tara] Mac Aulay and others warned MacAskill, Beckstead and Karnofsky about her co-founder’s alleged duplicity and unscrupulous business ethics, according to four people with knowledge of those discussions. Mac Aulay specifically flagged her concerns about Bankman-Fried’s honesty and trustworthiness, his maneuvering to control 100% of the company despite promising otherwise, his pattern of unethical behavior, and his inappropriate relationships with subordinates, sources say.

[Naia] Bouscal recalled speaking to Mac Aulay immediately after one of Mac Aulay’s conversations with MacAskill in late 2018. “Will basically took Sam’s side,” said Bouscal, who recalls waiting with Mac Aulay in the Stockholm airport while she was on the phone. (Bouscal and Mac Aulay had once dated; though no longer romantically involved, they remain close friends.) “Will basically threatened Tara,” Bouscal recalls. “I remember my impression being that Will was taking a pretty hostile stance here and that he was just believing Sam’s side of the story, which made no sense to me.”

“He was treating it like a ‘he said-she said,’ even though every other long-time EA involved had left because of the same concerns,” Bouscal adds.

To me, this feels like the most important part of the whole story. 

EA Opportunity Board @ 2024-04-13T05:58 (+7) in response to Applications Open: Elevate Your Mental Resilience with Rethink Wellbeing's CBT Program

This seems like a great opportunity. It is now live on the EA Opportunity Board!

AnonymousTurtle @ 2024-04-12T13:52 (+1) in response to Mediocre EAs: career paths and how do they engage with EA?

I agree with some of this comment and disagree with other parts:

"people who initially set up Givewell, did the research and conivnced Dustin to donate his money did a truly amazing jop"

AFAIK Dustin would have donated a roughly similar amount anyway, at least after Gates levels of cost-effectiveness, so I don't think EA gets any credit for that (unless you include Dustin in EA, which you don't seem to do)

"The EA leadership has fucked up a bunch of stuff. Many 'elite EAs' were not part of the parts of EA that went well." I agree, but I think we're probably thinking of different parts of EA

"'Think for yourself about how to make the world better and then do it (assuming its not insane)' is probably both going to be better for you and better for the world" I agree with this, but I would be careful about where your thoughts are coming from

sapphire @ 2024-04-13T04:34 (+4)

There are a lot of possible answers to where thoughts come from and which thoughts are useful. One charitable thought is some Elite EAs tried to do things which were all of: hard, extremely costly if you fuck them up, they weren't able to achieve given the difficulty. I have definitely updated a lot toward trying things that are very crazy but at least obviously only hurt me (or people who follow my example, but those people made their own choice). Fail gracefullly. If you dont know how competent you are make sure not to mess things up for other people. There is a lot of 'theater' around this but most people don't internalize what it really means.

Kaspar Brandner @ 2024-04-12T20:49 (+5) in response to Repugnance and replacement

This is an interesting overview, as far as I can tell. Only it is hard to understand for someone who isn't already familiar with population ethics. For example:

Suppose you have to choose between the following 3 outcomes

It seems clear, e.g. from decision theory, that we can only ever "choose" between actions, never between outcomes. But A, A+, and Z are outcomes (world states), not actions. Apparently the only possible actions in this example would be intentionally moving between world states in time, i.e. "switching".

Then we would have three decision situations with three possible choice options each:

  • A is current
    • Switch from A to A+
    • Switch from A to Z
    • Stay in A (do nothing)
  • A+ is current
    • Switch from A+ to A
    • Switch from A+ to Z
    • Stay in A+ (do nothing)
  • Z is current
    • Switch from Z to A
    • Switch from Z to A+
    • Stay in Z (do nothing)

This changes things a lot. For example, it could both be bad to switch from current A to Z (because it would affect the current people [i.e. people in A] negatively) and to switch from current Z to A (because we would have to kill a lot of current people [i.e. people in Z] in order to "switch" to A). In that case it wouldn't make sense to say that A is simply better, worse, or equivalent to Z. What would we even mean by that?

Arguably we can only say a world state X is "better" than a world state Y iff both

  1. switching from X to Y is bad and
  2. switching from Y to X is good.

This is because "X is better than Y" implies "Y is worse than X", while the goodness of switching between outcomes doesn't necessarily have this property. It can't be represented with >, <, =, it's not an order relation.

MichaelStJules @ 2024-04-13T03:48 (+2)

Arguably we can only say a world state X is "better" than a world state Y iff both

  1. switching from X to Y is bad and
  2. switching from Y to X is good.

FWIW, people with person-affecting views would ask "better for whom?". Each set of people could have their own betterness order. Person-affecting views basically try to navigate these different possible betterness orders.

Kaspar Brandner @ 2024-04-12T20:49 (+5) in response to Repugnance and replacement

This is an interesting overview, as far as I can tell. Only it is hard to understand for someone who isn't already familiar with population ethics. For example:

Suppose you have to choose between the following 3 outcomes

It seems clear, e.g. from decision theory, that we can only ever "choose" between actions, never between outcomes. But A, A+, and Z are outcomes (world states), not actions. Apparently the only possible actions in this example would be intentionally moving between world states in time, i.e. "switching".

Then we would have three decision situations with three possible choice options each:

  • A is current
    • Switch from A to A+
    • Switch from A to Z
    • Stay in A (do nothing)
  • A+ is current
    • Switch from A+ to A
    • Switch from A+ to Z
    • Stay in A+ (do nothing)
  • Z is current
    • Switch from Z to A
    • Switch from Z to A+
    • Stay in Z (do nothing)

This changes things a lot. For example, it could both be bad to switch from current A to Z (because it would affect the current people [i.e. people in A] negatively) and to switch from current Z to A (because we would have to kill a lot of current people [i.e. people in Z] in order to "switch" to A). In that case it wouldn't make sense to say that A is simply better, worse, or equivalent to Z. What would we even mean by that?

Arguably we can only say a world state X is "better" than a world state Y iff both

  1. switching from X to Y is bad and
  2. switching from Y to X is good.

This is because "X is better than Y" implies "Y is worse than X", while the goodness of switching between outcomes doesn't necessarily have this property. It can't be represented with >, <, =, it's not an order relation.

MichaelStJules @ 2024-04-13T03:26 (+2)

Thanks for the feedback! I've edited the post with some clarification.

I think standard decision theory (e.g. expected utility theory) is actually often framed as deciding between (or ranking) outcomes, or prospects more generally, not between actions. But actions have consequences, so we just need actions with the above outcomes as consequences. Maybe it's pressing buttons, pulling levers or deciding government policy. Either way, this doesn't seem very important, and I doubt most people will be confused about this point.

On the issue of switching between worlds, for the sake of the thought experiment, assume the current world has 1 million people, the same people common to all three outcomes, but it’s not yet decided whether the world will end up like A, A+ or Z. That's what you're deciding. Choosing between possible futures (or world histories, past, present and future, but ignoring the common past).

I don't intend for you to be able to switch from A+ or Z to A by killing people. A is defined so that the extra people never exist. It's the way things could turn out. Creating extra people and then killing them would be a different future.

We could make one of the three options the "default future", and then we have the option to pick one of the others. If we’re consequentialists, we (probably) shouldn’t care about which future is default.

Or, maybe I add an uncontroversially horrible future, a 4th option, the 1 million people being tortured forever, as the default future. So, this hopefully removes any default bias.

Seth Ariel Green @ 2024-04-12T17:27 (+6) in response to Increasing risks of GCRs due to climate change

Animal welfare is an area where climate concerns and a canonical EA cause converge because factory farming is a major contributor to both problems. By that light, EAs are actually doing quite a lot to mitigate climate change, just under a different banner. 

Jason @ 2024-04-13T03:15 (+4)

At least often. I don't know if would be safe to assume that there will ~always be convergence -- for instance, switching to higher-welfare breeds that grow less efficiently and less rapidly might plausibly not be net positive from a climate perspective. Cultured meat might not be if its production were extremely energy intensive.

ABishop @ 2024-04-13T01:59 (+3) in response to ABishop's Quick takes

I would like to estimate how effective free hugs are. Can anyone help me?

Jason @ 2024-04-12T01:07 (+4) in response to Quick Update on Leaving the Board of EV

Not a crypto-focused platform (e.g., Joe's Crypto Podcast?) No particular reason to know or believe that the company (had / was going to) use something Person said as part of their marketing campaign? If negative to both, it doesn't affect my $5 Manifold bet.

Elizabeth @ 2024-04-13T01:32 (+2)

thanks, I appreciate all this info. 

Jack Malde @ 2024-04-13T00:24 (+21) in response to Jack Malde's Quick takes

Could it be more important to improve human values than to make sure AI is aligned?

Consider the following (which is almost definitely oversimplified):

 

ALIGNED AI

MISALIGNED AI

HUMANITY GOOD VALUES

UTOPIA

EXTINCTION

HUMANITY NEUTRAL VALUES

NEUTRAL WORLD

EXTINCTION

HUMANITY BAD VALUES

DYSTOPIA

EXTINCTION

For clarity, let’s assume dystopia is worse than extinction. This could be a scenario where factory farming expands to an incredibly large scale with the aid of AI, or a bad AI-powered regime takes over the world. Let's assume neutral world is equivalent to extinction.

The above shows that aligning AI can be good, bad, or neutral. The value of alignment exactly depends on humanity’s values. Improving humanity’s values however is always good. 

The only clear case where aligning AI beats improving humanity’s values is if there isn’t scope to improve our values further. An ambiguous case is whenever humanity has positive values in which case both improving values and aligning AI are good options and it isn’t immediately clear to me which wins.

The key takeaway here is that improving values is robustly good whereas aligning AI isn’t - alignment is bad if we have negative values. I would guess that we currently have pretty bad values given how we treat non-human animals and alignment is therefore arguably undesirable. In this simple model, improving values would become the overwhelmingly important mission. Or perhaps ensuring that powerful AI doesn't end up in the hands of bad actors becomes overwhelmingly important (again, rather than alignment).

This analysis doesn’t consider the moral value of AI itself. It also assumed that misaligned AI necessarily leads to extinction which may not be accurate (perhaps it can also lead to dystopian outcomes?).

I doubt this is a novel argument, but what do y’all think?



Comments on 2024-04-12

titotal @ 2024-04-12T08:59 (+22) in response to Dear EA, please be the reason people like me will actually see a better world. Help me make some small stride on extreme poverty where I live -- by the end of 2024.

I don't like that this "converting to lives" thing is being done on this kind of post and seemingly nowhere else? 

Like, if we applied it to the wytham abbey purchase (I don't know if the 15 mill figure is accurate but whatever), that's 2700 people EA let die in order to purchase a manor house. Or what about the fund that gave $28000 dollars to print out harry potter fanfiction and give it to math olympians? That's 6 dead children sacrificed for printouts of freely available fiction!

I hope you see why I don't like this type of rhetoric. 

Thomas Kwa @ 2024-04-12T23:25 (+38)

[Warning: long comment] Thanks for the pushback. I think converting to lives is good in other cases, especially if it's (a) useful for judging effectiveness, and (b) not used as a misleading rhetorical device [1].

The basic point I want to make is that all interventions have to pencil out. When donating, we are trying to maximize the good we create, not decide which superficially sounds better between the different strategies "empower beneficiaries to invest in their communities' infrastructure" and "use RCTs to choose lifesaving interventions" [2]. Lives are at stake, and I don't think those lives are less important simply because it's harder to put names and faces to the ~60 lives that were saved from a 0.04% chance of reduction of malaria deaths from a malaria net. Of course this applies equally to the Wytham Abbey purchase or anything else. But to point (a), we actually can compare the welfare gain from 61 lives saved to the economic security produced by this project. GiveWell has weights for doubling of consumption, partly based on interviews from Africans [3]. With other projects, this might be intractable due to entirely different cause areas or different moral preferences e.g. longtermism.

Imagine that we have a cost-effectiveness analysis made by a person with knowledge of local conditions and local moral preferences, domain expertise in East African agricultural markets, and the quantitative expertise of GiveWell analysts. If it comes out that one intervention is 5 or 10 times better than the other, as is very common, we need a very compelling reason why some consideration was missed to justify funding the other one. Compare this to our currently almost complete state of ignorance as to the value of building this plant, and you see the value of numbers. We might not get a CEA this good, but we should get close as we have all the pieces.

As to point (b), I am largely pro making these comparisons in most cases just to remind people of the value of our resources. But I feel like the Wytham and HPMOR cases, depending on phrasing, could exploit peoples' tendency to think of projects that save lives in emotionally salient ways as better than projects that save lives via less direct methods. It will always sound bad to say that intervention A is funded rather than saving X lives, and we should generally not shut down discussion of A by creating indignation. This kind of misleading rhetoric is not at all my intention; we all understand that allowing a large enough number of farmers access to sorghum markets can produce more welfare than preventing 61 deaths from malaria. We have the choice between saving 61 of someones' sons and daughters, and allowing X extremely poor people to perhaps buy metal roofs, send their children to school, and generally have some chance of escaping a millennia-long poverty trap. We should think: "I really want to know how large X is".

[1] and maybe (c) not bad for your mental health?

[2] Unless you believe empowering people is inherently better regardless of the relative cost, which I strongly disagree with.

[3] This is important-- Westerners may be biased here because we place different values on life compared to doubling consumption. But these interviews were from Kenya and Ghana, so maybe Uganda's weights slightly differ.

Jason @ 2024-04-12T15:16 (+2) in response to The Bunker Fallacy

- so I'm saying it should not only be pressure tested but be in continuous operation in order to flush out failure modes before a catastrophic scenario plays out, it needs to be providing value way before an extinction level event plays out. 

This seems to rely on an assumption that the failure modes that would exist in "normal mode" are related or correlated, to a fairly high degree, to the failure modes that could show up in "catastrophic mode." That's not obvious to me.
 

SimonKS @ 2024-04-12T23:19 (+3)

I'm not sure, if you look back at Biosphere 2 for example a large number of the failure modes were identified fairly early on. In my experience there are two things that cause unexpected failure modes, scale and duration. i.e. running something at a larger scale than was previously tested can often reveal unintuitive failure modes and running something for longer that previous can reveal failure modes. 

I get what your saying that running a service in a different environment to what it was tested in can cause unforseen issues, but I think with simulation and testing like they did for bejing airport or the kind of testing they do at SpaceX - we should be aiming to test these things to failure points. 

Jason @ 2024-04-12T15:46 (+2) in response to The Bunker Fallacy

One problem here is that a solution that operates continuously and provides "value way before an extinction level event plays out" is likely to be much more complex -- and have considerably more points of potential failure -- than a simpler solution.

Imagine I have a requirement that I be able to receive radio transmissions during an emergency. I could build a fairly simple emergency radio and pressure-test it. It doesn't provide me any value in ordinary times, so it just sits there on the shelf between pressure tests. It's cheap, so I could build several for redundancy. I could also build a smartphone with radio capacity -- this would be operated continuously and provide value in normal times. It is also much more complex and has many more failure modes than a simple radio. Many of those failure modes are common to both the ordinary-times use and the emergency-times use. But some are less so -- I might not be using the radio feature on my cellphone much at all in normal times because that particular function doesn't provide much if any pre-emergency value.

I think you're likely accepting a significantly higher level of failure modes for a better opportunity to detect and fix them. It's not clear to me if this is a good tradeoff or not. If you go back to the cellphone analogy, most things have to evolve and grow to continue providing enough value to justify their maintenance and operation. Maybe we could have a super-high-reliability smartphone if we locked in 2010 capabilities and worked on iterating the flaws out of it. But a 2010-level smartphone isn't going to get much use. And adding new features and capabilities adds new points of failure, so manufacturers are stuck playing a never-ending game of whack-a-mole.

I'm reminded of the troubles that the US government has had with getting cell phones up to the point that they can handle classified information. The poor user experience of cell phones approved for even secret information famously contributed to getting a former Secretary of State into some trouble. As of 2016, phones appeared to have only been approved to the secret level (not top secret, or ~"above top secret" like certain SCIs). In contrast, we've had landline devices cleared for all security levels since at least 1987 (I didn't check older devices). The security reliability risks are apparently a lot easier to manage with the simpler devices.

For similar reasons, the solution with pre-emergency value will likely be much more expensive to develop and operate. All critical components need to be hardened / highly reliable. The more you expect those components to do, the bigger / more complex they are going to be. If the power facility needs to power a small town, it will be much more expensive to harden than a solution for a bunker. If you rely on a different power solution for emergency situations and only harden that solution, then a critical system isn't going to be regularly operated. It's probably going to be considerably more expensive to run and maintain a hardened system than a standard one even in normal times -- so normal-time operation is likely to also require some significant subsidies. As a result, you're going to have to put a lot of eggs in a single basket for cost reasons alone.

SimonKS @ 2024-04-12T23:05 (+3)


If I take the power plant for example, let's imagine you want to build a hardened geothermal power plant in some kind of significantly sheltered  (maybe underground) environment as part of the citadelle concept. So you want a 40mwh power plant to support @30k people in the refuge. Well in normal times - that power goes straight into the grid and earns back revenue. Yes, it's probably cheaper to build a field of solar, but not that much cheaper, Dinorwig Power Station was built in the 70's in a mountain and that's still cost effective today even though it's slightly harder to build. 
This is where the paradox of scale economics comes into play, the more you do something at scale, actually the lower it costs and the more simple it becomes. It's like using rare minerals - the more you use, the greater the demand, the more get's found in new reserves.

I do agree that you have a valid point regarding the mobile phone vs. simple radio analogy. I wonder if there's principle that could be adopted to beat this complexity problem though - what your essentially saying is the goods produced by the modern world value chain are ill suited to a disaster scenario due to their dependency on components that could be impossible to source in a different context. Maybe a principle along the lines that any goods used in the refuge must be able to be 100% fabricated in the refuge by materials available to the refuge. The Mormons have been doing something fairly similar for decades and it seems to work quite well for them.

Vasco Grilo @ 2024-04-11T18:13 (+7) in response to What Does a Marginal Grant at LTFF Look Like? Funding Priorities and Grantmaking Thresholds at the Long-Term Future Fund

Caleb and Linch randomly selected grants from each group.

I think your procedure to select the grants was great. However, would it become even better by making the probability of each grant being selected proportional to its size? In theory, donors should care about the impact per dollar (not impact per grant), which justifies weighting by grant size. This may matter because there is significant variation in grant size. The 5th and 95th percentile amount granted by LTFF are 2.00 k$ and 169 k$, so, specially if one is picking just a few grants as you did (as opposed to dozens of grants), there is a risk of picking unrepresentatively small grants.

Linch @ 2024-04-12T21:19 (+4)

Thank you! This is a good point; your analysis makes a lot of sense to me.

Kaspar Brandner @ 2024-04-12T20:49 (+5) in response to Repugnance and replacement

This is an interesting overview, as far as I can tell. Only it is hard to understand for someone who isn't already familiar with population ethics. For example:

Suppose you have to choose between the following 3 outcomes

It seems clear, e.g. from decision theory, that we can only ever "choose" between actions, never between outcomes. But A, A+, and Z are outcomes (world states), not actions. Apparently the only possible actions in this example would be intentionally moving between world states in time, i.e. "switching".

Then we would have three decision situations with three possible choice options each:

  • A is current
    • Switch from A to A+
    • Switch from A to Z
    • Stay in A (do nothing)
  • A+ is current
    • Switch from A+ to A
    • Switch from A+ to Z
    • Stay in A+ (do nothing)
  • Z is current
    • Switch from Z to A
    • Switch from Z to A+
    • Stay in Z (do nothing)

This changes things a lot. For example, it could both be bad to switch from current A to Z (because it would affect the current people [i.e. people in A] negatively) and to switch from current Z to A (because we would have to kill a lot of current people [i.e. people in Z] in order to "switch" to A). In that case it wouldn't make sense to say that A is simply better, worse, or equivalent to Z. What would we even mean by that?

Arguably we can only say a world state X is "better" than a world state Y iff both

  1. switching from X to Y is bad and
  2. switching from Y to X is good.

This is because "X is better than Y" implies "Y is worse than X", while the goodness of switching between outcomes doesn't necessarily have this property. It can't be represented with >, <, =, it's not an order relation.

Yarrow Bouchard @ 2024-04-12T20:03 (–5) in response to Guilt at EA

What if consequentialism has negative consequences?

NickLaing @ 2024-04-12T18:28 (+2) in response to Dear EA, please be the reason people like me will actually see a better world. Help me make some small stride on extreme poverty where I live -- by the end of 2024.

Perhaps then in this case you just don't agree or disagree when it's 50/50? Looking at it now I also don't think my little CEA is plausible, I do think it perhaps got taken a bit too seriously though :D!

Jason @ 2024-04-12T20:00 (+2)

Yeah, I didn't vote either way, which is fine. I'm just confused about how to interpret the votes of those who did! Did they agree/disagree on both parts, or vote based on which part they thought was primary?

Jason @ 2024-04-12T15:12 (+9) in response to Dear EA, please be the reason people like me will actually see a better world. Help me make some small stride on extreme poverty where I live -- by the end of 2024.

Grassroots projects like this seem like a natural extension of this, where a community as a whole decides where they need resources in order to uplift everyone. 

This makes sense, but I don't think it gets us very far on the question of whether to fund this particular project. 

There are many grassroots projects in developing countries, and it is often difficult for a Westerner to evaluate the effectiveness of those projects (at least where their theory of impact is more complex than bednets --> less malaria). It's even difficult for us to assess the relative extent of informed public support for a grassroots project from afar. Those issues are less formidable with GiveDirectly; if the main theory of impact is to benefit Person X, we have both theoretical reasons and experiential evidence that giving Person X money is a good way to accomplish that. That doesn't necessarily scale well to supporting grassroots work.

That being said, I can see a decent theoretical argument for (as it were) GiveDirectly for Communities -- give a community a certain amount of money, and let the community decide what needs that money should go to. I can see a number of practical problems with that, though. I think your average Westerner is going to be considerably worse at evaluating projects valued at ~50 times GDP per capita than in making their own consumption/investment decisions, and I suspect that may be true in many places. The quality of many decisions made by various democratic political systems also gives me pause. So I think there would need to be evaluation and selection of proposals rather than the total deference of the GiveDirectly approach.

Ideally, we would have evaluation organizations that were more local to the populations that were being served, rather than having the big GH/D evaluator be in the United States. That should give us evaluators with more local knowledge, and (to be honest) those with a cost structure and business processes that would make evaluating five-to-six figure projects more feasible.

David T @ 2024-04-12T19:31 (+6)

I think the combination of bottom-up approach of local communities proposing their own improvements with EA-style rigorous quantitative evaluation (which, like you say would be best undertaken by evaluators based in similar LMICs) is potentially really powerful, and I'm not sure the extent to which it's already been tried in mainstream aid. 

Or possibly even better from a funding perspective, turn that round and have an organization that helps local social entrepreneurs secure institutional funding for their projects (a little bit like Charity Entrepreneurship). Existing aid spend is enormous, but I don't think it's easy for people like Antony to access.

I also think there's the potential for interesting online interaction between the different local social entrepreneurs (especially those who have already part-completed projects with stories to share), putative future donors and other generally interested Westerners who might bring other perspectives to the table.  I'm not sure to what extent and where that happens at the moment.

John Salter @ 2024-04-11T20:38 (+39) in response to Mediocre EAs: career paths and how do they engage with EA?

I think there's a ton of obvious things that people neglect because they're not glamorous enough:

1. Unofficially beta-test new EA stuff e.g. if someone announces something new, use it and give helpful feedback regularly
2. Volunteer to do boring stuff for impactful organisations e.g. admin
3. Deeply fact-check popular EA forum posts
4. Be a good friend to people doing things you think are awesome
5. Investigate EA aligned charities on the ground, check that they are being honest in their reporting
6. Openly criticise grifters who people fear to speak out against for fear of reprisal 
7.  Stay up-to-date on the needs of different people and orgs, and connect people who need connecting

In generally, looking for the most anxiety provoking, boring, and lowest social status work is a good way of finding impactful opportunities. 

Elizabeth @ 2024-04-12T19:16 (+19)

I'm so glad someone wrote this. EA has real gaps caused by selecting a narrow group of people and demanding they only work on the most important things. Tasks that are unglamorous or just don't make sense as a job duty accumulate like housekeeping when you're sick. People who have the slack to do these without displacing a higher priority task have some highly leveraged actions available to them.

Jason @ 2024-04-12T16:49 (+3) in response to Understanding FTX's crimes

That's fair.

Broader concerns about sovereignty and the integrity of the extradition system likely played a role. Although it's understandable why US prosecutors asked for provisional arrest and extradition despite not having obtained an indictment for campaign-finance violations yet, it's also understandable to me why the Bahamas wants to signal its expectation that the US comply with the treaty as written -- especially where the argument that the first extradition was good enough for these charges is legally weak.

Note that the US could have asked for permission to proceed on other charges -- see Article 14(b) of the treaty -- but I think that may have required going back through the Bahamas legal system again, and probably delaying the trial. Hence, the prosecution severed off those charges, and then decided not to proceed after getting its conviction on others. I don't know Bahamas campaign or extradition law, but there would also be difficulties if the offense was deemed one of a political character (Article 3(1)(a)) or was one that was not a criminal offense punishable by more than a year in prison under Bahamas law (Article 2(1)).

Also, I don't think "completely uninvestigated" is a correct characterization -- they were investigated enough to be presented to a grand jury, which indicted SBF for campaign-finance violations. Federal prosecutors do not generally indict without a pretty good investigation first, especially in high-profile cases. I think we have a pretty decent idea of what he did (see pp. 18-22 of the prosecution's sentencing memo). Moreover, Salame and Singh -- who don't have extradition-related issues -- pled guilty to campaign-finance violations.

Habryka @ 2024-04-12T19:04 (+2)

Also, I don't think "completely uninvestigated" is a correct characterization -- they were investigated enough to be presented to a grand jury, which indicted SBF for campaign-finance violations. Federal prosecutors do not generally indict without a pretty good investigation first, especially in high-profile cases. I think we have a pretty decent idea of what he did (see pp. 18-22 of the prosecution's sentencing memo). Moreover, Salame and Singh -- who don't have extradition-related issues -- pled guilty to campaign-finance violations.

Oh, that is interesting and an update for me. I had interpreted the relevant section of the memo as more of a "we didn't really investigate it, but here is what we have", but you know this stuff much better than I do.

titotal @ 2024-04-11T16:18 (+8) in response to Summary: Mistakes in the Moral Mathematics of Existential Risk (David Thorstad)

Instead it's saying that it may be more natural to have the object-level conversations about transitions rather than about risk-per-century.

 

Hmm, I definitely think there's an object level disagreement about the structure of risk here. 

Take the invention of nuclear weapons for example. This was certainly a "transition" in society relevant to existential risk. But it doesn't make sense to me to analogise it to a risk in putting up a sail. Instead, nuclear weapons are just now a permanent risk to humanity, which goes up or down depending on geopolitical strategy. 

I don't see why future developments wouldn't work the same way. It seems that since early humanity the state risk has only been increasing further and further as technology develops. I know there are arguments for why it could suddenly drop, but I agree with the linked Thorstadt analysis that this seems unlikely. 

Vasco Grilo @ 2024-04-12T19:01 (+6)

Nice discussion, Owen and titotal!

But it doesn't make sense to me to analogise it to a risk in putting up a sail.

I think this depends on the timeframe. Over a longer one, looking into the estimated destroyable area by nuclear weapons, nuclear risk looks like a transition risk (see graph below). In addition, I think the nuclear extinction risk has decreased even more than the destroyable area, since I believe greater wealth has made society more resilient to the effects of nuclear war and nuclear winter. For reference, I estimated the current annual nuclear extinction risk is 5.93*10^-12.

AnonymousTurtle @ 2024-04-09T16:57 (+13) in response to Sam Harris and William MacAskill on SBF & EA

Like with Wytham Abbey, I'm really surprised by people in this thread confusing investments with donations.

If SBF had invested some billions in Twitter, the money wouldn't be burned, see e.g. what happened with Anthropic.

From his (and most people's) perspective, SBF was running FTX with ~1% the employees of comparable platforms, so it seemed plausible he could buy Twitter, cut 90% of the workforce like Musk did, and make money while at the same time steering it to be more scout-mindset and truth-seeking oriented.

Stuart Buck @ 2024-04-12T18:43 (+5)

Is the consensus currently that the investment in Twitter has paid off or is ever likely to do so? 

Stephen Clare @ 2024-04-12T14:47 (+20) in response to Stephen Clare's Quick takes

Many organizations I respect are very risk-averse when hiring, and for good reasons. Making a bad hiring decision is extremely costly, as it means running another hiring round, paying for work that isn't useful, and diverting organisational time and resources towards trouble-shooting and away from other projects. This leads many organisations to scale very slowly.

However, there may be an imbalance between false positives (bad hires) and false negatives (passing over great candidates). In hiring as in many other fields, reducing false positives often means raising false negatives. Many successful people have stories of being passed over early in their careers. The costs of a bad hire are obvious, while the costs of passing over a great hire are counterfactual and never observed.

I wonder  whether, in my past hiring decisions, I've properly balanced the risk of rejecting a potentially great hire against the risk of making a bad hire. One reason to think we may be too risk-averse, in addition to the salience of the costs, is that the benefits of a great hire could grow to be very large, while the costs of a bad hire are somewhat bounded, as they can eventually be let go.

Sarah Levin @ 2024-04-12T18:38 (+3)

the costs of a bad hire are somewhat bounded, as they can eventually be let go.

This depends a lot on what "eventually" means, specifically. If a bad hire means they stick around for years—or even decades, as happened in the organization of one of my close relatives—then the downside risk is huge

OTOH my employer is able to fire underperforming people after two or three months, which means we can take chances on people who show potential even if there are some yellow flags. This has paid off enormously, e.g. one of our best people had a history of getting into disruptive arguments in nonprofessional contexts, but we had reason to think this wouldn't be an issue at our place... and we were right, as it turned out, but if we lacked the ability to fire relatively quickly, then I wouldn't have rolled those dice. 

The best advice I've heard for threading this needle is "Hire fast, fire fast". But firing people is the most unpleasant thing a leader will ever have to do, so a lot of people do it less than they should.

Jason @ 2024-04-12T14:23 (+4) in response to Dear EA, please be the reason people like me will actually see a better world. Help me make some small stride on extreme poverty where I live -- by the end of 2024.

This comment shows the challenge of the agreevote/disagreevote system:

  •  I agree with the direction toward showing more cost-effectiveness analyses in other fields versus reducing their importance in global health/development. 
  • I do not think the fanfic CEA is plausible, even at the 2-minute level, for some of the reasons identified by @titotal. That being said, the people funding fanfic distribution were probably interested in "people drawn to AI safety / x-risk mitigation" as their outcome variable / theory of impact, not donations made or individual lives saved. So the BOTEC is simultaneous too kind to and too demanding of that project.
NickLaing @ 2024-04-12T18:28 (+2)

Perhaps then in this case you just don't agree or disagree when it's 50/50? Looking at it now I also don't think my little CEA is plausible, I do think it perhaps got taken a bit too seriously though :D!

Amber Dawn @ 2024-04-11T09:45 (+41) in response to Mediocre EAs: career paths and how do they engage with EA?

I’ve been thinking about this quite a bit recently. It’s not that I see myself as a “mediocre” EA, and in fact I work with EAs, so I am engaging with the community through my work. But I feel like a lot of the attitudes around career planning in EA sort of assume that you are formidable within a particular, rather narrow mould. You talk about mediocre EAs, but I’d also extend this to people who have strong skills and expertise that’s not obviously convertable into ‘working in the main EA cause areas’.

And the thing is, this kind of makes sense: like, if you’re a hardcore EA, it makes sense to give lots of attention and resources to people who can be super successful in the main EA cause areas, and comparatively neglect people who can’t. Inasmuch as the community’s primary aim is to do more good according to a specific set of assumptions and values, and not to be a fuzzy warm inclusive space, it makes sense that there aren’t a lot of resources for people who are less able to have an impact. But it's kind of annoying if you're one of those people! 

Or like: most EA jobs are crazy competitive nowadays. And from the point of view of "EA" (as an ideology), that's fine; impactful jobs should have large hiring pools of talented committed people. But from the point of view of people in the hiring pool, who are constantly applying to and getting rejected from EA jobs - or competitive non-EA jobs - because they've been persuaded these are the only jobs worth having, it kinda sucks.

There’s this well-known post ‘don’t be bycatch’; I currently suspect that EA structurally generates bycatch. By ‘structurally’ I mean ‘the behaviour of powerful actors in EA is kinda reasonable, but also it predictably creates situations where lots of altruistic, committed people get drawn into the community but can’t succeed within the paradigms the community has defined as success’. 

David T @ 2024-04-12T18:18 (+3)

I’d also extend this to people who have strong skills and expertise that’s not obviously convertable into ‘working in the main EA cause areas’.

I think this is a key part. "Main EA cause areas" does centre a lot on a small minority of people with very specific technical skills and the academic track record to participate in (especially if you're taking 80k Hours for guidance on that front) 

But people can have a lot of impact in areas like fundraising with a completely different skillset (one that is less likely to benefit from a quantitative degree from an elite university) or earn well enough to give a lot without having any skills in research report writing, epidemiology or computer science.

And if your background isn't one that the "do cutting edge research or make lots of money to give away" advice is tailored to at all, there are a lot of organizations doing a lot of effective good that really really, really need people with the right motivations allied to less niche skillsets. So I don't think people should feel they're not a 'success' if they end up doing GHD work rather than paying for it, and if their organization isn't particularly adjacent to EA they might have more scope to positively influence its impactfulness.

Also, people shouldn't label themselves mediocre :) 

Stephen Clare @ 2024-04-12T14:47 (+20) in response to Stephen Clare's Quick takes

Many organizations I respect are very risk-averse when hiring, and for good reasons. Making a bad hiring decision is extremely costly, as it means running another hiring round, paying for work that isn't useful, and diverting organisational time and resources towards trouble-shooting and away from other projects. This leads many organisations to scale very slowly.

However, there may be an imbalance between false positives (bad hires) and false negatives (passing over great candidates). In hiring as in many other fields, reducing false positives often means raising false negatives. Many successful people have stories of being passed over early in their careers. The costs of a bad hire are obvious, while the costs of passing over a great hire are counterfactual and never observed.

I wonder  whether, in my past hiring decisions, I've properly balanced the risk of rejecting a potentially great hire against the risk of making a bad hire. One reason to think we may be too risk-averse, in addition to the salience of the costs, is that the benefits of a great hire could grow to be very large, while the costs of a bad hire are somewhat bounded, as they can eventually be let go.

Brad West @ 2024-04-12T17:59 (+3)

One way of thinking about the role is how varying degrees of competence correspond with outcomes. 

You could imagine a lot of roles have more of a satisficer quality- if a sufficient degree of competence is met, the vast majority of the value possible from that role is met. Higher degrees of excellence would have only marginal value increases; insufficient competence could reduce value dramatically. In such a situation, risk-aversion makes a ton of sense: the potential benefit of getting grand slam placements is very small in relation to the harm caused by an incompetent hire.

On the other hand, you might have roles where the value scales very well with incredible placements. In these situations, finding ways to test possible fit may be very worth it even if there is a risk of wasting resources on bad hires.

Robi Rahman @ 2024-04-12T17:19 (+2) in response to Dear EA, please be the reason people like me will actually see a better world. Help me make some small stride on extreme poverty where I live -- by the end of 2024.

I agree. Mods, is there a reason why I can't downvote the community tag on this post?

Lorenzo Buonanno @ 2024-04-12T17:43 (+7)

It was marked community by the author, we can remove the tag if he wants us to. I agree it's more a request for funding than something about the EA community.

AFAIK users can't remove the community tag from posts because of worries of misuse

sawyer @ 2024-04-12T17:43 (+2) in response to sawyer's Quick takes

Within EA, work on x-risk is very siloed by type of threat: There are the AI people, the bio people, etc. Is this bad, or good?

Which of these is the correct analogy?

  1. "Biology is to science as AI safety is to x-risk," or 
  2. "Immunology is to biology as AI safety is to x-risk"

EAs seem to implicitly think analogy 1 is correct: some interdisciplinary work is nice (biophysics) but most biologists can just be biologists (i.e. most AI x-risk people can just do AI).

The "existential risk studies" model (popular with CSER, SERI, and lots of other non-EA academics) seems to think that analogy 2 is correct, and that interdisciplinary work is totally critical—immunologists alone cannot achieve a useful understanding of the entire system they're trying to study, and they need to exchange ideas with other subfields of medicine/biology in order to have an impact, i.e. AI x-risk workers are missing critical pieces of the puzzle when they neglect broader x-risk studies.

Seth Ariel Green @ 2024-04-12T17:27 (+6) in response to Increasing risks of GCRs due to climate change

Animal welfare is an area where climate concerns and a canonical EA cause converge because factory farming is a major contributor to both problems. By that light, EAs are actually doing quite a lot to mitigate climate change, just under a different banner. 

Nathan Young @ 2024-04-12T10:04 (+30) in response to Dear EA, please be the reason people like me will actually see a better world. Help me make some small stride on extreme poverty where I live -- by the end of 2024.

I sense this post shouldn't be a community post. I know it's written to the EA community, but it's discussing a specific project. Feels like it shouldn't be relegated to the community section because of style.

Robi Rahman @ 2024-04-12T17:19 (+2)

I agree. Mods, is there a reason why I can't downvote the community tag on this post?

Habryka @ 2024-04-12T15:32 (+6) in response to Understanding FTX's crimes

I... think I will continue describing this as "weird" though it makes sense that as a lawyer it's not that weird to you.

It feels very off to have a bunch of crimes uninvestigated because someone fled the country and then was extradited, and I am pretty confused why the Bahamas cooperated with Sam here (my guess is that it's a case of political corruption, though I can also imagine other reasons). It's not like the Bahamas had anything obvious to gain from Sam not being convicted of the campaign finance violations.

Jason @ 2024-04-12T16:49 (+3)

That's fair.

Broader concerns about sovereignty and the integrity of the extradition system likely played a role. Although it's understandable why US prosecutors asked for provisional arrest and extradition despite not having obtained an indictment for campaign-finance violations yet, it's also understandable to me why the Bahamas wants to signal its expectation that the US comply with the treaty as written -- especially where the argument that the first extradition was good enough for these charges is legally weak.

Note that the US could have asked for permission to proceed on other charges -- see Article 14(b) of the treaty -- but I think that may have required going back through the Bahamas legal system again, and probably delaying the trial. Hence, the prosecution severed off those charges, and then decided not to proceed after getting its conviction on others. I don't know Bahamas campaign or extradition law, but there would also be difficulties if the offense was deemed one of a political character (Article 3(1)(a)) or was one that was not a criminal offense punishable by more than a year in prison under Bahamas law (Article 2(1)).

Also, I don't think "completely uninvestigated" is a correct characterization -- they were investigated enough to be presented to a grand jury, which indicted SBF for campaign-finance violations. Federal prosecutors do not generally indict without a pretty good investigation first, especially in high-profile cases. I think we have a pretty decent idea of what he did (see pp. 18-22 of the prosecution's sentencing memo). Moreover, Salame and Singh -- who don't have extradition-related issues -- pled guilty to campaign-finance violations.

AnonymousTurtle @ 2024-04-12T13:31 (+9) in response to Mediocre EAs: career paths and how do they engage with EA?

I agree with the examples, but for the record I think it's very misleading to claim Imma is a "mediocre EA".

If I understand correctly, she moved to a different country so she could donate more, which enables her to donate a lot with her "normal" tech job (much more than the median EA). Before that, she helped kickstart the now booming Dutch EA community, and helped with "Doing Good Better" (she's in the credits)

My understanding is that she's not giving millions every year or founding charities, but she still did much more than a "median EA" would be able to

James Herbert @ 2024-04-12T16:49 (+2)

Big +1, thanks for bigging up Imma (and the Dutch EA community!)

Jason @ 2024-04-12T13:40 (+6) in response to Mediocre EAs: career paths and how do they engage with EA?

I have mixed feelings about this answer.

On the one hand, I think there's a lot of wisdom in it. The idea that people should be either all-in on EA or out isn't helpful, and isn't effective either. If you think about other social movements that have made a difference (for good and/or ill) on the world, they generally offer opportunities at a number of different commitment and ability points. Take most religions for example -- they do not encourage everyone to become a member of the clergy, to pursue advanced theological studies, and so on.

On the other hand, the reference to a "hobby" and some of the specific suggestions trend a little more toward EA-as-spectator-sport than I'd like. I certainly approve of your bouldering hobby, but it sounds like you (and maybe your friend group) are internalizing most of the benefits thereof. I think we can (and should) aim a little higher than that with EA, even EA for the masses.

To use a loose religion metaphor, we might have something vaguely like:

  • For the relatively few -- do a EA job / make EA your main professional focus
  • For a larger minority -- make EA a significant part of your life, take the GWWC pledge, be active on the Forum, go to conferences if that adds value
  • For the majority -- donate 1%, consume at least a few hours of content a year, fast on one day a year and give the money you would have spent on food to the global poor
James Herbert @ 2024-04-12T16:41 (+3)

Yeah I think I agree with you, and I think considering those three levels to be appropriate is consistent with the statement 'I think most people should think of EA as little more than a hobby'. 

I feel like pushing the 'treat it like a hobby' thing is good at the mo because I see a lot of people in the EA community feeling they ought to do more, and then they feel bad when it doesn't work out, and that sucks. I worry they begin to tie their self-worth to whether they are a 'good EA' or not. I want to be like, hey, take it easy, you're doing a good job - y'know?[1] 

I'm reminded of when I spoke to a therapist at my uni because I was struggling with anxiety and perfectionism. I wanted to get the best grades and do great things, but in pursuing that goal so relentlessly it was a) undermining my ability to study well and b) making me unhappy. He reminded me that being a student was only a small part of my life. I was also a friend, a partner, a citizen, a son, etc., and these parts of my life were all equally valuable (if not more so).   

I might take a different approach if I was talking with a member of the general population. Rutger Bregman's School of Moral Ambition does that. He's very much, 'Yo, you've got all this potential, you should be more morally ambitious'. But then again, maybe I wouldn't because the most thorough definition of EA I know of is non-normative, and I'm glad this is the case. 

  1. ^

    I thought 80k's episode on altruistic perfectionism was great and we could do with more of it.

James Herbert @ 2024-04-12T16:21 (+1) in response to Mediocre EAs: career paths and how do they engage with EA?

Oh but I did put 'donate some money' in my 'hobby' list - or am I misunderstanding you?

RedStateBlueState @ 2024-04-12T16:33 (+2)

Yes, I just would have emphasized it more. I sort of read it as “yeah this is something you might do if you’re really interested”, while I would more say “this is something you should really probably do”

jackva @ 2024-04-12T16:31 (+7) in response to Increasing risks of GCRs due to climate change

My sense is that it is not a big priority.

However, I would also caution against the view that expected climate risk has increased over the past years.

Even if impacts are faster than predicted, most GCR-climate risk does probably not come from developments in the 2020s, but on emissions paths over this century.

And the big story there is that the expected cumulative emissions have much decreased (see e.g. here).

As far as I know no one has done the math on this, but I would expect that the decrease in likelihood of high warming futures dominates somewhat higher-than-anticipated warming at lower level of emissions.

Joseph Pusey @ 2024-04-11T21:01 (+5) in response to Mediocre EAs: career paths and how do they engage with EA?

I think this is a lovely framing and one I'm going to apply more widely, both to myself and other people. I think the EA community, given our obvious difficulty productively deploying large numbers of talented people, ought to hold it in higher esteem also. Thanks, James.

James Herbert @ 2024-04-12T16:23 (0)

Thanks for the kind words! Glad you found my framing helpful :)

RedStateBlueState @ 2024-04-12T14:19 (+3) in response to Mediocre EAs: career paths and how do they engage with EA?

Mostly agreed, but I do think that donating some money, if you are able, is a big part of being in EA. And again this doesn’t mean reorienting your entire career to become a quant and maximize your donation potential.

James Herbert @ 2024-04-12T16:21 (+1)

Oh but I did put 'donate some money' in my 'hobby' list - or am I misunderstanding you?

Joseph Lemien @ 2024-04-12T14:37 (+4) in response to Dear EA, please be the reason people like me will actually see a better world. Help me make some small stride on extreme poverty where I live -- by the end of 2024.

I just want to chime in to say how lovely it is to see a disagreement on the internet that doesn't degrade. It was very nice to read each of you describe what you believe to be true, cite sources, explain reasoning without exaggerations or ad hominems, consider context and hypothesize about possibilities, and move a step closer to 'truth.' Bravo.

Anthony Kalulu, a rural farmer in eastern Uganda. @ 2024-04-12T16:20 (+1)

Thanks too Joseph, for appreciating this. Anthony.

taoburga @ 2024-04-11T21:04 (+4) in response to Reasons for optimism about measuring malevolence to tackle x- and s-risks

Thanks Clare! Your comment was super informative and thorough. 

One thing that I would lightly dispute is that 360 feedback is easily gameable. I (anecdotally) feel like people with malevolent traits (“psychopaths” here) often have trouble remaining “undiscovered” and so have to constantly move or change social circles. 

Of course, almost by definition I wouldn’t know any psychopaths that are still undiscovered. But 360 feedback could help discover the “discoverable” subgroup, since the test is not easily gameable by them.
Any thoughts?

Clare_Diane @ 2024-04-12T16:03 (+2)

Thank you, Tao - I’m glad you found it informative! 
 

And thank you for that - I think that’s a great point. I was probably a bit too harsh in dismissing 360 degree reviews: at least in some circumstances, I agree with you - it seems like they’d be hard to game. 
 

Having said that, I think it would mostly depend on the level of power held by the person being subjected to a 360 review. The more power the person had, the more I’d be concerned about the process failing to detect malevolent traits. If respondents thought the person of interest was capable of (1) inferring who negative feedback came from and (2) exacting retribution (for example), then I imagine that this perception could have a chilling effect on the completeness and frankness of feedback. 
 

For people who aren’t already in a position of power, I agree that 360 degree reviews would probably be less gameable. But in those cases, I’d still be somewhat concerned if they had high levels of narcissistic charm (since I’d expect those people to have especially positive feedback from their “fervent followers,” such that - even in the presence of negative feedback from some people - their high levels of malevolent traits may be more likely to be missed, especially if the people reviewing the feedback were not educated about the potential polarizing effects of people with high levels of narcissism).
 

If 360 reviews were done in ways that guaranteed (to the fullest extent possible) that the person of interest could not pinpoint who negative feedback came from, and if the results were evaluated by people who had been educated about the different ways in which malevolent traits can present, I would be more optimistic about their utility. And I could imagine that information from such carefully conducted and interpreted reviews could be usefully combined with other (ideally more objective) sources of information. In hindsight, my comment didn’t really address these ways in which 360 reviews might be useful in conjunction with other assessments, so thank you so much for catching this oversight!
 

I’d always be interested in discussing any of these points further.
 

Thank you again for your feedback and thoughts! 

titotal @ 2024-04-12T13:56 (+23) in response to Dear EA, please be the reason people like me will actually see a better world. Help me make some small stride on extreme poverty where I live -- by the end of 2024.

I'm fine with CEA's, my problem is that this seems to have been trotted out selectively in order to dismiss Anthony's proposal in particular, even though EA discusses and sometimes funds proposals that make the supposed "16 extra deaths" look like peanuts by comparison. 

The Wytham abbey project has been sold, so we know it's overall impact was to throw something like a million pounds down the drain (when you factor in stamp duty, etc). I think it's deeply unfair to frame Anthony's proposal as possibly letting 16 people die, while not doing the same for Wytham, which (in this framing) definitively let 180 people die. 

Also, the cost effectiveness analysis hasn't even been done yet! I find it kind of suspect that this is getting such a hostile response when EA insiders propose ineffective projects all the time with much less pushback. There are also differing factors here worth considering, like helping EA build links with grassroots orgs, indirectly spreading EA ideas to organisers in the third world, etc. EA spends plenty of money on "community building", would this not count? 

The HPMOR thing is a side note, but I vehemently disagree with your analysis, and the initial grant, because the counterfactual in this case is not doing nothing, it's sending them a link to the website where HPMOR is hosted for free for everybody, which costs nothing. Plus HPMOR only tangentially advocates for EA causes anyway! A huge number of people have read HPMOR, and only a small proportion have gone on to become EA members. Your numbers are absurdly overoptimistic. 

NickLaing @ 2024-04-12T16:03 (+2)

I also probably "disagree" with my analysis

Disclaimer I don't know much about the HPMOR thing - for example I didn't know it only tangentially plugged EA. I was just giving a 2 minute example of the kind of analysis you might do (obviously with better info then I had), and that it is possible to do that CEA. I wasn't trying to justify the grant at all my apologies if it came across that way!

Also I don't think this post is getting that hostile a response?

Leonora_Camner @ 2024-04-12T16:02 (+2) in response to A Gentle Introduction to Risk Frameworks Beyond Forecasting

Thank you for this very informative post on an important issue. I'm going to take some time to read through this a few times and absorb all of the information. As I wrote in my other post, I'm thinking a lot about the risk of GCRs due to climate change, so I'm looking forward enhancing my understanding about this. 

yanni kyriacos @ 2024-04-12T10:14 (+1) in response to Yanni Kyriacos's Quick takes

What are some historical examples of a group (like AI Safety folk) getting something incredibly wrong about an incoming Technology? Bonus question: what led to that group getting it so wrong? Maybe there is something to learn here.

titotal @ 2024-04-12T15:54 (+12)

In the 90's and 2000's, many people such as Eric Drexler were extremely worried about nanotechnology and viewed it as an existential threat through the "gray goo" scenario. Yudkowsky predicted drexler style nanotech would occur by 2010, using very similar language to what he is currently saying about AGI. 

It turned out they were all being absurdly overoptimistic about how soon the technology would arrive, and the whole drexlerite nanotech project flamed out by the end of the 2000's and has pretty much not progressed since. I think a similar dynamic playing out with AGI is less likely, but still very plausible. 

Jason @ 2024-04-12T15:46 (+2) in response to The Bunker Fallacy

One problem here is that a solution that operates continuously and provides "value way before an extinction level event plays out" is likely to be much more complex -- and have considerably more points of potential failure -- than a simpler solution.

Imagine I have a requirement that I be able to receive radio transmissions during an emergency. I could build a fairly simple emergency radio and pressure-test it. It doesn't provide me any value in ordinary times, so it just sits there on the shelf between pressure tests. It's cheap, so I could build several for redundancy. I could also build a smartphone with radio capacity -- this would be operated continuously and provide value in normal times. It is also much more complex and has many more failure modes than a simple radio. Many of those failure modes are common to both the ordinary-times use and the emergency-times use. But some are less so -- I might not be using the radio feature on my cellphone much at all in normal times because that particular function doesn't provide much if any pre-emergency value.

I think you're likely accepting a significantly higher level of failure modes for a better opportunity to detect and fix them. It's not clear to me if this is a good tradeoff or not. If you go back to the cellphone analogy, most things have to evolve and grow to continue providing enough value to justify their maintenance and operation. Maybe we could have a super-high-reliability smartphone if we locked in 2010 capabilities and worked on iterating the flaws out of it. But a 2010-level smartphone isn't going to get much use. And adding new features and capabilities adds new points of failure, so manufacturers are stuck playing a never-ending game of whack-a-mole.

I'm reminded of the troubles that the US government has had with getting cell phones up to the point that they can handle classified information. The poor user experience of cell phones approved for even secret information famously contributed to getting a former Secretary of State into some trouble. As of 2016, phones appeared to have only been approved to the secret level (not top secret, or ~"above top secret" like certain SCIs). In contrast, we've had landline devices cleared for all security levels since at least 1987 (I didn't check older devices). The security reliability risks are apparently a lot easier to manage with the simpler devices.

For similar reasons, the solution with pre-emergency value will likely be much more expensive to develop and operate. All critical components need to be hardened / highly reliable. The more you expect those components to do, the bigger / more complex they are going to be. If the power facility needs to power a small town, it will be much more expensive to harden than a solution for a bunker. If you rely on a different power solution for emergency situations and only harden that solution, then a critical system isn't going to be regularly operated. It's probably going to be considerably more expensive to run and maintain a hardened system than a standard one even in normal times -- so normal-time operation is likely to also require some significant subsidies. As a result, you're going to have to put a lot of eggs in a single basket for cost reasons alone.

Amber Dawn @ 2024-04-11T09:45 (+41) in response to Mediocre EAs: career paths and how do they engage with EA?

I’ve been thinking about this quite a bit recently. It’s not that I see myself as a “mediocre” EA, and in fact I work with EAs, so I am engaging with the community through my work. But I feel like a lot of the attitudes around career planning in EA sort of assume that you are formidable within a particular, rather narrow mould. You talk about mediocre EAs, but I’d also extend this to people who have strong skills and expertise that’s not obviously convertable into ‘working in the main EA cause areas’.

And the thing is, this kind of makes sense: like, if you’re a hardcore EA, it makes sense to give lots of attention and resources to people who can be super successful in the main EA cause areas, and comparatively neglect people who can’t. Inasmuch as the community’s primary aim is to do more good according to a specific set of assumptions and values, and not to be a fuzzy warm inclusive space, it makes sense that there aren’t a lot of resources for people who are less able to have an impact. But it's kind of annoying if you're one of those people! 

Or like: most EA jobs are crazy competitive nowadays. And from the point of view of "EA" (as an ideology), that's fine; impactful jobs should have large hiring pools of talented committed people. But from the point of view of people in the hiring pool, who are constantly applying to and getting rejected from EA jobs - or competitive non-EA jobs - because they've been persuaded these are the only jobs worth having, it kinda sucks.

There’s this well-known post ‘don’t be bycatch’; I currently suspect that EA structurally generates bycatch. By ‘structurally’ I mean ‘the behaviour of powerful actors in EA is kinda reasonable, but also it predictably creates situations where lots of altruistic, committed people get drawn into the community but can’t succeed within the paradigms the community has defined as success’. 

trevor1 @ 2024-04-12T15:45 (+2)

There's people who are good at EA-related thinking and people who are less good at that.

There's people who are good at accumulating resume padding, and people who are less good at that.

Although these are correlated, there will still be plenty of people who are good at EA thinking, but bad at accumulating resume padding. You can think of these people as having fallen through the cracks of the system.

Advances in LLMs give me the impression that we're around ~2-5 years out from most EA orgs becoming much better at correctly identifying/drawing talent from this pool e.g. higher-quality summaries of posts and notes, or tracing upstream origins of original ideas.

I'm less optimistic about solutions to conflict theory/value alignment issues, but advances in talent sourcing/measurement might give orgs more room to focus hiring/evaluation energy on character traits. If talent is easy to measure then there's less incentive to shrug and focus on candidates based on metrics that historically correlated with talent e.g. credentials.

mhendric @ 2024-04-12T15:36 (+3) in response to Guilt at EA

Hey there. Thanks for this post; I am sure many people can relate to your experience. I have found many EA's, especially within the first few years of encountering EA, are incredibly harsh with themselves - often to a highly unproductive degree. I think part of this may be related to moral burnout: as one starts embracing a very demanding ethical theory, there is no longer a clearly visible threshold above which one is 'safe', and can stop worrying. After all, every dollar or minute spent could be better spent elsewhere, or so our short-sighted brains tell us. Suddenly, every decision must be justified. 

I suspect a lot of this comes from a somewhat unfortunate framing. Rather than seeing EA as a way of having an outsized positive impact even with rather limited means, many people seem to see EA as an ideal of how to prioritize their every decision and see themselves as failing when not being able to optimize each and every decision. Your post sounds a bit as if you adopt the latter mindset, scolding yourself for failing to become vegetarian or hanging out with people you are related to (which no Utilitarian would object to!). But notice that there are many ways of helping animals and strangers that don't require you to be a vegetarian/ignoring those you are related to, such as via donations or volunteering. 

I have come to believe that this phenomenon is quite well-studied in other professions that frequently require tradeoffs and unfullfillable moral obligations, such as in healthcare settings. There, it is discussed under the monicker of moral burnout. I am currently working on a research project relating moral burnout and the demandingness of EA and other utilitarian theories. I have presented it to a few academic audiences (I am a philosopher), and I hope to finish the project this year. If I do, I'll post about it on the forum.

I'd also be very interested in compiling different such reports by EAs. I think an emotional first-aid kit may also make sense, but I would not be qualified to help with that.

I find Julia Wise very insightful on this topic. I recommend you check out her work, or maybe even reach out to her!