I recommend the China sections of this recent CNAS report as a starting point for discussion (it's definitely from a relatively hawkish perspective, and I don't think of myself as having enough expertise to endorse it, but I did move in this direction after reading). 

From the executive summary:

Taken together, perhaps the most underappreciated feature of emerging catastrophic AI risks from this exploration is the outsized likelihood of AI catastrophes originating from China. There, a combination of the Chinese Communist Party’s efforts to accelerate AI development, its track record of authoritarian crisis mismanagement, and its censorship of information on accidents all make catastrophic risks related to AI more acute.

From the "Deficient Safety Cultures" section:

While such an analysis is of relevance in a range of industry- and application-specific cultures, China’s AI sector is particularly worthy of attention and uniquely predisposed to exacerbate catastrophic AI risks [footnote]. China’s funding incentives around scientific and technological advancement generally lend themselves to risky approaches to new technologies, and AI leaders in China have long prided themselves on their government’s large appetite for risk—even if there are more recent signs of some budding AI safety consciousness in the country [footnote, footnote, footnote]. China’s society is the most optimistic in the world on the benefits and risks of AI technology, according to a 2022 survey by the multinational market research firm Institut Public de Sondage d’Opinion Secteur (Ipsos), despite the nation’s history of grisly industrial accidents and mismanaged crises—not least its handling of COVID-19 [footnote, footnote, footnote, footnote]. The government’s sprint to lead the world in AI by 2030 has unnerving resonances with prior grand, government-led attempts to accelerate industries that have ended in tragedy, as in the Great Leap Forward, the commercial satellite launch industry, and a variety of Belt and Road infrastructure projects [footnote, footnote, footnote]. China’s recent track record in other hightech sectors, including space and biotech, also suggests a much greater likelihood of catastrophic outcomes [footnote, footnote, footnote, footnote, footnote].

From "Further Considerations"

In addition to having to grapple with all the same safety challenges that other AI ecosystems must address, China’s broader tech culture is prone to crisis due to its government’s chronic mismanagement of disasters, censorship of information on accidents, and heavy-handed efforts to force technological breakthroughs. In AI, these dynamics are even more pronounced, buoyed by remarkably optimistic public perceptions of the technology and Beijing’s gigantic strategic gamble on boosting its AI sector to international preeminence. And while both the United States and China must reckon with the safety challenges that emerge from interstate technology competitions, historically, nations that perceive themselves to be slightly behind competitors are willing to absorb the greatest risks to catch up in tech races [footnote]. Thus, even while the United States’ AI edge over China may be a strategic advantage, Beijing’s self-perceived disadvantage could nonetheless exacerbate the overall risks of an AI catastrophe.

Also, unless one understands the Chinese situation, one should avoid moves that risk escalating a race, like making loud and confident predictions that a race is the only way.

For those who are not deep China nerds but want a somewhat approachable lowdown, I can highly recommend Bill Bishop's newsletter Sinocism (enough free issues to be worthwhile) and his podcast Sharp China (the latter is a bit more approachable but requires a subscription to Stratechery).

I'm not a China expert so I won't make strong claims, but I generally agree that we should not treat China as an unknowable, evil adversary who has exactly the same imperial desires as 'the west' or past non-Western regimes. I think it was irresponsible of Aschenbrenner to assume this without better research & understanding, since so much of his argument relies on China behaving in a particular way.

I share your concerns. I spent a decade in China, and I can't count the number of times I've seen people confidently share low-quality or inaccurate perspectives on China. I wish that I had a solution better than "assign everyone the read these [NUMBER] different books."

Even best selling books and articles by well-respected writers sometimes have misleading and inaccurate narratives in them. But it is hard to parse them critically  and to provide a counter argument without both the appropriate background^[1], and a large number of hours dedicated to the specific effort.

1. ^{^}

   I would be surprised if someone is able to do so without at least an undergraduate background in something like Chinese studies/sinology (or the equivalent, such as a large amount of self-study and independent exploration).

This reading list is an excellent place to start for getting a sense of China x AI (though it doesn't have that much about China's political objectives in general).

Note that you should also understand a) how the US government sees China and why, b) how China sees the US and why in order to be able to have a full analysis here.

My first reaction is that working on AI safety directly is more specialised and niche, so it might be your comparative advantage, while the 80k one might be filled by candidates from a wider range of backgrounds

It feels like when I'm comparing the person who does object-level work to the person who does meta-level work that leads to 2 people (say) doing object-level work, the latter really does seem better all things equal, but the intuition that calls this model naive is driven by a sense that it's going to turn out to not "actually" be 2 additional people, that additionality is going to be lower than you think, that the costs of getting that result are higher than you think, etc. etc.

But this intuition is not as clear as I'd like on what the extra costs / reduced benefits are, and how big a deal they are. Here are the first ones I can think of:

• Perhaps the people that you recruit instead aren't as good at the job as you would have been.
• If your org's hiring bottlenecks are not finding great people, but instead having the management capacity to onboard them or the funding capacity to pay for them, doing management or fundraising, or work that supports the case for fundraising, might matter more.
  • but 80k surely also needs good managers, at least as a general matter
• I think when an org hires you, there's an initial period of your onboarding where you consume more staff time than you produce, especially if you weight by seniority. Different roles differ strongly on where their break-even point is. I've worked somewhere who thought their number was like 6-18 months (I forget what they said exactly, but in that range) and I can imagine cases where it's more like... day 2 of employment. Anyway, one way or another, if you cause object level work to happen by doing meta level work, you're introducing another onboarding delay before stuff actually happens. If the area you're hoping to impact is time-sensitive, this could be a big deal? But usually I'm a little skeptical of time-sensitivity arguments, since people seem to make them at all times.
• it's easy to inadvertently take credit for a person going to role that they would actually have gone to anyway, or not to notice when you guide someone into a role that's worse (or not better, or not so much better) than what they would have done otherwise (80k are clearly aware of this and try to measure it in various ways, but it's not something you can do perfectly)

I suppose the complement to the naive thing I said before is "80k needs a compelling reason to recruit people to EA, and needs EA to be compelling to the people to recruit to it as well; by doing an excellent job at some object-level work, you can grow the value of 80k recruiting, both by making it easier to do and by making the outcome a more valuable outcome. Perhaps this might be even better for recruiting than doing recruiting."

I think there are a bunch of meta effects from working in an object level job:

• The object level work makes people more likely to enter the field as you note. (Though this doesn't just route through 80k and goes through a bunch of mechanisms.)
• You'll probably have some conversations with people considering entering the field from a slightly more credible position at least if the object level stuff goes well.
• Part of the work will likely involve fleshing stuff out so people with less context can more easily join/contribute. (True for most / many jobs.)

sometimes I feel bone-headedly stuck on even apparently-simple things like "if nonprofit growth is easier than for-profit growth, does that mean that nonprofits should spend more effort on growth, or less?"

Your AI timelines would likely be an important factor here.

Is your question how we should think about meta vs object level work, excluding considerations of personal fit? Because, at least in this example, I would expect fit considerations to dominate.

Basically, it seems to me that for any given worker, these career options would have pretty different levels of expected productivity, influenced by things like aptitude and excitement/motivation. And my prior is that in most cases, these productivity differences should swamp the sort of structural considerations you bring up here.

I feel at least somewhat qualified to speak on this, having read a bunch about human resources, being active in an HR professionals chat group nearly every day, and having worked in HR at a few different organizations (so I have seen some of the variance that exists). I hope you'll forgive me for my rambling on this topic, as there are several different ideas that came to mind when reading your paragraphs.

The first thing is that I agree with you on at least one aspect: rather than merely creating a department and walking away, adopting and adapting best practices and relevant expertise would be more helpful. If the big boss is okay with [insert bad behavior here] and isn't open to the HR Manager's new ideas, then the organization probably isn't going to change. If an HR department is defending and upholding sucky corporate culture, that is usually because senior leadership is instructing them to do so. Culture generally comes from the top. And if the leader isn't willing to be convinced by or have his mind changed by the new HRO he hired, then things probably won't be able to get much better.^[1]

"HR is not your friend" is normally used to imply that you can't trust HR, or that HR is out to get you, or something like that. Well, In a sense it is true that "HR is not your friend." If you are planning to do jump ship, don't confide in the HR manager about it trusting that they won't take action. If that person has a responsibility to take action on the information you provide, you should think twice before volunteering that information and consider if the action is beneficial to you or not. The job of the people on an HR team (just like the job of everyone else employed by an organization) is to help the organization achieve it's goals. Sometime that means pay raises for everyone, because the aren't salaries competitive and the company wants to have low attrition. Sometimes that means downsizing, because growth forecast were wrong and the company over-hired. The accountant is also not your friend, nor is the janitor, nor is marketing executive, nor is any other role at the organization. So I guess what I am getting at here is HR is not really more your friend or less your friend than any other department, but HR is the only department that carries out actions that might adversely affect employees. And note that just because HR carries out the actions, doesn't mean HR make the decision or put the company in that situation; this is the shooting the messenger.

While this may be true that in some organizations and for some people HR is "primarily about protecting the employer, often against the employee," I'm skeptical that this is representative of people who do HR work more generally. On the one hand, yes, the job is to help the organization achieve it's goals. But when talking about the individuals that work in HR, when this topic comes up among HR people the general reaction is along the lines of "I want to do as much as I can for the employees, and the boundaries limiting me are from upper management. I want to give our staff more equitable pay, but leadership doesn't care that we have high turnover rates. I want to provide parental leave, but the head honcho disagrees. I really do not want to fire John Doe, because it seems unreasonable and unfair and unjust, but this is what leadership has decided."^[2] 

The other thought I have about this parallels computer programmers/software engineers/developers and their thoughts on project managers. If you look at online discussions of programmers you will find no shortage of complaints about project managers (and about Scrum, and about agile), and many people writing about how useless their project manager is. But you shouldn't draw the conclusion that project management isn't useful. Instead, an alternative explanation is that these programmers are working with project managers that aren't very skillful, so their impression is biased. Working with a good project manager can be incredibly beneficial. So to leave the parallel and go back to HR, it is easy to find complaints on the internet about bad things that are attributed to HR. I would ask how representative those anecdotes are.

1. ^{^}

   Alternatively, if the leader is simply unaware of some bad things and the new HR manager can bring attention to those things, then improvements are probably on the way. But having HR is not sufficient on it's own.

2. ^{^}

   The other common response that tends to come up is to focus on all the things that HR does for the employees, things which are generally framed as limiting the company's power over employees: No, you can't pay the employees that little, because it is illegal. No, you can't fire this person without a documented history of poor performance, and no, scowling at you doesn't count as poor performance. Yes, you really do need to justify hiring your friend, and him being a 'great guy' isn't enough of a business case. No, it isn't reasonable to expect staff to be on call for mandatory unpaid overtime every weekend, because we will hemorrhage employees.

I think mainstream HR comes primarily from the private sector and is primarily about protecting the employer, often against the employee. They often cast themselves in a role of being there to help you, but a common piece of folk wisdom is "HR is not your friend". I think frankly that a lot of mainstream HR culture is at worst dishonest and manipulative, and I'd be really sad to see us uncritically importing more of that.

I see a lot of this online, but it doesn't match my personal experience. People working in HR that I've been in contact with seem generally kind people, aware of tradeoffs, and generally care about the wellbeing of employees.

I worry that the online reputation of HR departments is shaped by a minority of terrible experiences, and we overgeneralize that to think that HR cannot or will not help, while in my experience they are often really eager to try to help (in part because they don't want you and others to quit, but also because they are nice people).

Maybe it's also related to minimum-wage non-skilled jobs vs higher paying jobs, where employment tends to be less adversarial and less exploitative.

I like this. I've occasionally thought previously about what value there would be in having a 'historian.' There are many things that I took a while to figure out (such as the history/lineage of various ideas and organizations, or why there was a strategy shift from one thing to another thing), as well as the many things which I've simply never encountered. I imagine that there are plenty of lessons that can be learned from those.

EA as a community tends to do a better-than-normal job when it comes to writing and sharing retrospectives, but there are lots of things that I don't understand and that (I think) aren't easily available. (simplistic example: was asking for randomized control trials (or other methods) to demonstrate effectiveness really shockingly revolutionary in development work?)

On (1), these issues seem to be structural in nature, but exploited by idiosyncrasies. In theory, both OpenAI's non-profit board & Anthropic's LTBT should perform the roughly same oversight function. In reality, a combination of Sam's rebellion, Microsoft's financial domination, and the collective power of the workers shifted the decision to being about whether OpenAI would continue independently with a new board or re-form under Microsoft. Anthropic is just as susceptible to this kind of coup (led by Amazon), but only if their leadership and their workers collectively want it, which, in all fairness, I think they're a lot less likely to.

But in some sense, no corporate structure can protect against all of the key employees organising to direct their productivity somewhere else. Only a state-backed legal structure really has that power. If you're worried about some bad outcome, I think you either have to trust that the Anthropic people have good intentions and won't sell themselves to Amazon, or advocate for legal restrictions on AI work.

Say more about Conjecture's structure?

As weird as it sounds, I think the downvote button should make you a bit less concerned with contribution quality. If it's obviously bad, people will downvote and read it less. If it's wrong without being obviously bad, then others likely share the same misconception, and hopefully someone steps in to correct it.

In practice, the failure mode for the forum seems to be devoting too much attention to topics that don't deserve it. If your topic deserves more attention, I wouldn't worry a ton about accidentally repeating known info? For one thing, it could be valuable spaced repetition. For another, discussions over time can help turn something over and look at it from various angles. So I suppose the main risk is making subject matter experts bored?

In some sense you could consider the signal/noise question separate from the epistemic hygiene question. If you express uncertainty properly, then in theory, you can avoid harming collective epistemics even for a topic you know very little about.

On the current margin, I actually suspect EAs should be deferring less and asking dumb questions more. Specific example: In a world where EA was more willing to entertain dumb questions, perhaps we could've discovered AI Pause without Katja Grace having to write a megapost. We don't want to create "emperor has no clothes" type situations. Right now, "EA is a cult" seems to be a more common outsider critique than "EAs are ignorant and uneducated".

Using Kialo for debates rather than the Forum would go a long way. It's hard to get off the ground because it's attractiveness to use is roughly proportional to the number of EAs using it, and at present, the number of EAs using it is zero.

https://www.kialo-edu.com/

I was just thinking about this a few days ago when I was flying for the holidays. Outside the plane was a sign that said something like

Warning: Jet fuel emits chemicals that may increase the risk of cancer.

And I was thinking about whether this was a justified double-hedge. The author of that sign has a subjective belief that exposure to those chemicals increases the probability that you get cancer, so you could say "may give you cancer" or "increases the risk of cancer". On the other hand, perhaps the double-hedge is reasonable in cases like this because there's some uncertainty about whether a dangerous thing will cause harm, and there's also uncertainty about whether a particular thing is dangerous, so I supposed it's reasonable to say "may increase the risk of cancer". It means "there is some probability that this increases the probability that you get cancer, but also some probability that it has no effect on cancer rates."

Today I got a dose of Novavax for free, largely by luck that's probably not reproducible.

It turns out that vials of Novavax contain 5 doses and only last a short time, I think for 24 hours. Pharmacies therefore need to batch bookings together, and I guess someone got tired of waiting and opted to just buy the entire vial for themselves, letting whoever pick up the other doses. I then found about this via Rochelle Harris, who in turn found out about it via a Facebook group (UK Novavax Vaccine info) for coordinating these things.

I've been linked to The benefits of Novavax explained which is optimistic about the strengths of Novavax, suggesting it has the potential to offer longer-term protection, and protection against variants as well.

I think the things the article says or implies about pushback from mRNA vaccine supporters seem unlikely to me -- my guess is that in aggregate Wall Street benefits much more from eliminating COVID than it does from selling COVID treatments, though individual pharma companies might feel differently -- but they seem like the sort of unlikely thing that someone who had reasonable beliefs about the science but spent too much time arguing on Twitter might end up believing. Regardless, I'm left unsure how to feel about its overall reliability, and would welcome thoughts one way or the other.

Agree that there's a different incentive for cooperative writing than for clickbait-y news in particular. And I agree with your recommendations. That said, I think many community writers may undervalue making their content more goddamn readable. Scott Alexander is a verbose and often spends paragraphs getting to the start of his point, but I end up with a better understanding of what he's saying by virtue of being fully interested.

All in all though, I'd recommend people try to write like Paul Graham more than either Scott Alexander or an internal memo. He is in general more concise than Scott and more interesting than a memo.

He has several essays about how he writes.

Writing, Briefly — Laundry list of tips

Write like you talk

The Age of the Essay — History of the essays we write in school versus the essays that are useful

A Version 1.0 — "The Age of the Essay" in rough draft form with color coding for if it was kept

The "non-tweet" feels vague and unsubsantiated (at this point anyway). I hope we'll get a full article and explanation as to what he means exactly because obviously he's making HUGE calls.

Also see:

https://marginalrevolution.com/marginalrevolution/2017/08/can-short-apocalypse.html

credit to AGB for (in this comment) reminding me where to find the Scott Alexander remarks that pushed me a lot in this direction:

Second, if we never get AI, I expect the future to be short and grim. Most likely we kill ourselves with synthetic biology. If not, some combination of technological and economic stagnation, rising totalitarianism + illiberalism + mobocracy, fertility collapse and dysgenics will impoverish the world and accelerate its decaying institutional quality. I don’t spend much time worrying about any of these, because I think they’ll take a few generations to reach crisis level, and I expect technology to flip the gameboard well before then. But if we ban all gameboard-flipping technologies (the only other one I know is genetic enhancement, which is even more bannable), then we do end up with bioweapon catastrophe or social collapse. I’ve said before I think there’s a ~20% chance of AI destroying the world. But if we don’t get AI, I think there’s a 50%+ chance in the next 100 years we end up dead or careening towards Venezuela. That doesn’t mean I have to support AI accelerationism because 20% is smaller than 50%. Short, carefully-tailored pauses could improve the chance of AI going well by a lot, without increasing the risk of social collapse too much. But it’s something on my mind.

(emphasis mine)

My original shortform tried to be measured / neutral, but I want to also say I found this passage very alarming when I first read it, and it's wildly more pessimistic than I am by default. I think if this is true, it's really important to know, but I made my shortform because if it's false, that's really important to know too. I hope we can look into it in a way that moves the needle on our best understanding.

Edit: This is now The illusion of consensus about EA celebrities

Something to try to dispel the notion that every EA thinker is respected/ thought highly of by every EA community member. Like, you tend to hear strong positive feedback, weak positive feedback, and strong negative feedback, but weak negative feedback is kind of awkward and only comes out sometimes

something about the role of emotions in rationality and why the implicit / perceived Forum norm against emotions is unhelpful, or at least not precisely aimed

(there's a lot of nuance here, I'll put it in dw)

edit: I feel like the "notice your confusion" meme is arguably an example of emotional responses providing rational value.

Take a list of desirable qualities of a non-profit board (either Holden's or another that was posted recently) and look at some EA org boards and do some comparison / review their composition and recent activity.

edit: I hear Nick Beckstead has written about this too

The Optimal Number of Innocent People's Careers Ruined By False Allegations Is Not Zero

(haha just kidding... unless? 🥺)

I think "human-level" is often a misleading benchmark for AI, because we already have AIs that are massively superhuman in some respects and substantially subhuman in others. I sometimes worry that this is leading people to make unwarranted assumptions of how closely future dangerous AIs will track humans in terms of what they're capable of. This is related to a different post I'm writing, but maybe deserves its own separate treatment too.

A problem with a lot of AI thoughts I have is that I'm not really in enough contact with the AI "mainstream" to know what's obvious to them or what's novel. Maybe "serious" AI people already don't say human-level, or apply a generous helping of "you know what I mean" when they do?

I have an intuition that the baseline average for institutional dysfunction is quite high, and I think I am significantly less bothered by negative news about orgs than many people because I already expect the average organisation (from my experience both inside and outside EA) to have a few internal secrets that seem "shockingly bad" to a naive outsider. This seems tricky to communicate / write about because my sense of what's bad enough to be worthy of action even relative to this baseline is not very explicit, but maybe something useful could be said.

Things I've learned about good mistake culture, no-blame post-mortems, etc. This is pretty standard stuff without a strong EA tilt so I'm not sure it merits a place on the forum, but it's possible I overestimate how widely known it is, and I think it's important in basically any org culture.

Disclosure-based regulation (in the SEC style) as a tool either for internal community application or perhaps in AI or biosecurity

Something contra "excited altruism": lots of our altruistic opportunities exist because the world sucks and it's ok to feel sad about that and/or let down by people who have failed to address it.

edit: relevant prior work:

• https://forum.effectivealtruism.org/posts/Nk5nJYPYYheQsZ6zn/impossible-ea-emotions
• https://forum.effectivealtruism.org/posts/bkjNa2WAZvqahqpoH/it-s-supposed-to-feel-like-this-8-emotional-challenges-of

Encouraging people to take community health interventions into their own hands. Like, ask what you wish someone in community health would do, and then consider just doing it. With some caveats for unilateralist curse risks.

I think the forum would be better if people didn't get hit so hard by negative feedback, or by people not liking what they have to say. I don't know how to fix this with a post, but at least arguing the case might have some value.

Assessments of non-AI x-risk are relevant to AI safety discussions because some of the hesitance to pause or slow AI progress is driven by a belief that it will help eliminate other threats if it goes well.

I tend to believe that risk from non-AI sources is pretty low, and I'm therefore somewhat alarmed when I see people suggest or state relatively high probabilities of civilisational collapse without AI intervention. Could be worth trying to assess how widespread this view is and trying to argue directly against it.

This one might be for LW or the AF instead / as well, but I'd like to write a post about:

• should we try to avoid some / all alignment research casually making it into the training sets for frontier AI models?
• if so, what are the means that we can use to do this? how do they fare on the ratio between reduction in AI access vs. reduction in human access?

my other quick take, AI Safety Needs To Get Serious About Chinese Political Culture is basically a post idea, but it was substantial enough I put it at the top level rather than have it languish in the comments here. Nevertheless, here it is so I can keep all the things in one place.

"ask not what you can do for EA, but what EA can do for you"

like, you don't support EA causes or orgs because they want you to and you're acquiescing, you support them because you want to help people and you believe supporting the org will do that – when you work an EA job, instead of thinking "I am helping them have an impact", think "they are helping me have an impact"

of course there is some nuance in this but I think broadly this perspective is the more neglected one

I have a Google Sheet set up that daily records the number of unread emails in my inbox. Might be a cute shortform post.

Some criticism of the desire to be the donor of last resort, skepticism of the standard counterfactual validity concerns.

If everyone has no idea what other people are funding and instead just donates a scaled down version of their ideal community-wide allocation to everything, what you get is a wealth-weighted average of everyone's ideal portfolios. Sometimes this is an okay outcome. There's some interesting dynamics to write about here, but equally I'm not sure it leads to anything actionable.

I'd like to write something about my skepticism of for-profit models of doing alignment research. I think this is a significant part of why I trust Redwood more than Anthropic or Conjecture.

(This could apply to non-alignment fields as well, but I'm less worried about the downsides of product-focused approaches to (say) animal welfare.)

That said, I would want to search for existing discussion of this before I wade into it.

Something about the value of rumours and the whisper network

"earning to receive"

(I think this is Habiba's joke)

L/acc, who think that LEEP have gone too far

(possibly this one was Amber's idea)

SummaryBot has executed a treacherous turn and now runs the EA forum

Tools for shaping probability intuitions. You can give a bunch of events, casual relationships or implications between them, and probabilities for each, or their conjunctions, or conditional probabilities for such things. The tool will infer what you don't supply to the extent possible, and will point out contradictions in your conditional vs. absolute probabilities, and give you recommendations for how to resolve them.

Automated interface between Twitter and the Forum (eg a bot that, when tagged on twitter, posts the text and image of a tweet on Quick Takes and vice versa)