Ozzie Gooen's Quick takes

I’ve heard multiple reports of people being denied jobs around AI policy because of their history in EA. I’ve also seen a lot of animosity against EA from top organizations I think are important - like A16Z, Founders Fund (Thiel), OpenAI, etc. I’d expect that it would be uncomfortable for EAs to apply or work in to these latter places at this point.

This is very frustrating to me.

First, it makes it much more difficult for EAs to collaborate with many organizations where these perspectives could be the most useful. I want to see more collaborations and cooperation - not having EAs be allowed in many orgs makes this very difficult.

Second, it creates a massive incentive for people not to work in EA or on EA topics. If you know it will hurt your career, then you’re much less likely to do work here.

And a lighter third - it’s just really not fun to have a significant stigma associated with you. This means that many of the people I respect the most, and think are doing some of the most valuable work out there, will just have a much tougher time in life.

Who’s at fault here? I think the first big issue is that resistances get created against all interesting and powerful groups. There are similar stigmas against people across the political spectrum, for example, to certain crowds. A big part of “talking about morality and important issues, while having something non-obvious to say” is being hated by a bunch of people. In this vein, arguably we should be aiming for a world where it winds up that there’s a larger stigma.

But a lot clearly has to do with the decisions made by what seems like a few EAs. FTX hurt the most. I think the OpenAI board situation resulted in a lot of ea-paranoia, arguably with very little upside. More recently, I think that certain EA actions in ai policy are getting a lot of flak.

There was a brief window, pre-FTX-fail, where there was a very positive EA media push. I’ve seen almost nothing since. I think that “EA marketing” has been highly neglected, and that doesn’t seem to be changing.

Also, I suspect that the current EA AI policy arm could find ways to be more diplomatic and cooperative. When this arm upsets people, all of EA gets blamed.

My guess is that there are many other changes to do here too.

CEA is the obvious group to hypothetically be in charge of the EA parts of this. In practice, it seems like CEA has been very busy with post-FTX messes and leadership changes.

So I think CEA, as it becomes stable, could do a lot of good work making EA marketing work somehow. And I hope that the AI safety governance crowd can get better at not pissing off people. And hopefully, other EAs can figure out other ways to make things better and not worse.

If the above doesn’t happen, honestly, it could be worth it for EAs themselves to try to self-fund or coordinate efforts on this. The issue isn’t just one of “hurting long-term utility”, it’s one that just directly hurts EAs - so it could make a lot of sense for them to coordinate on improvements, even just in their personal interests.

I'm nervous that the EA Forum might be having a small role for x-risk and some high-level prioritization work.
- Very little biorisk content here, perhaps because of info-hazards.
- Little technical AI safety work here, in part because that's more for LessWrong / Alignment Forum.
- Little AI governance work here, for whatever reason.
- Not too much innovative, big-picture longtermist prioritization projects happening at the moment, from what I understand. 
- The cause of "EA community building" seems to be fairly stable, not much bold/controversial experimentation, from what I can tell.
- Fairly few updates / discussion from grantmakers. OP is really the dominant one, and doesn't publish too much, particularly about their grantmaking strategies and findings.

It's been feeling pretty quiet here recently, for my interests. I think some important threads are now happening in private slack / in-person conversations or just not happening. 

Personal reflections on self-worth and EA

My sense of self-worth often comes from guessing what people I respect think of me and my work. 

In EA... this is precarious. The most obvious people to listen to are the senior/powerful EAs.

In my experience, many senior/powerful EAs I know:
1. Are very focused on specific domains.
2. Are extremely busy.
3. Have substantial privileges (exceptionally intelligent, stable health, esteemed education, affluent/ intellectual backgrounds.)
4. Display limited social empathy (ability to read and respond to the emotions of others)
5. Sometimes might actively try not to sympathize/empathize with many people, because they are judging them for grants, and want don't want to be biased. (I suspect this is the case for grantmakers). 
6. Are not that interested in acting as a coach/mentor/evaluator to people outside their key areas/organizations.
7. Don't intend or want others to care too much about what they think outside of cause-specific promotion and a few pet ideas they want to advance.

A parallel can be drawn with the world of sports. Top athletes can make poor coaches. Their innate talent and advantages often leave them detached from the experiences of others. I'm reminded by David Foster Wallace's How Tracy Austin Broke My Heart.

If you're a tennis player, tying your self-worth to what Roger Federer thinks of you is not wise. Top athletes are often egotistical, narrow-minded, and ambivalent to others. This sort of makes sense by design - to become a top athlete, you often have to obsess over your own abilities to an unnatural extent for a very long period.

Good managers are sometimes meant to be better as coaches than they are as direct contributors. In EA, I think those in charge seem more like "top individual contributors and researchers" than they do "top managers." Many actively dislike management or claim that they're not doing management. (I believe funders typically don't see their work as "management*", which might be very reasonable.)

But that said, even a good class of managers wouldn't fully solve the self-worth issue. Tying your self-worth too much to your boss can be dangerous - your boss already has much power and control over you, so adding your self-worth to the mix seems extra precarious.

I think if I were to ask any senior EA I know, "Should I tie my self-worth with your opinion of me?" they would say something like,

"Are you insane? I barely know you or your work. I can't at all afford the time to evaluate your life and work enough to form an opinion that I'd suggest you take really seriously."

They have enough problems - they don't want to additionally worry about others trying to use them as judges of personal value.

But this raises the question, Who, if anyone, should I trust to inform my self-worth?

Navigating intellectual and rationalist literature, I've grown skeptical of many other potential evaluators. Self-judgment carries inherent bias and ability to Goodhart. Many "personal coaches" and even "executive coaches" raise my epistemic alarm bells. Friends, family, and people who are "more junior" come with different substantial biases.

Some favored options are "friends of a similar professional class who could provide long-standing perspective" and "professional coaches/therapists/advisors."

I’m not satisfied with any obvious options here. I think my next obvious move forward is to acknowledge that my current situation seems subpar and continue reflecting on this topic. I've dug into the literature a bit but haven't found answers I've yet found compelling.

(This is a draft I wrote in December 2021. I didn't finish+publish it then, in part because I was nervous it could be too spicy. At this point, with the discussion post-chatGPT, it seems far more boring, and someone recommended I post it somewhere.)

Thoughts on the OpenAI Strategy

OpenAI has one of the most audacious plans out there and I'm surprised at how little attention it's gotten.

First, they say flat out that they're going for AGI.

Then, when they raised money in 2019, they had a clause that says investors will be capped at getting 100x of their returns back.

"Economic returns for investors and employees are capped... Any excess returns go to OpenAI Nonprofit... Returns for our first round of investors are capped at 100x their investment (commensurate with the risks in front of us), and we expect this multiple to be lower for future rounds as we make further progress."[1]

On Hacker News, one of their employees says,

"We believe that if we do create AGI, we'll create orders of magnitude more value than any existing company." [2]

You can read more about this mission on the charter:

"We commit to use any influence we obtain over AGI’s deployment to ensure it is used for the benefit of all, and to avoid enabling uses of AI or AGI that harm humanity or unduly concentrate power.

Our primary fiduciary duty is to humanity. We anticipate needing to marshal substantial resources to fulfill our mission, but will always diligently act to minimize conflicts of interest among our employees and stakeholders that could compromise broad benefit."[3]

This is my [incredibly rough and speculative, based on the above posts] impression of the plan they are proposing:

1. Make AGI
2. Turn AGI into huge profits
3. Give 100x returns to investors
4. Dominate much (most?) of the economy, have all profits go to the OpenAI Nonprofit
5. Use AGI for "the benefit of all"?

I'm really curious what step 5 is supposed to look like exactly. I’m also very curious, of course, what they expect step 4 to look like.

Keep in mind that making AGI is a really big deal. If you're the one company that has an AGI, and if you have a significant lead over anyone else that does, the world is sort of your oyster.[4] If you have a massive lead, you could outwit legal systems, governments, militaries.

I imagine that the 100x return cap means that the excess earnings would go to the hands of the nonprofit; which essentially means Sam Altman, senior leadership at OpenAI, and perhaps the board of directors (if legal authorities have any influence post-AGI).

This would be a massive power gain for a small subset of people.

If DeepMind makes AGI I assume the money would go to investors, which would mean it would be distributed to all of the Google shareholders. But if OpenAI makes AGI, the money will go to the leadership of OpenAI, on paper to fulfill the mission of OpenAI.

On the plus side, I expect that this subset is much more like the people reading this post than most other AGI competitors would be. (The Chinese government, for example). I know some people at OpenAI, and my hunch is that the people there are very smart and pretty altruistic. It might well be about the best we could expect from a tech company.

And, to be clear, it’s probably incredibly unlikely that OpenAI will actually create AGI, and even more unlikely they will do so with a decisive edge over competitors.

But, I'm sort of surprised so few other people seem at least a bit concerned and curious about the proposal? My impression is that most press outlets haven't thought much at all about what AGI would actually mean, and most companies and governments just assume that OpenAI is dramatically overconfident in themselves. 

(Aside on the details of Step 5)
I would love more information on Step 5, but I don’t blame OpenAI for not providing it.

• Any precise description of how a nonprofit would spend “a large portion of the entire economy” would upset a bunch of powerful people.
• Arguably, OpenAI doesn’t really need to figure out Step 5 unless their odds of actually having a decisive AGI advantage seem more plausible.
• I assume it’s really hard to actually put together any reasonable plan now for Step 5. 

My guess is that we really could use some great nonprofit and academic work to help outline what a positive and globally acceptable (wouldn’t upset any group too much if they were to understand it) Step 5 would look like. There’s been previous academic work on a “windfall clause”[5] (their 100x cap would basically count), having better work on Step 5 seems very obvious.

[1] https://openai.com/blog/openai-lp/

[2] https://news.ycombinator.com/item?id=19360709

[3] https://openai.com/charter/
[4] This was titled a “decisive strategic advantage” in the book Superintelligence by Nick Bostrom

[5] https://www.effectivealtruism.org/articles/cullen-okeefe-the-windfall-clause-sharing-the-benefits-of-advanced-ai/

Also, see:
https://www.cnbc.com/2021/03/17/openais-altman-ai-will-make-wealth-to-pay-all-adults-13500-a-year.html
Artificial intelligence will create so much wealth that every adult in the United States could be paid $13,500 per year from its windfall as soon as 10 years from now.

https://www.techtimes.com/articles/258148/20210318/openai-give-13-500-american-adult-anually-sam-altman-world.htm

https://moores.samaltman.com/

https://www.reddit.com/r/artificial/comments/m7cpyn/openais_sam_altman_artificial_intelligence_will/

Around EA Priorities:

Personally, I feel fairly strongly convinced to favor interventions that could help the future past 20 years from now. (A much lighter version of "Longtermism").

If I had a budget of $10B, I'd probably donate a fair bit to some existing AI safety groups. But it's tricky to know what to do with, say, $10k. And the fact that the SFF, OP, and others have funded some of the clearest wins makes it harder to know what's exciting on-the-margin.

I feel incredibly unsatisfied with the public EA dialogue around AI safety strategy now. From what I can tell, there's some intelligent conversation happening by a handful of people at the Constellation coworking space, but a lot of this is barely clear publicly. I think many people outside of Constellation are working on simplified models, like "AI is generally dangerous, so we should slow it all down," as opposed to something like, "Really, there are three scary narrow scenarios we need to worry about."

I recently spent a week in DC and found it interesting. But my impression is that a lot of people there are focused on fairly low-level details, without a great sense of the big-picture strategy. For example, there's a lot of work into shovel-ready government legislation, but little thinking on what the TAI transition should really look like.

This sort of myopic mindset is also common in the technical space, where I meet a bunch of people focused on narrow aspects of LLMs, without much understanding of how their work exactly fits into the big picture of AI alignment. As an example, a lot of work seems like it would help with misuse risk, even when the big-picture EAs seem much more focused on accident risk.

Some (very) positive news is that we do have far more talent in this area than we did 5 years ago, and there's correspondingly more discussion. But it still feels very chaotic.

A bit more evidence - it seems like OP has provided very mixed messages around AI safety. They've provided surprisingly little funding / support for technical AI safety in the last few years (perhaps 1 full-time grantmaker?), but they have seemed to provide more support for AI safety community building / recruiting, and AI policy. But all of this still represents perhaps ~30% or so of their total budget, and I don't sense that that's about to change. Overall this comes off as measured and cautious. Meanwhile, it's been difficult to convince other large donors to get into this area. (Other than Jaan Tallinn, he might well have been the strongest dedicated donor here).

Recently it seems like the community on the EA Forum has shifted a bit to favor animal welfare. Or maybe it's just that the AI safety people have migrated to other blogs and organizations.

But again, I'm very hopeful that we can find interventions that will help in the long-term, so few of these excite me. I'd expect and hope that interventions that help the long-term future would ultimately improve animal welfare and more.

So on one hand, AI risk seems like the main intervention area for the long-term, but on the other, the field is a bit of a mess right now.

I feel quite frustrated that EA doesn't have many other strong recommendations for other potential donors interested in the long-term. For example, I'd really hope that there could be good interventions to make the US government or just US epistemics more robust, but I barely see any work in that area.

"Forecasting" is one interesting area - it currently does have some dedicated support from OP. But it honestly seems to be in a pretty mediocre state to me right now. There might be 15-30 full-time people in the space at this point, and there's surprisingly little in terms of any long-term research agendas.

I really don't like the trend of posts saying that "EA/EAs need to | should do X or Y".

EA is about cost-benefit analysis. The phrases need and should implies binaries/absolutes and having very high confidence.

I'm sure there are thousands of interventions/measures that would be positive-EV for EA to engage with. I don't want to see thousands of posts loudly declaring "EA MUST ENACT MEASURE X" and "EAs SHOULD ALL DO THING Y," in cases where these mostly seem like un-vetted interesting ideas. 

In almost all cases I see the phrase, I think it would be much better replaced with things like;
"Doing X would be high-EV"
"X could be very good for EA"
"Y: Cost and Benefits" (With information in the post arguing the benefits are worth it)
"Benefits|Upsides of X" (If you think the upsides are particularly underrepresented)"

I think it's probably fine to use the word "need" either when it's paired with an outcome (EA needs to do more outreach to become more popular) or when the issue is fairly clearly existential (the US needs to ensure that nuclear risk is low). It's also fine to use should in the right context, but it's not a word to over-use. 

On the funding-talent balance:

When EA was starting, there was a small amount of talent, and a smaller amount of funding. As one might expect, things went slowly for the first few years.

Then once OP decided to focus on X-risks, there was ~$8B potential funding, but still fairly little talent/capacity. I think the conventional wisdom then was that we were unlikely to be bottlenecked by money anytime soon, and lots of people were encouraged to do direct work.

Then FTX Future Fund came in, and the situation got even more out-of-control. ~Twice the funding. Projects got more ambitious, but it was clear there were significant capacity (funder and organization) constraints.

Then (1) FTX crashed, and (2) lots of smart people came into the system. Project capacity grew, AI advances freaked out a lot of people, and successful community projects helped train a lot of smart young people to work on X-risks.

But funding has not kept up. OP has been slow to hire for many x-risk roles (AI safety, movement building, outreach / fundraising). Other large funders have been slow to join in.

So now there's a crunch for funding. There are a bunch of smart-seeming AI people now who I bet could have gotten funding during the FFF, likely even before then with OP, but are under the bar now.

I imagine that this situation will eventually improve, but of course, it would be incredibly nice if it could happen sooner. It seems like EA leadership eventually fix things, but it often happens slower than is ideal, with a lot of opportunity loss in that time.

Opportunistic people can fill in the gaps. Looking back, I think more money and leadership in the early days would have gone far. Then, more organizational/development capacity during the FFF era. Now, more funding seems unusually valuable.

If you've been thinking about donating to the longtermist space, specifically around AI safety, I think it's likely that funding this year will be more useful than funding in the next 1-3 years. (Of course, I'd recommend using strong advisors or giving to funds, instead of just choosing directly, unless you can spend a fair bit of time analyzing things).

If you're considering entering the field as a nonprofit employee, heed some caution. I still think the space can use great talent, but note that this is an unusually competitive time to get many paid roles or to get nonprofit grants.

EA seems to have been doing a pretty great job attracting top talent from the most prestigious universities. While we attract a minority of the total pool, I imagine we get some of the most altruistic+rational+agentic individuals. 

If this continues, it could be worth noting that this could have significant repercussions for areas outside of EA; the ones that we may divert them from. We may be diverting a significant fraction of the future "best and brightest" in non-EA fields. 

If this seems possible, it's especially important that we do a really, really good job making sure that we are giving them good advice. 

Some musicians have multiple alter-egos that they use to communicate information from different perspectives. MF Doom released albums under several alter-egos; he even used these aliases to criticize his previous aliases.

Some musicians, like Madonna, just continued to "re-invent" themselves every few years.

Youtube personalities often feature themselves dressed as different personalities to represent different viewpoints. 

It's really difficult to keep a single understood identity, while also conveying different kinds of information.

Narrow identities are important for a lot of reasons. I think the main one is predictability, similar to a company brand. If your identity seems to dramatically change hour to hour, people wouldn't be able to predict your behavior, so fewer could interact or engage with you in ways they'd feel comfortable with.

However, narrow identities can also be suffocating. They restrict what you can say and how people will interpret that. You can simply say more things in more ways if you can change identities. So having multiple identities can be a really useful tool.

Sadly, most academics and intellectuals can only really have one public identity.

---

EA researchers currently act this way.

In EA, it's generally really important to be seen as calibrated and reasonable, so people correspondingly prioritize that in their public (and then private) identities. I've done this. But it comes with a cost.

One obvious (though unorthodox) way around this is to allow researchers to post content either under aliases. It could be fine if the identity of the author is known, as long as readers can keep these aliases distinct.

I've been considering how to best do this myself. My regular EA Forum name is just "Ozzie Gooen". Possible aliases would likely be adjustments to this name.

- "Angry Ozzie Gooen" (or "Disagreeable Ozzie Gooen")

- "Tech Bro Ozzie Gooen"

- "Utility-bot 352d3"

These would be used to communicate in very different styles, with me attempting what I'd expect readers to expect of those styles.

(Normally this is done to represent viewpoints other than what they have, but sometimes it's to represent viewpoints they have, but wouldn't normally share)

Facebook Discussion

I can't seem to find much EA discussion about [genetic modification to chickens to lessen suffering]. I think this naively seems like a promising area to me. I imagine others have investigated and decided against further work, I'm curious why. 

I think I broadly like the idea of Donation Week. 

One potential weakness is that I'm curious if it promotes the more well-known charities due to the voting system. I'd assume that these are somewhat inversely correlated with the most neglected charities.

Related, I'm curious if future versions could feature specific subprojects/teams within charities. "Rethink Priorities" is a rather large project compared to "PauseAI US", I assume it would be interesting if different parts of it were put here instead. 

(That said, in terms of the donation, I'd hope that we could donate to RP as a whole and trust RP to allocate it accordingly, instead of formally restricting the money, which can be quite a hassle in terms of accounting) 

When I hear of entrepreneurs excited about prediction infrastructure making businesses, I feel like they gravitate towards new prediction markets or making new hedge funds.

I really wish it were easier to make new insurance businesses (or similar products). I think innovative insurance products could be a huge boon to global welfare. The very unfortunate downside is that there's just a ton of regulation and lots of marketing to do, even in cases where it's a clear win for consumers.

Ideally, it should be very easy and common to get insurance for all of the key insecurities of your life.

• Having children with severe disabilities / issues
• Having business or romantic partners defect on you
• Having your dispreferred candidate get elected
• Increases in political / environmental instability
• Some unexpected catastrophe will hit a business
• Nonprofits losing their top donor due to some unexpected issue with said donor (i.e. FTX)

I think a lot of people have certain issues that both:

1. They worry about a lot
2. They over-weight the risks of these issues

In these cases, insurance could be a big win!

In a better world, almost all global risks would be held primarily by asset managers / insurance agencies. Individuals could have highly predictable lifestyles.

(Of course, some prediction markets and other markets can occasionally be used for this purpose as well!)

A few junior/summer effective altruism related research fellowships are ending, and I’m getting to see some of the research pitches.

Lots of confident-looking pictures of people with fancy and impressive sounding projects.

I want to flag that many of the most senior people I know around longtermism are really confused about stuff. And I’m personally often pretty skeptical of those who don’t seem confused.

So I think a good proposal isn’t something like, “What should the EU do about X-risks?” It’s much more like, “A light summary of what a few people so far think about this, and a few considerations that they haven’t yet flagged, but note that I’m really unsure about all of this.”

Many of these problems seem way harder than we’d like for them to be, and much harder than many seem to assume at first. (perhaps this is due to unreasonable demands for rigor, but an alternative here would be itself a research effort).

I imagine a lot of researchers assume they won’t stand out unless they seem to make bold claims. I think this isn’t true for many EA key orgs, though it might be the case that it’s good for some other programs (University roles, perhaps?).

Not sure how to finish this post here. I think part of me wants to encourage junior researchers to lean on humility, but at the same time, I don’t want to shame those who don’t feel like they can do so for reasons of not-being-homeless (or simply having to leave research). I think the easier thing is to slowly spread common knowledge and encourage a culture where proper calibration is just naturally incentivized.

Facebook Thread

Could/should altruistic activist investors buy lots of Twitter stock, then pressure them to do altruistic things?

---

So, Jack Dorsey just resigned from Twitter.

Some people on Hacker News are pointing out that Twitter has had recent issues with activist investors, and that this move might make those investors happy.

https://pxlnv.com/linklog/twitter-fleets-elliott-management/

From a quick look... Twitter stock really hasn't been doing very well. It's almost back at its price in 2014.

Square, Jack Dorsey's other company (he was CEO of two), has done much better. Market cap of over 2x Twitter ($100B), huge gains in the last 4 years.

I'm imagining that if I were Jack... leaving would have been really tempting. On one hand, I'd have Twitter, which isn't really improving, is facing activist investor attacks, and worst, apparently is responsible for global chaos (of which I barely know how to stop). And on the other hand, there's this really tame payments company with little controversy.

Being CEO of Twitter seems like one of the most thankless big-tech CEO positions around.

That sucks, because it would be really valuable if some great CEO could improve Twitter, for the sake of humanity.

One small silver lining is that the valuation of Twitter is relatively small. It has a market cap of $38B. In comparison, Facebook/Meta is $945B and Netflix is $294B.

So if altruistic interests really wanted to... I imagine they could become activist investors, but like, in a good way? I would naively expect that even with just 30% of the company you could push them to do positive things. $12B to improve global epistemics in a major way.

The US could have even bought Twitter for 4% of the recent $1T infrastructure bill. (though it's probably better that more altruistic ventures do it).

If middle-class intellectuals really wanted it enough, theoretically they could crowdsource the cash.

I think intuitively, this seems like clearly a tempting deal.

I'd be curious if this would be a crazy proposition, or if this is just not happening due to coordination failures.

Admittingly, it might seem pretty weird to use charitable/foundation dollars on "Buying lots of Twitter" instead of direct aid, but the path to impact is pretty clear.


Facebook Thread

One futarchy/prediction market/coordination idea I have is to find some local governments and see if we could help them out by incorporating some of the relevant techniques.

This could be neat if it could be done as a side project. Right now effective altruists/rationalists don't actually have many great examples of side projects, and historically, "the spare time of particularly enthusiastic members of a jurisdiction" has been a major factor in improving governments.

Berkeley and London seem like natural choices given the communities there. I imagine it could even be better if there were some government somewhere in the world that was just unusually amenable to both innovative techniques, and to external help with them.

Given that EAs/rationalists care so much about global coordination, getting concrete experience improving government systems could be interesting practice.

There's so much theoretical discussion of coordination and government mistakes on LessWrong, but very little discussion of practical experience implementing these ideas into action.

(This clearly falls into the Institutional Decision Making camp)

Facebook Thread

On AGI (Artificial General Intelligence):

I have a bunch of friends/colleagues who are either trying to slow AGI down (by stopping arms races) or align it before it's made (and would much prefer it be slowed down).

Then I have several friends who are actively working to *speed up* AGI development. (Normally just regular AI, but often specifically AGI)[1]

Then there are several people who are apparently trying to align AGI, but who are also effectively speeding it up, but they claim that the trade-off is probably worth it (to highly varying degrees of plausibility, in my rough opinion).

In general, people seem surprisingly chill about this mixture? My impression is that people are highly incentivized to not upset people, and this has led to this strange situation where people are clearly pushing in opposite directions on arguably the most crucial problem today, but it's all really nonchalant.

[1] To be clear, I don't think I have any EA friends in this bucket. But some are clearly EA-adjacent.

More discussion here: https://www.facebook.com/ozzie.gooen/posts/10165732991305363

There seem to be several longtermist academics who plan to spend the next few years (at least) investigating the psychology of getting the public to care about existential risks.
 

This is nice, but I feel like what we really could use are marketers, not academics. Those are the people companies use for this sort of work. It's somewhat unusual that marketing isn't much of a respected academic field, but it's definitely a highly respected organizational one.

Around discussions of AI & Forecasting, there seems to be some assumption like:

1. Right now, humans are better than AIs at judgemental forecasting.
2. When humans are better than AIs at forecasting, AIs are useless.
3. At some point, AIs will be better than humans at forecasting.
4. At that point, when it comes to forecasting, humans will be useless.

This comes from a lot of discussion and some research comparing "humans" to "AIs" in forecasting tournaments.

As you might expect, I think this model is incredibly naive. To me, it's asking questions like,
"Are AIs better than humans at writing code?"
"Are AIs better than humans at trading stocks?"
"Are AIs better than humans at doing operations work?"

I think it should be very clear that there's a huge period, in each cluster, where it makes sense for humans and AIs to overlap. "Forecasting" is not one homogeneous and singular activity, and neither is programming, stock trading, or doing ops. There's no clear line for automating "forecasting" - there are instead a very long list of different skills one could automate, with a long tail of tasks that would get increasingly expensive to automate. 

Autonomous driving is another similar example. There's a very long road between "helping drivers with driver-assist features" and "complete level-5 automation, to the extent that almost no human are no longer driving for work purposes."

A much better model is a more nuanced one. Break things down into smaller chunks, and figure out where and how AIs could best augment or replace humans at each of those. Or just spend a lot of time working with human forecasting teams to augment parts of their workflows.

I think that the phrase ["unaligned" AI] is too vague for a lot of safety research work.

I prefer keywords like:
- scheming 
- naive
- deceptive
- overconfident
- uncooperative

I'm happy that the phrase "scheming" seems to have become popular recently, that's an issue that seems fairly specific to me. I have a much easier time imagining preventing an AI from successfully (intentionally) scheming than I do preventing it from being "unaligned."

When discussing forecasting systems, sometimes I get asked,

“If we were to have much more powerful forecasting systems, what, specifically, would we use them for?”

The obvious answer is,

“We’d first use them to help us figure out what to use them for”

Or,

“Powerful forecasting systems would be used, at first, to figure out what to use powerful forecasting systems on”

For example,

1. We make a list of 10,000 potential government forecasting projects.
2. For each, we will have a later evaluation for “how valuable/successful was this project?”.
3. We then open forecasting questions for each potential project. Like, “If we were to run forecasting project #8374, how successful would it be?”
4. We take the top results and enact them.

Stated differently,

1.  Forecasting is part of general-purpose collective reasoning.
2. Prioritization of forecasting requires collective reasoning.
3. So, forecasting can be used to prioritize forecasting.

I think a lot of people find this meta and counterintuitive at first, but it seems pretty obvious to me.

All that said, I can’t be sure things will play out like this. In practice, the “best thing to use forecasting on” might be obvious enough such that we don’t need to do costly prioritization work first. For example, the community isn’t currently doing much of this meta stuff around Metaculus. I think this is a bit mistaken, but not incredibly so.

Facebook Thread

I’m sort of hoping that 15 years from now, a whole lot of common debates quickly get reduced to debates about prediction setups.

“So, I think that this plan will create a boom for the United States manufacturing sector.”

“But the prediction markets say it will actually lead to a net decrease. How do you square that?”

“Oh, well, I think that those specific questions don’t have enough predictions to be considered highly accurate.”

“Really? They have a robustness score of 2.5. Do you think there’s a mistake in the general robustness algorithm?”

—-

Perhaps 10 years later, people won’t make any grand statements that disagree with prediction setups.

(Note that this would require dramatically improved prediction setups! On that note, we could use more smart people working in this!)

Facebook Thread

I occasionally hear implications that cyber + AI + rogue human hackers will cause mass devastation, in ways that roughly match "lots of cyberattacks happening all over." I'm skeptical of this causing over $1T/year in damages (for over 5 years, pre-TAI), and definitely of it causing an existential disaster.

There are some much more narrow situations that might be more X-risk-relevant, like [A rogue AI exfiltrates itself] or [China uses cyber weapons to dominate the US and create a singleton], but I think these are so narrow they should really be identified individually and called out. If we're worried about them, I'd expect we'd want to take very different actions then to broadly reduce cyber risks. 

I'm worried that some smart+influential folks are worried about the narrow risks, but then there's various confusion, and soon we have EAs getting scared and vocal about the broader risks. 

Some more discussion in this Facebook Post.

Here's the broader comment against cyber + AI + rogue human hacker risks, or maybe even a lot of cyber + AI + nation state risks. 

Note: This was written quickly, and I'm really not a specialist/expert here. 

1. There's easily $10T of market cap of tech companies that would be dramatically reduced if AI systems could invalidate common security measures. This means a lot of incentive to prevent this.

2. AI agents could oversee phone calls and video calls, and monitor other conversations, and raise flags about potential risks. There's already work here, there could be a lot more.

3. If LLMs could detect security vulnerabilities, this might be a fairly standardized and somewhat repeatable process, and actors with more money could have a big advantage. If person A spend $10M using GPT5 to discover 0-days, they'd generally find a subset compared to person B, who spends $100M. This could mean that governments and corporations would have a large advantage. They could do such investigation during the pre-release of software, and have ongoing security checks as new models are released. Or, companies would find bugs before attackers would. (There is a different question of whether the bug is cost-efficient to fix).

4. The way to do a ton of damage with LLMs and cyber is to develop offensive capabilities in-house, then release a bunch of them at once in a planned massive attack. In comparison, I'd expect that many online attackers using LLMs wouldn't be very coordinated or patient. I think that attackers are already using LLMs somewhat, and would expect this to scale gradually, providing defenders a lot of time and experience.

5. AI code generation is arguably improving quickly. This could allow us to build much more secure software, and to add security-critical features.

6. If the state of cyber-defense is bad enough, groups like the NSA might use it to identify and stop would-be attackers. It could be tricky to have a world where it's both difficult to protect key data, but also, it's easy to remain anonymous when going after other's data. Similarly, if a lot of the online finance world is hackable, then potential hackers might not have a way to store potential hacking earnings, so could be less motivated. It just seems tough to fully imagine a world where many decentralized actors carry out attacks that completely cripple the economy.

7. Cybersecurity has a lot of very smart people and security companies. Perhaps not enough, but I'd expect these people could see threats coming and respond decently.

8. Very arguably, a lot of our infrastructure is fairly insecure, in large part because it's just not attacked that much, and when it is, it doesn't cause all too much damage. Companies historically have skimped on security because the costs weren't prohibitive. If cyberattacks get much worse, there's likely a backlog of easy wins, once companies actually get motivated to make fixes.

9. I think around our social circles, those worried about AI and cybersecurity generally talk about it far more than those not worried about it. I think this is one of a few biases that might make things seem scarier than they actually are.

10. Some companies like Apple of gotten good at rolling out security updates fairly quickly. In theory, an important security update to iPhones could reach 50% penetration in a day or so. These systems can improve further.

11. I think we have yet to see the markets show worry about cyber-risk. Valuations of tech companies are very high, cyber-risk doesn't seem like a major factor when discussing tech valuations. Companies can get cyber-insurance - I think the rates have been going up, but not exponentially.

12. Arguably, there's many trillions of dollars being held to by billionaires and others that they don't know what to do with. If something like this actually causes 50%+ global wealth to drop, it would be an enticing avenue for such money to go. Basically, we do have large reserves to spend, if the EV is positive enough, as a planet.

13. In worlds with much better AI, many AI companies (and others) will be a lot richer, and be motivated to keep the game going.

14. Very obviously, if there's 10T+ at stake, this would be a great opportunity for new security companies and products to enter the market.

15. Again, if there's 10T+ at stake, I'd assume that people could change practices a lot to use more secure devices. In theory all professionals could change to one of a few locked-down phones and computers.

16. The main scary actors potentially behind AI + Cyber would be nation states and rogue AIs. But nation-states have traditionally been hesitant to make these (meaning $1T+ damage) attacks outside of wartime, for similar reasons that they are hesitant to do military attacks outside wartime.

17. I believe that the US leads on cyber now. The US definitely leads on income. More cyber/hacking abilities would likely be used heavily by the US state. So, if they become much more powerful, the NSA/CIA might become far better at using cyber attacks to go after other potential international attackers. US citizens might have a hard time being private and secure, but so would would-be attackers. Cyber-crime becomes far less profitable if the attackers themselves can preserve their own privacy and security. There are only 8 Billion people in the world, so in theory it might be possible to oversee everyone with a risk of doing damage (maybe 1-10 million people)? Another way of putting this is that better cyber offense could directly lead to more surveillance by the US department. (This obviously has some other downsides, like US totalitarian control, but that is a very different risk) 

I wonder if some of the worry on AI + Cyber is akin to the "sleepwalking fallacy". Basically, if AI + Cyber becomes a massive problem, I think we should expect that there will be correspondingly massive resources spent then trying to fix it. I think that many people (but not all!) worried about this topic aren't really imagining what $1-10T of decently-effective resources spent on defense would do.

I think that AI + Cyber could be critical threat vector for malicious and powerful AIs in the case of AI takeover. I also could easily see it doing $10-$100B/year of damage in the next few years. But I'm having trouble picturing it doing $10T/year of damage in the next few years, if controlled by humans.

Around prediction infrastructure and information, I find that a lot of smart people make some weird (to me) claims. Like:

1. If a prediction didn't clearly change a specific major decision, it was worthless.
2. Politicians don't pay attention to prediction applications / related sources, so these sources are useless.

There are definitely ways to steelman these, but I think on the face they represent oversimplified models of how information leads to changes.

I'll introduce a different model, which I think is much more sensible:

1. Whenever some party advocates for belief P, they apply some pressure for that belief to those who notice this advocacy.
2. This pressure trickles down, often into a web of resulting beliefs that are difficult to trace.
3. People both decide what decisions to consider, and what choices to make, based on their beliefs.

For any agent having an important belief P, this is expected to have been influenced by the beliefs of those that they pay attention to. One can model this with social networks and graphs.

Generally, introducing more correct beliefs, and providing more support to them in directions where important decisions happen, is expected to make those decisions better. This often is not straightforward, but I think we can make decent and simple graphical models of how said beliefs propagate.

Decisions aren't typically made all-at-once. Often they're very messy. Beliefs are formed over time, and people randomly decide what questions to pay attention to or what decisions to even consider. Information changes the decisions one chooses to make, not just the outcomes of these decisions.

For example - take accounting. A business leader might look at their monthly figures without any specific decisions in mind. But if they see something that surprises them, they might investigate further, and eventually change something important.  

This isn't at all to say "all information sources are equally useful" or "we can't say anything about what information is valuable".

But rather, more like,

"(directionally-correct) Information is useful on a spectrum. The more pressure it can excerpt on decision-relevant beliefs of people with power, the better."

Some AI questions/takes I’ve been thinking about:
1. I hear people confidently predicting that we’re likely to get catastrophic alignment failures, even if things go well up to ~GPT7 or so. But if we get to GPT-7, I assume we could sort of ask it, “Would taking this next step, have a large chance of failing?“. Basically, I’m not sure if it’s possible for an incredibly smart organization to “sleepwalk into oblivion”. Likewise, I’d expect trade and arms races to get a lot nicer/safer, if we could make it a few levels deeper without catastrophe. (Note: This is one reason I like advanced forecasting tech)

2. I get the impression that lots of EAs are kind of assuming that, if alignment issues don’t kill us quickly, 1-2 AI companies/orgs will create decisive strategic advantages, in predictable ways, and basically control the world shortly afterwards. I think this is a possibility, but would flag that right now, probably 99.9% of the world’s power doesn’t want this to happen (basically, anyone who’s not at the top of OpenAI/Anthropic/the next main lab). It seems to me like these groups would have to be incredibly incompetent to just let one org predictably control the world, within 2-20 years. This both means that I find this scenario unlikely, but also, almost every single person in the world should be an ally in helping EAs make sure these scenarios don’t happen.

3. Related to #2, I still get the impression that it’s far easier to make a case of, “Let’s not let one organization, commercial or government, get a complete monopoly on global power, using AI”, then, “AI alignment issues are likely to kill us all.” And a lot of the solutions to the former also seem like they should help the latter.

Epistemic status: I feel positive about this, but note I'm kinda biased (I know a few of the people involved, work directly with Nuno, who was funded)

ACX Grants just announced.~$1.5 Million, from a few donors that included Vitalik.

https://astralcodexten.substack.com/p/acx-grants-results

Quick thoughts:

• In comparison to the LTFF, I think the average grant is more generically exciting, but less effective altruist focused. (As expected)
• Lots of tiny grants (<$10k), $150k is the largest one.
• These rapid grant programs really seem great and I look forward to them being scaled up.
• That said, the next big bottleneck (which is already a bottleneck) is funding for established groups. These rapid grants get things off the ground, but many will need long-standing support and scale.
• Scott seems to have done a pretty strong job researching these groups, and also has had access to a good network of advisors. I guess it's no surprise; he seems really good at "doing a lot of reading and writing", and he has an established peer group now.
• I'm really curious how/if these projects will be monitored. At some point, I think more personnel would be valuable.
• This grant program is kind of a way to "scale up" Astral Codex Ten. Like, instead of hiring people directly, he can fund them this way.
• I'm curious if he can scale up 10x or 1000x, we could really use more strong/trusted grantmakers. It's especially promising if he gets non-EA money. :)

On specific grants:

• A few forecasters got grants, including $10k for Nuño Sempere Lopez Hidalgo for work on Metaforecast. $5k for Nathan Young to write forecasting questions.
• $17.5k for 1DaySooner/Rethink Priorities to do surveys to advance human challenge trials.
• $40k seed money to Spencer Greenberg to "produce rapid replications of high-impact social science papers". Seems neat, I'm curious how far $40k alone could go though.
• A bunch of biosafety grants. I like this topic, seems tractable.
• $40k for land value tax work.
• $20k for a "Chaotic Evil" prediction market. This will be interesting to watch, hopefully won't cause net harm.
• $50k for the Good Science Project, to "improve science funding in the US". I think science funding globally is really broken, so this warms my heart.
•  Lots of other neat things, I suggest just reading directly.

I like the idea of AI Engineer Unions.

Some recent tech unions, like the one in Google, have been pushing more for moral reforms than for payment changes.

Likewise, a bunch of AI engineers could use collective bargaining to help ensure that safety measures get more attention, in AI labs.

There are definitely net-negative unions out there too, so it would need to be done delicately. 

In theory there could be some unions that span multiple organizations. That way one org couldn't easily "fire all of their union staff" and hope that recruiting others would be trivial. 

Really, there aren't too many AI engineers, and these people have a ton of power, so they could be a highly advantaged place to make a union.

I made a quick Manifold Market for estimating my counterfactual impact from 2023-2030. 

One one hand, this seems kind of uncomfortable - on the other, I'd really like to feel more comfortable with precise and public estimates of this sort of thing.

Feel free to bet!

Still need to make progress on the best resolution criteria. 

 

The following things could both be true:

1) Humanity has a >80% chance of completely perishing in the next ~300 years.

2) The expected value of the future is incredibly, ridiculously, high!

The trick is that the expected value of a positive outcome could be just insanely great. Like, dramatically, incredibly, totally, better than basically anyone discusses or talks about.

Expanding to a great deal of the universe, dramatically improving our abilities to convert matter+energy to net well-being, researching strategies to expand out of the universe.

A 20%, or even a 0.002%, chance at a 10^20 outcome, is still really good.

One key question is the expectation of long-term negative[1] vs. long-term positive outcomes. I think most people are pretty sure that in expectation things are positive, but this is less clear.

So, remember:

Just because the picture of X-risks might look grim in terms of percentages, you can still be really optimistic about the future. In fact, many of the people most concerned with X-risks are those *most* optimistic about the future.

I wrote about this a while ago, here:

https://www.lesswrong.com/.../critique-my-model-the-ev-of...

[1] Humanity lasts, but creates vast worlds of suffering. "S-risks"


https://www.facebook.com/ozzie.gooen/posts/10165734005520363

Opinions on charging for professional time?

(Particularly in the nonprofit/EA sector)

I've been getting more requests recently to have calls/conversations to give advice, review documents, or be part of extended sessions on things. Most of these have been from EAs.

I find a lot of this work fairly draining. There can be surprisingly high fixed costs to having a meeting. It often takes some preparation, some arrangement (and occasional re-arrangement), and a fair bit of mix-up and change throughout the day.

My main work requires a lot of focus, so the context shifts make other tasks particularly costly.

Most professional coaches and similar charge at least $100-200 per hour for meetings. I used to find this high, but I think I'm understanding the cost more now. A 1-hour meeting at a planned time costs probably 2-3x as much time as a 1-hour task that can be done "whenever", for example, and even this latter work is significant.

Another big challenge is that I have no idea how to prioritize some of these requests. I'm sure I'm providing vastly different amounts of value in different cases, and I often can't tell.

The regular market solution is to charge for time. But in EA/nonprofits, it's often expected that a lot of this is done for free. My guess is that this is a big mistake. One issue is that people are "friends", but they are also exactly professional colleagues. It's a tricky line.

One minor downside of charging is that it can be annoying administratively. Sometimes it's tricky to get permission to make payments, so a $100 expense takes $400 of effort.

Note that I do expect that me helping the right people, in the right situations, can be very valuable and definitely worth my time. But I think on the margin, I really should scale back my work here, and I'm not sure exactly how to draw the line.

[All this isn't to say that you shouldn't still reach out! I think that often, the ones who are the most reluctant to ask for help/advice, represent the cases of the highest potential value. (The people who quickly/boldly ask for help are often overconfident). Please do feel free to ask, though it's appreciated if you give me an easy way out, and it's especially appreciated if you offer a donation in exchange, especially if you're working in an organization that can afford it.]

https://www.facebook.com/ozzie.gooen/posts/10165732727415363