Greg_Colbourn's Quick takes

Donated $180k to PauseAI (US and Global). Calling on more people to donate significant amounts. Pretty much the only way we're going to survive the next 5-10 years is by such efforts being successful. [X post]

As part of an AMA I put on X, I was asked for my "top five EA hot takes". If you'll excuse the more X-suited tone and spiciness, here they are:

1. OpenAI, Anthropic (and to a lesser extent DeepMind) were the worst cases of Unilateralists Curse of all time. EAs love to discourage enthusiastic newcomers by warning to not do "net negative" unilateralist actions (i.e. don't start new projects in case they crowd out better, more "well thought through" projects in future, with "more competent" people doing them), but nothing will ever top the monumental unilateralist curse fuck up that was supporting Big AGI in it's beginnings. 

2. AI Safety is nothing without a Pause. Too many EAs are stuck in the pre-GPT-4 paradigm of maxing research, when it'll all be for nothing unless we get a Pause first. More EAs should switch to Notkilleveryoneism/PauseAI/StopAGI. 

3. EA is too elitist. We should be triaging the world's problems like crazy, and the top 1-2% of people are more than capable of that (most jobs that need doing in EA don't require top 0.1%). 

4. EA is too PR focused - to the point where it actually backfires spectacularly and now there is lots of bad press [big example: SBF's bad character being known about but not addressed].

5. Despite all it's flaws, EA is good (and much better than the alternatives in most cases).

CSET report on AI and Compute does not acknowledge Alignment

It was disappointing to see that in this recent report by CSET, the default (mainstream) assumption that continued progress in AI capabilities is important was never questioned. Indeed, AI alignment/safety/x-risk is not mentioned once, and all the policy recommendations are to do with accelerating/maintaining the growth of AI capabilities! This coming from an org that OpenPhil has given over $50M to set up.

UK Gov consultation on AI closing in ~36 hours: https://www.gov.uk/government/publications/ai-regulation-a-pro-innovation-approach. I'm limited on time, but I dashed off a very quick 2-min response just answering the couple of questions in the "foundation models" section (stressing need for moratorium on further development, and global limits on data and compute to ensure that models larger than GPT-4 aren't trained).

Who else thinks we should be aiming for a global moratorium on AGI research at at this point? I'm considering ending every comment I make with "AGI research cessandum est", or "Furthermore, AGI research must be stopped".

Tristan Cook @ 2023-03-23T19:29 (+6)

Strong agreement that a global moratorium would be great.

I'm unsure if aiming for a global moratorium is the best thing to aim for rather than a slowing of the race-like behaviour -- maybe a relevant similar case is whether to aim directly for the abolition of factory farms or just incremental improvements in welfare standards.

This post from last year - What an actually pessimistic containment strategy looks like -  has some good discussion on the topic of slowing down AGI research.

Greg_Colbourn @ 2023-03-25T11:40 (+4)

Loudly and publicly calling for a global moratorium should have the effect of slowing down race-like behaviour, even if it is ultimately unsuccessful. We can at least buy some more time, it's not all or nothing in that sense. And more time can be used to buy yet more time, etc.

Factory farming is an interesting analogy, but the trade-off is different. You can think about whether abolitionism or welfarism has higher EV over the long term, but the stakes aren't literally the end of the world if factory farming continues to gain power for 5-15 more years (i.e. humanity won't end up in them).

The linked post is great, thanks for the reminder of it (and good to see it so high up the All Time top LW posts now). Who wants to start the institution lc talks about at the end? Who wants to devote significant resources to working on convincing AGI capabilities researchers to stop?

Isaac Dunn @ 2023-05-02T10:43 (+3)

Isn't it possible that calling for a complete stop to AI development actually counterfactually speeds up AI development?

The scenario I'm thinking of is something like:

• There's a growing anti-AI movement calling for a complete stop
• A lot of people in that movement are ignorant about AI, and about the nature AI risks
• It's therefore easy for pro-AI people to dismiss these concerns, because the reasons given for the stop are in fact wrong/bad
• Any other, well-grounded calls for AI slowdown aren't given the time of day, because they are assumed to be the same as the others
• Rather than thoughtful debate, the discourse turns into just attacking the other group

I'm not sure how exactly you're proposing to advocate for a complete stop, but my worry would be that coming across as alarmist and not being able to give compelling specific reasons that AI poses a serious existential threat would poison the well.

I think it's great that you're trying to act seriously on your beliefs, Greg, but I am worried about a dynamic like this.

Greg_Colbourn @ 2023-05-02T10:58 (+1)

Well I've articulated what I think are compelling, specific reasons, that AI poses a serious existential threat in my new post: AGI rising: why we are in a new era of acute risk and increasing public awareness, and what to do now. Hope this can positively impact the public discourse toward informed debate. (And action!)

Isaac Dunn @ 2023-05-02T11:02 (+3)

Yes, thank you for that! I'm probably going to write an object level comment there.

Greg_Colbourn @ 2023-03-25T17:36 (+4)

[Edit: I tweeted this]

Greg_Colbourn @ 2023-03-29T12:43 (+3)

Really great to see the FLI open letter with some big names attached, so soon after posting the above. Great to see some sense prevailing on this issue at a high level. This is a big step in pushing the global conversation on AGI forward toward a much needed moratorium. I'm much more hopeful than I was yesterday! But there is still a lot of work to be done in getting it to actually happen.

Greg_Colbourn @ 2023-03-24T12:32 (+2)

GPT-4 is advanced enough that it will be used to meaningfully speed up the development of GPT-5. If GPT-5 can make GPT-6 on it's own, it's game over.

Sanjay @ 2023-03-23T21:16 (+2)

I don't see how we could implement a moratorium on AGI research that does stop capabilities research but doesn't stop alignment research?

Greg_Colbourn @ 2023-03-24T11:16 (+4)

Cap the model size and sophistication somewhere near where it is now? Seems like there's easily a decade worth of alignment research that could be done on current models (and other theoretical work), which should be done before capabilities are advanced further. A moratorium would help bridge that gap. Demis Hassabis has talked about hitting the pause button as we get closer to the "grey zone". Now is the time!

Sanjay @ 2023-03-24T11:28 (+4)

A variant on your proposal could be a moratorium on training new large models (e.g. OpenAI would be forbidden from training GPT-5, for example).

• That would be more enforceable, because you need lots of compute to train a new model. I don't know how we would stop an academic thinking up new ideas on how to structure AI models better, and even if we could, it would be hard to disentangle this from alignment research.
• It would probably achieve most of what you want. For someone who's worried about short timelines, reducing the scope for the scaling hypothesis to apply is probably pretty powerful, at least in the short term

Greg_Colbourn @ 2023-03-24T12:03 (+10)

Interesting, yes such moratorium on training new LLMs could help. But we also need to make the research morally unacceptable too - I think stigmatisation of AGI capabilities research could go a long way. No one is working on human genetic enhancement or cloning, mainly because of the taboos around them. It's not like there is a lot of underground research there. (I'm thinking this is needed, because any limits on compute that are imposed could easily be got around).

Sanjay @ 2023-03-24T12:20 (+4)

A limit on compute designed to constrain OpenAI, Anthropic, or Google from training a new model sounds like a very high bar. I don't understand why that could easily be got around?

Greg_Colbourn @ 2023-03-24T12:28 (+2)

Spoofing accounts to combine multiple of them together (as in the Clippy story linked, but I'm imagining humans doing it). The kind of bending of the rules that happens when something is merely regulated but not taboo. It's not just Microsoft and Google we need to worry about. If the techniques and code are out there (open source models are not far behind cutting edge research), many actors will be trying to run them at scale.

Sean_o_h @ 2023-03-24T12:47 (+2)

These will still be massive, and massively expensive, training runs though - big operations that will constitute very big strategic decisions only available to the best-resourced actors. 

Greg_Colbourn @ 2023-04-24T20:30 (+2)

In the post-AutoGPT world, this seems like it will no longer be the case. There is enough fervour by AGI accelerationists that the required resources could be quickly amassed by crowdfunding (cf. crypto projects raising similar amounts to those needed).

Greg_Colbourn @ 2023-03-24T13:32 (+2)

Yes, but they will become increasingly cheaper. A taboo is far stronger than regulation.

Shorthand for hedging statements?

A lot of EA writing contains many hedging statements (along the lines of "I'm uncertain about this, but", "my best guess is", "this is potentially a good idea", "it might be good", "I'm tentatively going to say","I'm not confident in this assertion, but","I'm unsure of the level of support/evidence base for this" etc).

To make things more concise, perhaps [ha!] a shorthand could be developed, where (rough) probabilities are given for statements. Maybe [haha] it could take the form of a subscript with a number, with the statements bounded by apostrophes ('), except the apostrophes are also subscript. To be as minimal as possible, the numbers could be [lol] written as 9 for 0.9 of 90%, 75 for 0.75 or 75%, 05 for 0.05 or 5%, 001 for 0.001 or 0.1% etc (basically just taking the decimal probability and omitting the decimal point). Footnotes could be added for explanations where appropriate.

Maybe the statements (or numbers) could be colour coded for ease of spotting whether something is regarded as highly likely or highly unlikely, or somewhere in the middle. Although maybe all of this will disrupt the flow of reading too much?

AGI x-risk timelines: 10% chance (by year X) estimates should be the headline, not 50%.

Given the stakes involved (the whole world/future light cone), we should regard timelines of ≥10% probability of AGI in ≤10 years as crunch time, and -- given that there is already an increasingly broad consensus around this^[1] -- be treating AGI x-risk as an urgent immediate priority (not something to mull over leisurely as part of a longtermist agenda).

Of course it's not just time to AGI that is important. It's also P(doom|AGI & alignment progress). I think most people in AI Alignment would regard this as >50% given our current state of alignment knowledge and implementation^[2].

To borrow from Stuart Russell's analogy: if there was a 10% chance of aliens landing in the next 10-15 years^[3], we would be doing a lot more than we are currently doing^[4]. AGI is akin to an alien species more intelligent than us that is unlikely to share our values.

1. ^{^}

   Note that Holden Karnofsky's all-things-considered (and IMO conservative) estimate for the advent of AGI is >10% chance in (now) 14 years. Anecdotally, the majority of people I've spoke to on the current AGISF course have estimates for 10% chance of 10 years or less.

2. ^{^}

   Correct me if you think this is wrong; would be interesting to see a recent survey on this. Maybe there is more optimism factoring in extra progress before the advent of AGI.

3. ^{^}

   This is different to the original analogy, which was an email saying: "People of Earth: We will arrive on your planet in 50 years. Get ready." Say astronomers spotted something that looked like a space-craft, heading in approximately our direction, and estimated there was 10% chance that it was indeed a spacecraft heading to Earth.  

4. ^{^}

   Although perhaps we wouldn't. Maybe people would endlessly argue about whether the evidence is strong enough to declare a >10% probability. Or flatly deny it.

(EA) Hotel dedicated to events, retreats, and bootcamps in Blackpool, UK? 

I want to try and gauge what the demand for this might be. Would you be interested in holding or participating in events in such a place? Or work running them? Examples of hosted events could be: workshops, conferences, unconferences, retreats, summer schools, coding/data science bootcamps, EtG accelerators, EA charity accelerators, intro to EA bootcamps, AI Safety bootcamps, etc. 

This would be next door to CEEALAR (the building is potentially coming on the market), but most likely run by a separate, but close, limited company (which would charge, and funnel profits to CEEALAR, but also subsidise use where needed). Note that being in Blackpool in a low cost building would mean that the rates charged by such a company would be significantly less than elsewhere in the UK (e.g. £300/day for use of the building: 15 bedrooms and communal space downstairs to match that capacity). Maybe think of it as Whytham Abbey, but at the other end of the Monopoly board: only 1% of the cost! (A throwback to the humble beginnings of EA?)

From the early days of the EA Hotel (when we first started hosting unconferences and workshops), I have thought that it would be good to have a building dedicated to events, bootcamps and retreats, where everyone is in and out as a block, so as to minimise overcrowding during events, and inefficiencies of usage of the building either side of them (from needing it mostly empty for the events); CEEALAR is still suffering from this with it’s event hosting. The yearly calendar could be filled up with e.g. 4 10-12 week bootcamps/study programs, punctuated by 4 1-3 week conferences or retreats in between. 

This needn't happen straight away, but if I don't get the building now, the option will be lost for years. Having it next door in the terrace means that the building can be effectively joined to CEEALAR, making logistics much easier (and another option for the building could be a further expansion of CEEALAR proper^[1]). Note that this is properly viewed as an investment to take into account a time-limited opportunity, and shouldn't be seen as fungible with donations (to CEEALAR or anything else); if nothing happens I can just sell the building again and recoup most/all of the costs (selling shouldn’t be that difficult, given property prices are rising again in the area due to a massive new development in the town centre).

1. ^{^}

   CEEALAR has already expanded once. When I bought the second building it also wasn’t ideal timing, but it never is; I didn’t want to lose option value.

[Half-baked global health idea based on a conversation with my doctor: earlier cholesterol checks and prescription of statins]

I've recently found out that I've got high (bad) cholesterol, and have been prescribed statins. What surprised me was that my doctor said that they normally wait until the patient has a 10% chance of heart attack or stroke in the next 10 years before they do anything(!) This seems crazy in light of the amount of resources put into preventing things with a similar (or lower) risk profiles, such as Covid, or road traffic accidents. Would reducing that to, say 5%* across the board (i.e. worldwide), be a low hanging fruit? Say by adjusting things set at a high level. Or have I just got this totally wrong? (I've done ~zero research, apart from searching givewell.org for "statins", from which I didn't find anything relevant).

*my risk is currently at 5%, and I was pro-active about getting my blood tested.

My petition to UK Parliament: Seek a global moratorium on development of AI technology due to extinction risk. If you agree, please share and sign (open to UK citizens). Let's build momentum ahead of Rishi Sunak's AI summit! (Tweet)

CEEALAR is hiring for a full-time Operations Manager (again), please share with anyone you think may be interested: https://ceealar.org/job-operations-manager/ 

Blackpool, UK. To start mid-late September. £31,286 – £35,457 per year (40 hours a week). Applications due by 31st August.

CEEALAR is hiring for a full-time Community Manager, please share with anyone you think may be interested: https://ceealar.org/job-community-manager/

To start mid-late September. £31,286 – £35,457 per year (full time, 40 hours a week).

CEEALAR is hiring

Job: Operations Manager – We are seeking a part time Operations Manager to begin late September. Deadline to apply: 16th August.

Expression of Interest: Executive Director – We are hoping to hire an Executive Director to start around November. While we await funding, we are seeking expressions of interest in the role. Tentative deadline for expressing interest: end of September.

Voluntary role: Trustee – We are seeking an additional trustee for CEEALAR. Applications will close on 30th September.

CEEALAR is hiring for a full-time Operations Manager, please share with anyone you think may be interested: https://ceealar.org/job-operations-manager

To start mid-late February. £31,286 – £35,457 per year (full time, 40 hours a week).

CEEALAR is hiring for a full-time Operations Manager, please share with anyone you think may be interested: https://ceealar.org/job-operations-manager/ Blackpool, UK. 

To start mid-late July. £31,286 – £35,457 per year (40 hours a week).