#194 – Defensive acceleration and how to regulate AI when you fear government (Vitalik Buterin on the 80,000 Hours Podcast)

By 80000_Hours @ 2024-07-31T20:28 (+42)

We just published an interview: Vitalik Buterin on defensive acceleration and how to regulate AI when you fear government. Listen on Spotify, watch on Youtube or click through for other audio options, the transcript, and related links. Below are the episode summary and some key excerpts.

Episode summary

Can ‘effective accelerationists’ and AI ‘doomers’ agree on a common philosophy of technology? Common sense says no. But programmer and Ethereum cofounder Vitalik Buterin showed otherwise with his essay “My techno-optimism,” which both camps agreed was basically reasonable.

Seeing his social circle divided and fighting, Vitalik hoped to write a careful synthesis of the best ideas from both the optimists and the apprehensive.

Accelerationists are right: most technologies leave us better off, the human cost of delaying further advances can be dreadful, and centralising control in government hands often ends disastrously.

But the fearful are also right: some technologies are important exceptions, AGI has an unusually high chance of being one of those, and there are options to advance AI in safer directions.

The upshot? Defensive acceleration: humanity should run boldly but also intelligently into the future — speeding up technology to get its benefits, but preferentially developing ‘defensive’ technologies that lower systemic risks, permit safe decentralisation of power, and help both individuals and countries defend themselves against aggression and domination.

What sorts of things is he talking about? In the area of disease prevention it’s most easy to see: disinfecting indoor air, rapid-turnaround vaccine platforms, and nasal spray vaccines that prevent disease transmission all make us safer against pandemics without generating any apparent new threats of their own. (And they might eliminate the common cold to boot!)

Vitalik explains how he mentally breaks down defensive technologies into four broad categories:

• Defence against big physical things like tanks.
• Defence against small physical things like diseases.
• Defence against unambiguously hostile information like fraud.
• Defence against ambiguously hostile information like possible misinformation.

The philosophy of defensive acceleration has a strong basis in history. Mountain or island countries that are hard to invade, like Switzerland or Britain, tend to have more individual freedom and higher quality of life than the Mongolian steppes — where “your entire mindset is around kill or be killed, conquer or be conquered”: a mindset Vitalik calls “the breeding ground for dystopian governance.”

Defensive acceleration arguably goes back to ancient China, where the Mohists focused on helping cities build better walls and fortifications, an approach that really did reduce the toll of violent invasion, until progress in offensive technologies of siege warfare allowed them to be overcome.

In addition to all of that, host Rob Wiblin and Vitalik discuss:

• AI regulation disagreements being less about AI in particular, and more whether you’re typically more scared of anarchy or totalitarianism.
• Vitalik’s updated p(doom).
• Whether the social impact of blockchain and crypto has been a disappointment.
• Whether humans can merge with AI, and if that’s even desirable.
• The most valuable defensive technologies to accelerate.
• How to trustlessly identify what everyone will agree is misinformation
• Whether AGI is offence-dominant or defence-dominant.
• Vitalik’s updated take on effective altruism.
• Plenty more.

Producer and editor: Keiran Harris
Audio engineering team: Ben Cordell, Simon Monsour, Milo McGuire, and Dominic Armstrong
Transcriptions: Katy Moore

Highlights

Vitalik's "d/acc" alternative

Rob Wiblin: The alternative to negativity about technology and effective accelerationism — perhaps a Panglossian view of technology that you lay out — you call “d/acc”: with the “d” variously standing for defensive, decentralisation, democracy, and differential. What is the d/acc philosophy or perspective on things?

Vitalik Buterin: Basically, I think it tries to be a pro-freedom and democratic kind of take on answering the question of what kinds of technologies can we make that basically push the offence/defence balance in a much more defence-favouring direction? The argument basically being that there’s a bunch of these very plausible historical examples of how, in defence-favouring environments, things that we like and that we consider utopian about governance systems are more likely to thrive.

The example I give is Switzerland, which is famous for its amazing kind of utopian, classical liberal governance, relatively speaking; the land where nobody knows who the president is. But in part it’s managed to do that because it’s protected by mountains. And the mountains have protected it while it was surrounded by Nazis for about four years during the war; it’s protected it during a whole bunch of eras previously.

And the other one was Sarah Paine’s theory of continental versus maritime powers: basically the idea that if you’re a power that is an island and that goes by sea — the British Empire is one example of this — then you’re more likely to do things like valuing freedom, being democratic, being pro-foreigner, being open-minded, being interested in trade. Versus if you are on the Mongolian steppes, then your entire mindset is around kill or be killed, conquer or be conquered, be on the top or be on the bottom. And that sort of thing is the breeding ground for basically everything that all of us consider to be dystopian governance. If you want more utopian governance and less dystopian governance, then find ways to basically change the landscape, to try to make the world look more like mountains and rivers and less like the Mongolian steppes.

And then I go into four big categories of technology, where I split it up into the world of bits and the world of atoms. And in the world of atoms, I have macro scale and micro scale. Macro scale is what we traditionally think of as being defence. Though one of the things I point out is you can think of that defence in a purely military context. Think about how, for example, in Ukraine, I think the one theatre of the war that Ukraine has been winning the hardest is naval. They don’t have a navy, but they’ve managed to totally destroy a quarter of the Black Sea Fleet very cheaply.

You could ask, well, if you accelerate defence, and you make every island impossible to attack, then maybe that’s good. But then I also kind of caution against it — in the sense that, if you start working on military technology, it’s just so easy for it to have unintended consequences. You know, you get into the space because you’re motivated by a war in Ukraine, and you have a particular perspective on that. But then a year later something completely different is happening in Gaza, right? And who knows what might be happening five years from now. I’m very sceptical of this idea that you identify one particular player, and you trust the idea that that player is going to continue to be good, and is also going to continue to be dominant.

But I talk there about also just basically survival and resilience technologies. A good example of this is Starlink. Starlink basically allows you to stay connected with much less reliance on physical infrastructure. So the question is, can we make the Starlink of electricity? Can we get to a world where every home and village actually has independent solar power? Can you have the Starlink of food and have a much stronger capacity for independent food production? Can you do that for vaccines, potentially?

The argument there is that if you look at the stats or the projections for where the deaths from say a nuclear war would come, basically everyone agrees that in a serious nuclear war, the bulk of the deaths would not come from literal firebombs and radiation; they would come from supply chain disruption. And if you could fix supply chain disruption, then suddenly you’ve made a lot of things more livable, right? So that’s a large-scale physical defence.

Biodefence

Vitalik Buterin: I also talk about micro-scale physical defence, which is basically biodefence. So in biodefence, in a sense we’ve been through this: you know, we’ve had COVID, and we’ve had various countries’ various different attempts to deal with COVID. That’s been, in a sense, in some ways, a kind of success in terms of boosting a lot of technology.

But in a much larger sense, it’s also been a missed opportunity. Basically, the challenge is that I feel like around 2022… I mean, realistically, if you had to pin an exact date for when COVID as a media event became over, it probably just would be February 24. You know, the media can only think about one very bad thing at a time, right? And basically, a lot of people were sick and tired of lockdowns. I mean, they wanted to go back to doing just regular human socialising, have kids in schools, actually be able to have regular lives again.

And I think it’s totally legitimate to value those things so much you’re willing to take percentage chances of death for it. But at the same time, I feel like people’s desire to stop thinking about the problem just went so far that now in 2023 and 2024, we’re just neglecting really basic things. Like the vaccine programmes: huge success, delivered vaccines way more quickly than anyone was expecting. Where are they now? It just kind of stalled. If we look at indoor air filtering, everyone in theoryland agrees that it’s cool and that it’s important. And like every room, including this room, should have HEPA or UVC at some point in the future. But where’s the actual effort to make that happen everywhere?

Basically, there’s just so many things that require zero authoritarianism and maybe at most $5 billion of government money, and they’re not happening. If we can just put some more extra intentional effort into getting some of those technologies ready, then we’d have a world that’s much more protected against diseases. And potentially things like bioweapons, you could imagine a future even if someone releases an airborne super plague, there’s just lots of infrastructure in place that just makes that much less of an event and much easier to respond to.

Maybe I could go through the happy story of what that might look like. So imagine someone releases a super plague — let’s say 45% mortality rate, R0 of 18, spreads around a lot, has a long incubation period. Let’s give it all the worst. We’ll give it all the worst stats. And then what happens today? Well, it just spreads around. And by the time anyone even starts realising what’s going on and thinking about how to respond to it, it’s already halfway across the world and it’s in every major city.

So now let’s go and kind of shift over our view to the positive vision. Step one: we have much better early detection. What does early detection mean? There is wastewater surveillance, so you can check wastewater and basically try to look for signs of unusual pathogens. Then there is basically open-source intelligence on social media: you can analyse Twitter and you can basically find spikes in people reporting themselves not feeling well. You can do all kinds of things, right? With good OSINT [open source intelligence] we might have plausibly been able to detect COVID maybe even like a month before we actually did.

The other thing is, if it’s done in a way that depends on very open-source infrastructure available to anyone, there’s lots of people participating — both international governmental and hobbyist. You know, a single government would not even be able to hide it if it’s starting to happen in their country, right?

How much do people actually disagree?

Rob Wiblin: A lot of things have bothered me about this debate, but one that has bothered me in particular is you went on this other show, Bankless — it’s a good podcast if people haven’t heard of it — but the debate has gotten a little bit sandwiched into the idea that some people are pro-tech and some people are anti-tech. And I think literally on that show, they said, “There’s the e/acc folks who are pro-AI and pro-technology, and then there’s effective altruism, which is anti-technology.” I think one of the hosts literally said that. I mean, they probably hadn’t heard about effective altruism before, and this is kind of all that they’d heard. And basically the thumbnail version was effective altruists hate technology. Which is extraordinary. It’s like I’m in a parallel world.

Vitalik Buterin: Yeah. I mean, it’s extraordinary from the point of view of even like 2020. Remember when Scott Alexander got doxxed by The New York Times? Remember what the vibes were? I think the people who were EA and the people who were e/acc were totally on the same team, and basically the people who kind of were perceived to be anti-tech are like the lefty cancel culture, like woke social justice types or whatever you call them, and everyone is united against them. And just like, if you’re an e/acc and you think EAs are anti technologies, think back even three years and remember what they said back at that particular time.

Rob Wiblin: It’s incredible. It would be really worth clarifying that. I mean, there are people who are anti-technology for sure. You’re mentioning degrowthers: people who just actually think the world is getting worse because of technology, and if we just continue on almost any plausible path, it’s going to get worse and worse. But all of the people we’re talking about in this debate, they all want, I think, all good and useful technologies — which is many of them — to be invented in time.

The debate is such a narrow one. It’s about whether it really matters, whether the ordering is super important. Like, do we have to work on A before B because we need A to make B safe, or does it not really matter? And we should just work on A or B or C and not be too fussy because the ordering isn’t that important? But ultimately, everyone wants A, B and C eventually.

Vitalik Buterin: Yeah. I think if I had to defend the case that the debate is not narrow, and the debate really is deep and fundamental and hits at the most important questions, I would say that the infrastructure to build the… To actually execute on the kind of pausing that EAs want probably requires a very high level of things that we would call global government. And that infrastructure, once it exists, would absolutely be used to prevent all kinds of technologies, including things that, for example, traditionally pro-tech people would be fans of, but degrowth people would be very against.

It’s like step one: you’re banning stuff around like just a little bit of stuff around superintelligence. And it’s like, OK, now we’ve agreed that it’s possible to go too far. Well, great, let’s talk about genetically engineering humans to increase our intelligence. And that’s the sort of thing where actually part of my post was being explicitly in favour of things like that, and saying we’ve gotta accelerate humans and make ourselves stronger because that is key to the happy human future.

But then there’s a lot of people that don’t feel that way. And then you imagine things expanding and expanding, and you basically might actually get the sort of world-government-enforced degrowth, right? So the question is, does that slippery slope exist? And does even building the infrastructure that’s needed to prevent this one thing… Which realistically is a very profitable thing: if you build something that’s one step below “superintelligence that’s going to kill everyone,” you’ve made an amazing product and you can make trillions of dollars. Or if you’re a country, you might be able to take over the world.

And then the kind of global political infrastructure needed to prevent people from doing that is going to need to be pretty powerful. And that is not a narrow thing, right? Once that exists, that is a lever that exists. And once that lever exists, lots of people will try to gain control of it and seize it for all kinds of partisan ends that they’ve had already.

Rob Wiblin: The sense in which people agree is that at least everyone would agree that if we set up this organisation in order to control things, to make AI safe, and then that was used to shut down technological progress across the board, people could at least agree that that’s an undesirable side effect, rather than an unintended goal of the policy — which I guess some people might be in favour of that.

It’s interesting, you just said, “the kind of pausing AI that effective altruists are in favour of.” The crazy thing is that people who are influenced by effective altruism, or have been involved in the kind of the social scene in the past, are definitely at the forefront of groups like Pause AI who want to just basically say, the simple message is we need to pause this so that we can buy time to make it safe. They’re also involved in the companies that are building AI and in many ways have been criticised a lot for potentially pushing forward capabilities enormously. It is a very bizarre situation that a particular philosophy has led people to take seemingly almost diametrically opposed actions in some ways. And I understand that people are completely bemused and confused about that.

Distrust of authority is a big deal

Vitalik Buterin: One thing to keep in mind regarding distrust of authority is I think it’s easy to get the impression that this is a weird libertarian thing, and there’s like a small percentage of people that’s maybe concentrated in America that cares about this stuff. But in reality, if you think about it a step more abstractly, it’s a prime motivator for half of geopolitics.

If you look at, for example, the reasons why a lot of centralised US technology gets banned in a lot of countries worldwide, half the argument is that the government wants the local versions to win so they can spy on people. But the other half of the argument — and it’s often a half that’s crucial to get those bans to win politically and be accepted by people — is they’re afraid of being spied on by the US, right? There’s the level of the individual having a fear of their own government, but then there’s a fear of governments having a fear of other governments.

And I think if you frame it as, how big of a cost is it for your own government to be this super world dictator and take over everything, that might be acceptable to a lot of people. But if you frame it as, let’s roll the dice and pick a random major government from the world to have it take over everything, then guess what? Could be the US one, could be the Russian one, could be the Chinese one. If it’s the US one, prediction markets are saying it’s about 52% chance it’ll be Trump and about 35% it’ll be Biden.

So yeah, the distrust of authority, especially once you think of it not just as an individual-versus-state thing, but as countries-distrusting-each-other thing, is I think definitely a very big deal that motivates people. So if you can come up with an AI safety approach that avoids that pitfall, then you’re not just appealing to libertarians, but you’re also, I think, really appealing to huge swaths of foreigners and both government and people that really want to be a first-class part of the great 22nd-and-beyond-century future of humanity, and don’t want to be disempowered.

Rob Wiblin: This idea felt like a hopeful one to me, because in my mind, I guess I know that for myself, rogue AI is maybe my number one concern, but not that far behind it is AI enabling totalitarianism, or AI enabling really dangerous centralisation or misuse or whatever. But I guess that might not be immediately apparent to people who just read things that I write, because I tend to talk about the rogue AI more because it is somewhat higher for me.

But if everyone kind of agrees that all of these are risks, and they just disagree about that ordering — of which one is number two and which one is number one — then there’s actually maybe a lot more agreement. There could be a lot more agreement than is immediately obvious. And if you could just get people to realise how much common ground there was, then they might fight a bit less.

Vitalik Buterin: Of course. I absolutely think that’s true. And a big part of it is just making it more clear to people that that common agreement exists. I think a lot of the time people don’t realise that it does.

And I think the other big thing is that ultimately people need a vision to be fighting for, right? Like, if all that you’re doing is saying, let’s delay AI, let’s pause AI, let’s lock AI in a box and monopolise it, then you’re buying time. And the question is like, what are you buying time for? One of those questions is like, what is the end game of how you want the transition to some kind of superintelligence to happen? And then the other question is like, what does the world look like after that point? You know, are humans basically relegated to being video game characters? Or is there something else for us?

These are the kinds of conversations that I think are definitely really worth having. And I think people have been having a little bit in the context of sci-fi for a while, but now that things are becoming much more real, there’s more and more people having it, and I think that’s a very healthy thing.

Info defence and X's Community Notes

Vitalik Buterin: Info defence is a much more subjective thing. Info defence is about defending against threats, such as what people think of when we talk about scams, misinformation, deepfakes, fraud, like all of those kinds of things. And those are very fuzzy things. There definitely are things that any reasonable person would agree is a scam, but there definitely is a big boundary. If you talk to a Bitcoin maximalist, if you ever have any of them on 80,000 Hours, they will very proudly tell you that Ethereum is a scam and Vitalik Buterin is a scammer, right? And look, as far as misinformation goes, there’s just lots and lots of examples of people confidently declaring a topic to be misinformation and then that turning out to be totally true and them totally wrong, right?

So the way that I thought about a d/acc take on all of those topics is basically: one is defending against those things is obviously necessary and important, and we can’t kind of head-in-the-sand pretend problems don’t exist. But on the other hand, the big problem with the traditional way of dealing with those problems is basically that you end up pre-assuming an authority that knows what is true and false and good and evil, and ends up enforcing its perspectives on everyone else. And basically trying to ask the question of, what would info defence that does not make that assumption actually look like?

And Community Notes I think is one of those really good examples. I actually ended up writing a really long review of Community Notes a few months before the post on techno-optimism. And what it is is a system where you can put these notes up on someone else’s tweet that explain context or call them a liar, or explain why either what they’re saying is false, or in some cases explain why it’s true but there’s other important things to think about or whatever.

And then there is a voting mechanism by which people can vote on notes, and the notes that people vote on more favourably are the ones that actually get shown. And in particular, Community Notes has this interesting aspect to its voting mechanism where it’s not just counting votes and accepting who is the highest; it’s intentionally trying to favour notes that get high support from across the political spectrum. The way that it accomplishes this is it uses this matrix factorisation algorithm. Basically it takes like this big graph of basically which user voted on which note, and it tries to decompose it into a model that involves a small number of stats for every note and a small number of stats for every user. And it tries to find the parameters for that model that do the best possible job of describing the entire set of votes.

So the parameters that it tries to find for each note: it has two parameters for each note and two parameters for each user. And I called those parameters “helpfulness” and “polarity” for a note, and “friendliness” and “polarity” for a user. And the idea is if a note has high helpfulness, then everyone loves it; if a user has high friendliness, it loves everyone. But then polarity is like, you vote positively on something that agrees with your polarity and you vote negatively on something that disagrees with your polarity.

So basically the algorithm tries to isolate the part of the votes that are being voted positively because the note is being partisan and is in a direction that agrees with the voter, versus the votes that vote a note positively because it just has high quality. So it tries to automatically make that distinction, and basically discard agreements based on polarisation and only focus on notes being voted positively because they’re good across the spectrum.

I basically went through and I looked at what some of the highest helpfulness notes are and also what some of the highest polarity notes are in both directions. And it actually seems to do what it says it does. The notes with a crazy negative polarity are just very partisan, left-leaning stuff that accuses the right of being fascists and that sort of stuff. Then you have with positive polarity very hardline, right-leaning stuff, whether it’s complaining about trans or whatever the right topic of the day is. And then if you look at the high helpfulness notes, one of them was like someone made a picture that they claimed to be a drone show, I believe, over Mexico City. And I’m trying to remember, but I believe it was that the note just said that actually this was AI-generated or something like that. And that was interesting because it’s very useful context. It’s useful regardless of who you are. And both fans of Trump and fans of AOC and fans of Xi Jinping would agree that that’s a good note.

Rob Wiblin: Is there a common flavour that the popular Community Notes have now using this approach? Is it often just plain factual corrections?

Vitalik Buterin: A lot of the time. When I did that review, I had two examples of good helpfulness. One was that one, and then the other one was there was a tweet by Stephen King that basically said COVID is killing over 1,000 people a day. And then someone said no, the stat says that this is deaths per month. That one was interesting because it does have a partisan conclusion, right? Like it’s a fact that is inconvenient to you if you are a left-leaning COVID hawk. And it’s a fact that’s very convenient to you if you’re a COVID minimiser, right? But at the same time, the note was written in this very factual way. And you can’t argue with the facts, and it’s an incorrect thing that needs to be corrected.

Quadratic voting and funding

Vitalik Buterin: Quadratic voting is a form of voting where you can basically express not just in what direction you care about something, but also how strongly you care about something. And it uses this quadratic formula that basically says your first vote is cheap, your second vote is more expensive, your third vote is even more expensive, and so on. And that encourages you to make a number of votes that is proportional to how strongly you care about something, which is not what either regular voting or the ability to buy votes with money does.

And quadratic funding is an analogue of quadratic voting that just takes the same math and applies it to the use case of funding public goods, and basically helping a community identify which projects to fund and directing funding from a matching pool based on how many people participate.

And I created a version of quadratic funding called pairwise bounded quadratic funding. And what that does is it solves a big bug in the original quadratic funding design, which is basically that the original quadratic funding design was based on this very beautiful mathematical formula. Like, it all works and it’s perfect — but it depends on this really key assumption, which is non-collusion: that different actors are making their decisions totally independently. There’s no altruism, there’s no anti-altruism, there’s no people looking over anyone’s shoulders. There’s no people that hack to gain access to other people’s accounts. There’s no kind of equivalent of World of Warcraft multiboxing, where you’re controlling 25 shamans with the same keyboard.

And that’s, of course, an assumption that’s not true in real life. I actually have this theory that when people talk about the limits of the applicability of economics to the real world, a lot of the time people talk about, as being wrong assumptions, either perfect information or perfect rationality. And I actually think it’s true that both of those are false, but I think the falsity of both of those is overrated. I think the thing that’s underrated is this non-collusion assumption.

And yeah, when actors can collude with each other, lots of stuff breaks. And the quadratic funding actually ended up being maximally fragile against that. Basically, if you have even two participants and those two participants put in a billion dollars, then you get matching that’s proportional to the billion dollars, and they can basically squeeze the entire matching pot out, or they can squeeze 100% minus epsilon of the entire matching pot out and give it to themselves.

What pairwise bounded quadratic funding does is it basically says we will bound the amount of matching funds that a project gets by separately considering every pair of users and having a cap on how much money we give per pair of users that vote for a project. And then, you can mathematically prove that if you have an attacker, where that attacker can gain access to let’s say k identities, then the amount of money that the attacker can extract from the mechanism is bounded above by basically c x k², right? Proportional to the square of the number of accounts that they capture.

And this sort of stuff is important for quadratic funding, but I think it’s going to be super valuable for a lot of this kind of social mechanism design in general. Because there’s a lot of interest in these one-per-person proof of personhood protocols, but they’re never going to be perfect. You’re always going to get to the point where either someone’s going to get fake people past the system through AI, or someone’s just going to go off into a village in the middle of Ghana and they’re going to tell people like, “Scan your eyeballs into this thing and I’ll give you $35, but then I’m going to get your Worldcoin ID.” And then now I’ve bought an unlimited number of Worldcoin IDs for $35 each, right? And guess what? Russia’s already got lots of operatives in Africa. And if Worldcoin becomes the Twitter ID, they’re totally going to use this.

Vitalik's philosophy of half-assing everything

Vitalik Buterin: There’s these two different versions [of effective altruism]. And I guess I definitely still believe the milder internet version. I’m sure you remember my blog post about concave and convex dispositions, and how I identify with the concave side.

A lot of the time, my philosophy toward life is to kind of half-ass everything, right? Like, my philosophy toward diet is like, some people say that keto is good. Fine, I’m gonna not drink sugar anymore and I’m going to reduce my carbs. Some people say plant-based is good. OK, I’m gonna eat more veggies. Some people say intermittent fasting is good. OK, I’m gonna try to either skip breakfast or skip dinner. Some people say caloric restriction is good. Well, that one I unfortunately can’t do because my BMI for a long time was in the underweight range. I think it’s finally at the bottom end of the normal range. And some people say exercise is good. OK, I do my running, but I also don’t run every day. I have a personal floor of one 20k run a month, and then I do what I can beyond that.

So basically like half-ass every self-improvement philosophy that seems sane at the same time is like my approach. And my approach toward charity is also a kind of half-ass every charity philosophy at the same time. You know, this is just the way that I’ve operationally just always approached the world.

Rob Wiblin: Moderation in all things.

Vitalik Buterin: Exactly.

Rob Wiblin: I think that’s from a Scott Alexander blog, right?

Vitalik Buterin: Well, yeah. He’s got the even more fun blog post, where the top level is moderation versus extremism, and then the second level is are you moderate or extreme on that divide? And then you keep getting even further and you have weird things like you have gods with names. Like, I think [inaudible] was the name of one of the gods, which is chosen because if you write it in capital letters, it has rotational symmetry. So that was fun.

But I guess I believe that. I also still totally believe the core effective altruist ideas of even basic stuff, like scope. Like, scale is important. And a problem that affects a billion people is like 1,000 times a bigger deal than a problem that affects a million people. And the difference between a billion and a million is like the difference between a tiny light and the bright sun.

At the same time, I definitely have this kind of rationalist, old, I guess you might say this Scott Alexander / Yudkowskian mindset of like, remember the meta layer, and don’t just act as though you’re correct. Act in ways that you would find acceptable if the opposite team were acting. And that’s a thing that’s definitely informed my thinking all over the years. Those ideas have always been part of my thinking, and I feel like they’ve stood the test of time. I feel like if you look at either the SBF situation or probably even the OpenAI situation, those are not examples of people acting in ways where they’d be comfortable with their worst enemies acting the same way.

The other way I think about it is there are the two regimes. Where one regime is the regime where basically you can only do good — and I think bednets are one of those. There’s a couple of people who argue, what if they get wasted and if they pollute the rivers? But like realistically I guess that’s generally understood to be a weak criticism. Yeah, it was interesting seeing people who are normally in favour of e/acc-ing everything not being e/acc on the bednets.

But the other side of that, the other regime is the regime where it’s so easy to accidentally cause actions that cause harm, and where it’s hard to even tell whether or not the total impact of what you’re going to do is on the right side of zero. And there’s totally different moralities that apply there.