Moloch's Toolbox (2/2)

By EliezerYudkowsky @ 2017-11-06T21:34 (+9)

Previous: Moloch's Toolbox (1/2)

vii. Sticky traditions in belief-dependent Nash equilibria without common knowledge

Cecie: I could talk next about a tax system that makes it cheaper for corporations to pay for care instead of patients, and how that sets up a host of “decisionmaker is not the beneficiary” problems. But I suspect a lot of people reading this conversation understand that part already, so instead I’ll turn my attention to venture capital.

Visitor: It sounds like the “politicians” and the “voters” might be a more key issue, if the cultural translator is right about what those correspond to.

Cecie: Ah! But it turns out that venture capitalists and startups can be seen as a simpler version of voters and politicians, so it’s better to consider entrepreneurs first.

Besides, at this point I imagine the Visitor is wondering, “Why can’t anyone make any money by saving those babies? Doesn’t your society have a profit incentive that fixes this?”

Visitor: Actually, I don’t think that was high on my list of questions. It’s understood among my people that not every problem is one you can make a profit by fixing—persistent societal problems tend to be ones that don’t have easily capturable profits corresponding to their solution.

I mean, yes, if this was all happening on our world and it wasn’t already being addressed by the Serious People, then somebody would just mix the bleeping nutrients and sell it to the bleeping parents for bleeping money. But at this point I’ve already guessed that’s going to be illegal, or saving babies using money is going to be associated with the wrong Tower and therefore unprestigious, or your parents are using a particular kind of statistical analysis that requires baby sacrifices, or whatever.

Cecie: Hey, details matter!

Visitor: (in sad reflection) Do they? Do they really? Isn’t there some point where you just admit you can’t stop killing babies and it doesn’t really matter why?

Cecie: No. You can never say that if you want to go on being a cynical economist.

Now, there are several different kinds of molasses covering the world of startups and venture capital. It’s the tradition-bound aspects of that ecosystem that we’ll find especially interesting, since according to its own ideology, venture capitalists are supposed to chase strange new ideas that other venture capitalists don’t believe in. Walking through the simpler case of venture capital will help us understand the more complex reasons why voters and politicians are nailed into their own equilibria, underpinning the ultimate reasons why nobody can change the laws that prevent change.

Visitor: (gazing off into the distance) … I wonder if maybe there are some worlds that can’t be saved.

Cecie: Suppose it’s widely believed that the most successful entrepreneurs have red hair. If you’re an unusually smart venture capital company that realizes that, a priori, hair color doesn’t seem like it should correlate to entrepreneurial ability, you might think you could make an excess profit by finding some overlooked entrepreneur with blonde hair.

The key insight here is that venture capital is a multi-stage process. There’s the initial or pre-seed round, the seed round, the Series A, the Series B, the middle rounds, the Series C… and if the startup fails to raise money on any of those rounds before they become durably profitable, they’re dead. What this means is that the seed-round investors need to consider the probability that the company can successfully raise a Series A. If the angels invest in the seed round of a company whose entrepreneurs don’t have red hair, that company won’t be able to raise a Series A and will go bust and the angel investment will be worthless. So the angel investors need to decide where to invest, and what price to offer, based partially on their beliefs about what most Series A investors believe.

Simplicio: Ah, I’ve heard of this. It’s called a Keynesian beauty contest, where everyone tries to pick the contestant they expect everyone else to pick. A parable illustrating the massive, pointless circularity of the paper game called the stock market, where there’s no objective except to buy the pieces of paper you’ll think other people will want to buy.

Cecie: No, there are real returns on stocks—usually in the forms of buybacks and acquisitions, nowadays, since dividends are tax-disadvantaged. If the stock market has the nature of a self-fulfilling prophecy, it’s only to the extent that high stock prices directly benefit companies, by letting the company get more capital or issue bonds at lower interest. If not for the direct effect that stock prices had on company welfare, it wouldn’t matter at all to a 10-year investor what other investors believe today. If stock prices had zero effect on company welfare, you’d be happy to buy the stock that nobody else believed in, and wait for that company to have real revenues and retained assets that everyone else could see 10 years later.

Simplicio: But nobody invests on a 10-year horizon! Even pension companies invest to manage the pension manager’s bonus this year!

Visitor: Surely the recursive argument is obvious? If most managers invest with 1-year lookahead, a smarter manager can make a profit in 1 year by investing with a 2-year lookahead, and can continue to extract value until there’s no predictable change from 2-year prices to 1-year prices.

Cecie: In the entrepreneurial world, startups are killed outright, very quickly, by the equivalent of low stock prices. And for legal reasons there are no hedge funds that can adjust market prices en masse, so the recursive argument doesn’t apply. The upshot is that seed investors have a strong incentive to care about what Series A investors think. If the entrepreneurs don’t fit the stereotype of cool entrepreneurs who have red hair, you can’t make an excess return by going against the popular misapprehension, because the startup will die in the next funding round.

The key phenomenon underlying the social molasses is that there’s a self-reinforcing equilibrium of beliefs. Maybe a lot of the Series A investors think the idea of entrepreneurs needing to have red hair is objectively silly. But they expect Series B investors to believe it. So the Series A investors don’t invest in blonde-haired entrepreneurs. So the seed investors are right to believe that “Series A investors won’t invest in blonde-haired companies” even if a lot of the reason why Series A investors aren’t investing is not that they believe the stereotype but that they believe that Series B investors believe the stereotype. And from the outside, of course, all that investors can see is that most investors aren’t investing in blonde-haired entrepreneurs—which just goes to reinforce everyone’s belief that everyone else believes that red-haired entrepreneurs do better.¹⁰

Visitor: And you can’t just have everyone say those exact words aloud, in unison, and simultaneously wake up from the dream?

Simplicio: I’m afraid people don’t understand recursion as well as that would require.

Cecie: Perhaps, Simplicio, it is only that most VCs believe that most other VCs don’t understand recursion; that would have much the same effect in practice.

Simplicio: Or maybe most people are too stupid to understand recursion. Is that something you’d be able to accept, if it were true?

Cecie: Regardless, on a larger scale, what we’re seeing is an extra stickiness that results when the incentive to try an innovation requires you to believe that other people will believe the innovation will work. An equilibrium like that can be much stickier than a scenario where, if you believe that a project will succeed, you have an incentive to try it even if other people expect the project to fail.

Stereotypically, the startup world is supposed to consist of heroes producing an excess return by pursuing ideas that nobody else believes in. In reality, the multi-stage nature of venture capital makes it very easy for the field to end up pinned to traditions about whether entrepreneurs ought to have red hair—not because everyone believes it, but because everyone believes that everyone believes it.

viii. First-past-the-post and wasted votes

Visitor: Does this feed back into our primary question of why your society can’t stop itself from feeding poisonous substances to babies?

Cecie: It’s true that venture capitalists are now collectively skeptical of attempts at new drug development, but the real problem (at least for cases like this) is the enormous cost of approval and the long delays the FDA causes.¹¹ The actual reason I went into this is that by understanding venture capitalists and entrepreneurs, we can understand the more complex case of voters and politicians. Which is the key to the political equilibrium that pins down the FDA, and all the other laws that prevent anyone from doing better. Not always, but quite often, the ultimate foundations of failure trace back to the molasses covering voters and politicians.

Simplicio: I’d like to offer, throughout whatever theory follows, the alternative hypothesis that voters are in fact just fools, sheep, and knaves. I mean, you should at least be considering that possibility.

Cecie: The simplest way of understanding the analogy between venture capitalists and voters is that voters have to vote for politicians that are electable.

Visitor: Uh, what? When you write down your preference ordering on elected representatives, you need to put politicians that other voters prefer at the top of your preference ordering?

Cecie: Yes, that’s pretty much what it amounts to. In the US, at least, elections are run on what’s known as a “first-past-the-post” voting system. Whoever gets the most votes in the contest wins. People who study voting systems widely agree that first-past-the-post is among the worst voting systems—it’s provably impossible for one voting system to have all the intuitively good properties at once, but FPTP is one of the most broken.

Visitor: Why not vote to change the voting system, then?

Cecie: I’ll get to that!

There are several ways of explaining what’s wrong with FPTP, but a lovely explanation I recently encountered phrases the explanation in terms of “wasted votes”—the total number of votes that can be removed without changing the outcome.

The two classic forms of gerrymandering are cracking and packing. Let’s say the parties are Green and Orange, and the Green party is in charge of drawing the voting boundaries. As a Green, you want to draw up districts such that Green politicians win with 55% of the vote—with some room for error, but not all that much—and for Orange politicians to win with 100% of the vote.

Simplicio: Ah, so that the Orange politicians won’t need to be responsive to Orange voters because their re-election is nearly guaranteed, right?

Cecie: No, the plot is far more diabolical than that. Consider a district of 100,000 people, where a Green politician wins with 55% of the vote. When 50,001 Green voters had cast their ballots, the election was already decided, under first-past-the-post, so the next 4,999 Green votes are “wasted”—this is to be understood as a technical term, not a moral judgment—in that they don’t further change the outcome. Then 45,000 Orange votes are also “wasted,” in that they don’t change the outcome. And also, one notes, those Orange voters don’t get the representative they wanted.

In an Orange district of 100,000 where the politician wins with 100% of the vote, there are 50,000 potent Orange votes and 50,000 wasted Orange votes. In total, there are 50,000 potent Green votes, 5,000 wasted Green votes, 50,000 potent Orange votes, and 95,000 wasted Orange votes. On a larger scale, this means that you can control a majority of a state legislature with slightly more than 1/4 of the votes—just have 55% of the districts containing 55% Green voters, with everything else solid Orange.

Visitor: And then this quarter of the population rules cruelly over the remaining three-quarters, who in turn lack the weapons to rise up?

Cecie: No, the real damage is far subtler. Let’s say that Alice, Bob, and Carol have taken time off from their cryptographic shenanigans to run for political office. Alice is in the lead, followed by Bob and then by Carol. Suppose Dennis prefers Carol to Bob, and Bob to Alice. But Dennis can’t actually write “Carol > Bob > Alice” on a slip of paper that gets processed by a trivially more sophisticated voting system. Dennis is only allowed to write down one candidate’s name, and that’s his vote. Under a system where the candidate with the most votes wins, and there’s uncertainty about which of the two frontrunners might win, all votes for whoever is in third place will be wasted votes, and this fact is predictable to the voters.

Visitor: Ah, I see. That’s why you introduced your peculiar multi-stage system of venture capital, which I assume must be held in place by laws forbidding anyone else to go off and organize their own financial system differently, and observed how it creates a sticky equilibrium in which financiers must believe that other financiers will believe in a startup.

If Dennis doesn’t believe that other “voters” will believe in Carol, Dennis will vote for Bob, which makes your politics stickier than a system in which “voters” were permitted to support the people they actually liked.

Cecie: Well, you see the analogy, but I’m not sure you appreciate the true depth of the horror.

Visitor: I’m sure I don’t.

Cecie: The upshot of first-past-the-post is typically a political system dominated by exactly two parties.

Visitor: Parties?

Simplicio: Entities that tell sheep who to vote for.

Cecie: In elections that have a single winner, votes for any candidate who isn’t one of the top two choices are wasted. In a representative democracy where districts vote on representatives who vote on laws, the dynamics of the district vote are then influenced by the dynamics of the national vote. Even if a third-party candidate could win a district, they wouldn’t have anyone to work with in the legislature, and so their votes would generally be wasted.

In the absence of a way to solve a large coordination problem, there’s no way for a third party to gain marginal influence over time. Each individual who considers voting for a third-party candidate knows they’ll be wasting their vote. This also means that third parties can’t field good candidates, since potential candidates know they’d be running to lose, which is stressful and unrewarding for people with better life options. And that’s a sufficient multi-factor system to prevent strong third parties from arising. When you’re not allowed to vote for Carol, who you actually like, you’ll vote for whichever of Alice and Bob you dislike the least.

The resulting equilibrium… well, Abramowitz and Webster found that what mainly predicted voting behavior wasn’t how much the voter liked their preferred party, but how much they disliked the opposing party.¹² Essentially, the US has two major voting factions, “people who hate Red politicians” and “people who hate Blue politicians.” When the Red politicians do something that Red-haters really dislike, that gives the Blue politicians more leeway to do additional things that Red-haters mildly dislike, which can give the Red politicians more leeway of their own, and so the whole thing slides sideways.

Simplicio: Looking at the abstract of that Abramowitz and Webster paper, isn’t one of their major findings that this type of hate-based polarization has increased a great deal over the last twenty years?

Cecie: Well, yes. I don’t claim to know exactly why that happened, but I suspect the Internet had something to do with it.

In the US, the current two parties froze into place in the early twentieth century—before then, there was sometimes turnover (or threatened turnover). I suspect that the spread of radio broadcasting had something to do with the freeze. If you imagine a country in the pre-telegraph days, then it might be possible for third-party candidates to take hold in one state, then in nearby states, and so a global change starts from a local nucleus. A national radio system makes politics less local.

The Internet might have pushed this phenomenon further and caused most of politics to be about the same national issues, which in turn reinforces the Red-vs.-Blue dynamic that allows each party to sustain itself on hatred for the other.

But that’s just me trying to eyeball the phenomenon using American history—I haven’t studied it. Other countries that also have the radio and Internet and similar electoral dynamics do manage to have more than two relevant parties, possibly because of dynamics that cause the votes of third-party politicians to be less wasted.

Simplicio: Isn’t the solution here obvious, though? All of these problems are caused by voters’ willingness to compromise on their principles and accept the lesser of two evils.

Cecie: Would things be better if people chose the greater of two evils? If they acted ineffectually against that greater evil? The Nash equilibrium isn’t an illusion. Individuals would do worse by playing away from that Nash equilibrium. Wasted votes are wasted. The current system is an effective trap and the voters are trapped. They can’t just wish their way out of that trap.

There doesn’t need to be any way for good to win; and if there isn’t, the lesser evil really is the best that voters can do. Pretending otherwise may feel righteous, but it doesn’t change the equilibrium.

Visitor: Just one second. Isn’t this all window dressing, compared to the issue of whatever true ruler imposes these rules on the “voters”? Like, if you put me into an elaborate cage that gives me an electric shock each time I vote for Carol, obviously the person who really controls the system is whoever put the cage in place and determines which politicians you can vote for without electric shocks.

Simplicio: I like the way you think.

Cecie: It’s not quite true to say that the system is self-reinforcing and that the voters are the sole instrument of their own destruction. But the lack of any obvious, individual tyrant who personally decides who you’re allowed to vote for has indeed caused many voters to believe that they are in control. I mean, they don’t feel like they’re in control, but they think that “the voters” select politicians.

They aren’t able to personalize a complicated bad equilibrium as a tyrant—not like they would blame a jeweled king who was standing in the polling booth, ready to give them an electric shock if they wrote down Carol’s name.

Inspired by Allan Ginsberg’s poem Moloch, Scott Alexander once wrote of coordination failures:

Moloch is introduced as the answer to a question—C. S. Lewis’ question in Hierarchy Of Philosophers—what does it? Earth could be fair, and all men glad and wise. Instead we have prisons, smokestacks, asylums. What sphinx of cement and aluminum breaks open their skulls and eats up their imagination?

And Ginsberg answers: Moloch does it.

There’s a passage in the Principia Discordia where Malaclypse complains to the Goddess about the evils of human society. “Everyone is hurting each other, the planet is rampant with injustices, whole societies plunder groups of their own people, mothers imprison sons, children perish while brothers war.”

The Goddess answers: “What is the matter with that, if it’s what you want to do?”

Malaclypse: “But nobody wants it! Everybody hates it!”

Goddess: “Oh. Well, then stop.”

The implicit question is—if everyone hates the current system, who perpetuates it? And Ginsberg answers: “Moloch.” It’s powerful not because it’s correct—nobody literally thinks an ancient Carthaginian demon causes everything—but because thinking of the system as an agent throws into relief the degree to which the system isn’t an agent.¹³

Scott Alexander saw the face of the Enemy, and he gave it a name—thinking that perhaps that would help.

Visitor: So if you did do this to yourselves, all by yourselves with no external empire to prevent you from doing anything differently by force of arms, then why can’t you just vote to change the voting rules? No, never mind “voting”—why can’t you all just get together and change everything, period?

Cecie: It’s true that concepts like these are nontrivial to understand.

It’s not obvious to me that people couldn’t possibly understand them, if somebody worked for a while on creating diagrams and videos.

But the bigger problem is that people wouldn’t know they could trust the diagrams and videos. I suspect some of the dynamics in entrepreneur-land are there because many venture capitalists run into entrepreneurs that are smarter than them, but who still have bad startups. A venture capitalist who believes clever-sounding arguments will soon be talked into wasting a lot of money. So venture capitalists learn to distrust clever-sounding arguments because they can’t distinguish lies from truth, when they’re up against entrepreneurs who are smarter than them.

Similarly, the average politician is smarter than the average voter, so by now most voters are just accustomed to a haze of plausible-sounding arguments. It’s not that you can’t possibly explain a Nash equilibrium. It’s that there are too many people advocating changes in the system for their own reasons, who could also draw diagrams that sounded equally convincing to someone who didn’t already understand Nash equilibria. Any talk of systemic change on this level would just be lost in a haze of equally plausible-sounding-to-the-average-voter blogs, talking about how quantitative easing will cause hyperinflation.

Visitor: Maybe it’s naive of me… but I can’t help but think… that surely there must be some breaking point in this system you describe, of voting for the less bad of two awful people, where the candidates just get worse and worse over time. At some point, shouldn’t this be trumped by the “voters” just getting completely fed up? A spontaneous equilibrium-breaking, where they just didn’t vote for either of the standard lizards no matter what?

Cecie: Perhaps so! But my own cynicism can’t help but suspect that this “trumping” phenomenon of which you speak would be even worse.

Simplicio: I have a technical objection to your ascribing all these sins to first-past-the-post voting rather than, say, the personal vices of the voters. There are numerous parliamentary democracies outside the United States that practice proportional representation, where a party getting 30% of the votes gets 30% of the seats in parliament. And they don’t seem to have solved these problems.

Cecie: Omegaven does happen to be approved in Europe, however. Like, they are not in fact killing those particular babies—

Simplicio: Oh, come on! Yes, the European equivalent of the US’s FDA happens to be a bit less stupid. Lots of other things in European countries happen to be more stupid. Indeed, I’d say that in Europe you have much crazier people getting seats in parliaments, compared to the United States. The problem isn’t the voting system. The problem is the voters.

Cecie: There are indeed some voters who want stupid things, and under the European system, their voice can be heard. There are also voters who want smart things and whose voices can be heard, like in the Pirate Party in Finland. But European parliamentary systems have different problems stemming from different systemic flaws.

Proportional representation would be a good system for a legislature that needed to repeatedly vote on laws, where different legislators could form different coalitions for each vote. If instead you demand that a majority coalition “form a government” to appoint an executive, then you need to give concessions to some factions, while other factions get frozen out. I’m not necessarily saying that it would be easy to fix all the problems simultaneously. Still, I imagine that a proportionally represented legislature, combined with an executive elected at-large by Condorcet voting, might possibly be less stupid—

Simplicio: Or maybe it would just give stupid voters a louder voice. I don’t like the evil conspiracy of the press and political elites that governs my country from the shadows, but I am willing to consider the proposition that the alternative is Donald Trump. I mean, I intend to go on fighting the Conspiracy about many specific issues. But if you’re proposing a reform that puts more power into the hands of sheep not yet awakened, the results could be even worse.

Cecie: Well, I agree that the design of well-functioning political systems is hard. Singapore might be the best-governed country in the world, and their history is approximately, “Lee Kuan Yew gained very strong individual power over a small country, and unlike the hundreds of times in the history of Earth when that went horribly wrong, Lee Kuan Yew happened to know some economics.” But the Visitor asked me why we were killing babies, and I tried to answer in terms of the system that obtained in the part of the world that was actually killing those babies. You asked why Europe wasn’t a paradise since it used proportional representation, and my answer is that parliamentary systems have their own design flaws that induce a different kind of dysfunction.

Simplicio: Then if both systems are bad, how does your hypothesis have any observable consequences?

Cecie: Because different systems are bad in different ways. When you have a “crazy” new idea, whether it’s good or bad, the European parliaments will be allowed to talk about it first. Whether that’s Omegaven, basic income, gay marriage, legalized prostitution, ending the war on drugs, land value taxes, or fascist nationalism, you are more likely to find it talked about in systems of proportional representation. It also happens to be true that those governments bloat up faster because of the repeated bribes required to hold the “governing coalition” together, but that’s a different problem.

ix. The Overton window

Simplicio: I’m beginning to experience the same sort of confusion as the Visitor about your view of the world, Conventional Cynical Economist. If voters weren’t stupid, the world would look very different than it does.

If the ultimate source of stupidity were poorly designed governmental structures, then average voters would sound smarter than average politicians. I don’t think that’s actually true.

Cecie: There are deeper forms of psychological molasses that generalize beyond first-past-the-post political candidates. The still greater force locking bad political systems into place is an equilibrium of silence about policies that aren’t “serious.”

A journalist thinks that a candidate who talks about ending the War on Drugs isn’t a “serious candidate.” And the newspaper won’t cover that candidate because the newspaper itself wants to look serious… or they think voters won’t be interested because everyone knows that candidate can’t win, or something? Maybe in a US-style system, only contrarians and other people who lack the social skill of getting along with the System are voting for Carol, so Carol is uncool the same way Velcro is uncool and so are all her policies and ideas? I’m not sure exactly what the journalists are thinking subjectively, since I’m not a journalist. But if an existing politician talks about a policy outside of what journalists think is appealing to voters, the journalists think the politician has committed a gaffe, and they write about this sports blunder by the politician, and the actual voters take their cues from that. So no politician talks about things that a journalist believes it would be a blunder for a politician to talk about. The space of what it isn’t a “blunder” for a politician to talk about is conventionally termed the “Overton window.”

Simplicio: It’s all well and good to talk about complicated clever things, Cynical Economist, but what explanatory power does all this added complexity have? Why postulate politicians who believe that journalists believe that voters won’t take something seriously? Why not just say that people are sheep?

Cecie: To name a recent example from the United States, it explains how, one year, gay marriage is this taboo topic, and then all of a sudden there’s a huge upswing in everyone being allowed to talk about it for the first time and shortly afterwards it’s a done deal. If you suppose that a huge number of people really did hate gay marriage deep down, or that all the politicians mouthing off about the sanctity of marriage were engaged in a dark conspiracy, then why the sudden change?

With my more complicated model, we can say, “An increasing number of people over time thought that gay marriage was pretty much okay. But while that group didn’t have a majority, journalists modeled a gay marriage endorsement as a ‘gaffe’ or ‘unelectable’, something they’d write about in the sports-coverage overtone of a blunder by the other team—”

Simplicio: Ah, so you say it was a conspiracy by evil journalists?

Cecie: No! Those journalists weren’t consciously deciding the equilibrium. The journalists were writing “serious” articles, i.e., articles about Alice and Bob rather than Carol. The equilibrium consisted of the journalists writing sports coverage of elections, where everything is viewed through the lens of a zero-sum competition for votes between Alice’s team and Bob’s team. Viewed through that lens, the journalists thought a gay marriage endorsement would be a blunder. And if you do something that enough journalists think is a political blunder, it is a political blunder. The journalists’ sports coverage will describe you as an incompetent politician, and primates instinctively want to ally with likely winners. Which meant the equilibrium could have a sharp tipover point, without most of the actual population changing their minds sharply about gay marriage in that particular year. The support level went over a threshold where somebody tested the waters and got away with it, and journalists began to suspect it wasn’t a political blunder to support gay marriage, which let more politicians speak and get away with it, and then the change of belief about what was inside the Overton window snowballed. I think that’s what we saw.

Simplicio: Forgive me for resorting to Occam’s Razor, but is it not simpler just to say that people’s beliefs changed slowly until it reached some level where the military-industrial complex realized they couldn’t win the battle to suppress gay marriage outright, and so stopped fighting?

Cecie: In a sense, that’s not far off from what happened, except without the evil conspiracy part. We might or might not be approaching a similar tipover point about ending the War on Drugs—a long, slow, secular shift in opinion, followed by a sudden tipover point where journalists model politicians as being allowed to talk about it, which means that politicians can talk about it, and then a few years later everyone is acting like they always thought that way. At least, I hope that’s where the current trend is leading.

Simplicio: Several states have already passed laws legalizing marijuana. Why hasn’t that already broken the Overton window?

Cecie: Because voter initiatives don’t break the common belief about what it would be a “gaffe” for a serious, national-level politician to do.

Eliezer: (aside) What broke the silence about artificial general intelligence (AGI) in 2014 wasn’t Stephen Hawking writing a careful, well-considered essay about how this was a real issue. The silence only broke when Elon Musk tweeted about Nick Bostrom’s Superintelligence, and then made an off-the-cuff remark about how AGI was “summoning the demon.”

Why did that heave a rock through the Overton window, when Stephen Hawking couldn’t? Because Stephen Hawking sounded like he was trying hard to appear sober and serious, which signals that this is a subject you have to be careful not to gaffe about. And then Elon Musk was like, “Whoa, look at that apocalypse over there!!” After which there was the equivalent of journalists trying to pile on, shouting, “A gaffe! A gaffe! A… gaffe?” and finding out that, in light of recent news stories about AI and in light of Elon Musk’s good reputation, people weren’t backing them up on that gaffe thing.

Similarly, to heave a rock through the Overton window on the War on Drugs, what you need is not state propositions (although those do help) or articles in The Economist. What you need is for some “serious” politician to say, “This is dumb,” and for the journalists to pile on shouting, “A gaffe! A gaffe… a gaffe?” But it’s a grave personal risk for a politician to test whether the public atmosphere has changed enough, and even if it worked, they’d capture very little of the human benefit for themselves.

Visitor: So… if this is the key meta-level problem… then why can’t your civilization just consider and solve this entire problem on the meta level?

Cecie: Oh, I’m afraid that this entire meta-problem isn’t the sort of thing the “leading candidates” Alice and Bob talk about, so the problem itself isn’t viewed as serious. That is, journalists won’t think it’s serious. Meta-problems in general—even problems as simple as first-past-the-post versus instant runoff for particular electoral districts—are issues outside the Overton window. So the leading candidates Alice and Bob won’t talk about organizational design reform, because it would be very damaging to their careers if they visibly focused their attention on issues that journalists don’t think of as “serious.”

Visitor: Then perhaps the deeper question is, “Why does anyone listen to these ‘journalists’?” You keep attributing power to them, but you haven’t yet explained why they have that power under your equilibrium.

Cecie: People believe that other people believe what’s in the newspapers.

Well, no, that’s too optimistic. A lot of people do believe what’s in the newspapers, so long as it isn’t about a topic regarding which they have any personal knowledge or expertise. The Gell-Mann Amnesia Effect is the term for how we read the paper about subjects we know about, and it’s talking about how wet streets cause rain; and then we turn to the story about international affairs or dieting, and for some reason assume it’s more accurate.

There’s some level on which most people prefer to talk and believe within the same mental world as other people. Nowadays a lot of people believe what they read on, say, Tumblr, and hardly look at The New York Times at all. But even then they still believe that other people believe what’s in The New York Times. That’s what gives The New York Times its special power over the collective consciousness, far out of proportion to their dwindling readership or the vanishing real trust that individuals from various walks of life have in them—what’s printed in The New York Times determines what people believe other people believe.

Simplicio: Do you truly lay all the sins of humanity at the feet of all this weird recursion? Or is this just a sufficiently weird hypothesis that you find it more fun to think about than the alternatives?

Cecie: I’m not sure I’m pointing in exactly the right direction, but I feel that I’m pointing in the general direction of something that’s truly important to the Visitor’s most basic question. The Visitor keeps asking why, in some sense, on some sufficiently general level, we can’t just snap out of it. And to put it in the sort of terms you yourself might want to use, Simplicio, if we’re looking for an explanation of why we can’t just snap out of it, then it might make sense to point to a bad Nash equilibrium covering our collective consciousness and discussion. I suspect that the recursion, the dependency on what people believe other people believe, has a lot to do with making that a sticky equilibrium a la venture capital.

Eliezer: (aside) Returning to my day job: As of 2017, I pretty commonly hear from AI researchers who are worried about AGI safety, but who say that they don’t dare say anything like that aloud. You could see this as either a good sign or a very bad sign, depending on how pessimistic or optimistic you previously were about the adequacy of academic discussion.

Simplicio: But then what, on your view, is the better way?

Cecie: Again, I could pontificate about various ideas, but that’s a different and harder question than looking at the actual equilibrium that currently obtains and forces doctors to poison babies. There doesn’t have to be a better way.

x. Lower-hanging altruistic fruit and bigger problems

(The Visitor takes a deep breath. When the Visitor speaks again, it is louder.)

Visitor: Then what about your <untranslatable 17>?

Cecie: Sorry? That word didn’t come through.

Visitor: What about everyone on your entire planet who could possibly care about babies dying?

So your medical specialists are borked. From the magic-tower analogy, I assume your systems of learning are borked, and that means most of the parents whose responsibility it is to protect the child are borked. Your politicians are borked. Your voters are borked. Your planet has no Serious People who could be trusted to try alternative shoe designs, let alone lead the way on any more complex coordination problem. Your prediction markets, I suppose, are somehow borked in a way that prevents anyone from making a profit by correcting inaccurate policy forecasts… maybe they forecast wrongly bad consequences to unpopular policies, which therefore never get implemented in a way that shows up the inaccurate prediction, since you don’t have any way to test things on a smaller scale? Your economists must somehow be borked—

Cecie: It’s more that nobody ever listens to us. They pay us and then they don’t listen to us.

Visitor: —and your financial system is borked so that nobody can make a profit on saving those babies or doing anything else useful. I’m not stupid. I’ve picked up on the pattern at this point.

But what about everyone else? There are seven billion people on your planet. How is it that none of them step up to save these babies from death and brain damage? How is your entire planet failing to solve this problem?

Cecie: That… sounds like a weird question, to an Earth person.

Visitor: Whatever your problems are, surely out of seven billion human beings there have to be some who could see the problems as you’ve laid them out, who could try to rally others to the cause of saving those babies, who could do whatever it took to save them!

Even if your system declares that saving babies is only the responsibility of “doctors” or “politicians” or whoever is the Someone Else whose Problem it is, there’s no law of physics that stops someone else from walking up to the problem and accepting responsibility for it. Out of seven billion people in your world, I can’t believe that literally all of them are incapable of gathering together some friends and starting things down the path to getting a little fish oil into a baby’s nutritional mixture!

Eliezer: I think I’ll step in myself at this point. There’s one other very general conclusion we can draw from seeing this ever-growing heap of dead babies. We might say, “the inadequacy of the part implies the inadequacy of the whole”—as we’ve defined our terms, if a part of the system is inadequate in X lives saved for Y dollars, then the whole system is inadequate in X lives saved for Y dollars. Someone who is motivated and maximizing will first go after the biggest inadequacy anywhere that they think they can solve, and if they succeed, it pushes forward the adequacy frontier for the whole system. Thus, we can draw one other general conclusion from the observation that babies are still being fed soybean oil. We can conclude that everyone on the planet who is smart enough to understand this problem, and who cares about strangers’ lives, and who maximizes over their opportunities, must have something more important to do than getting started on solving it.

Visitor: (aghast) More important than saving hundreds of babies per year from dying or suffering permanent brain damage?

Eliezer: The observation stands: there must be, in fact, literally nobody on Earth who can read Wikipedia entries and understand that omega-6 and omega-3 fats are different micronutrients, who also cares and maximizes and can head up new projects, who thinks that saving a few hundred babies per year from death and permanent brain damage is the most important thing they could do with their lives.

Visitor: So you’re implying…

Eliezer: Well, mostly I’m implying that maximizing altruism is incredibly rare, especially when you also require sufficiently precise reasoning that you aren’t limited to cases where the large-scale, convincing study has already been done; and then we’re demanding the executive ability to start a new project on top of that. But yes, I’m also saying that here on Earth we have much more horrible problems to worry about.

Cecie: We’ve just been walking through a handful of lay economic concepts here, the kind whose structure I can explain in a few thousand words. If you truly perceived the world through the eyes of a conventional cynical economist, then the horrors, the abominations, the low-hanging fruits you saw unpicked would annihilate your very soul.

Visitor: …

Eliezer: And then some of us have much, much more horrible problems to worry about. Problems that take more than reading Wikipedia entries to understand, so that the pool of potential solvers is even smaller. But even just considering this particular heap of dead babies, we know from observation that this part must be true: If you imagine everyone on Earth who fits the qualifications for the dead-baby problem—enough scientific literacy to understand relevant facts about metabolic pathways, and the caring, and the maximization, and enough scrappiness to be the first one who gets started on it, meeting in a conference room to divide up Earth’s most important problems, with the first subgroup taking on the most neglected problems demanding the most specialized background knowledge, and the second taking on the second-most-incomprehensible set of problems, until the crowdedness of the previously most urgent problem decreases the marginal impact of further contributions to the point where the next-worst problem at that level of background knowledge and insight becomes attractive… and so on down the ladders of urgency inside the levels of discernment… then there must be such a long and terrible list of tasks left undone, and so few people to understand and care, that saving a few hundred babies per year from dying or suffering permanent brain damage didn’t make the list. So it has been observed, and so it must be.

Wandering Bystander: (interjecting) But I just can’t believe our planet would be that dysfunctional. Therefore, by backward chaining, I question the original observation on which you founded your inference. In particular, I’m starting to wonder whether omega-3 and omega-6 could really be such significantly different micronutrients. Maybe that’s just a crackpot diet theory that somehow made it into Wikipedia, and actually all fats are pretty much the same, so there’s nothing especially terrifying about the prospect of feeding babies exclusively fat from soybean oil instead of something more closely resembling the lipid profile of breast milk?

Eliezer: Ah, yes. I’m glad you spoke up. I’ll get to your modest proposal next.

Cross-posted to equilibriabook.com and Less Wrong. Next: Living in an Inadequate World.