A Paradox for Tiny Probabilities and Enormous Values
By Global Priorities Institute @ 2021-07-01T07:00 (+5)
This is a linkpost to https://globalprioritiesinstitute.org/wp-content/uploads/Beckstead-Thomas-A-Paradox-for-Tiny-Probabilities-and-Enormous-Values-Version-2.pdf
Abstract
We show that every theory of the value of uncertain prospects must have one of three unpalatable properties. Reckless theories recommend risking arbitrarily great gains at arbitrarily long odds for the sake of enormous potential; timid theories permit passing up arbitrarily great gains to prevent a tiny increase in risk; non-transitive theories deny the principle that, if A is better than B and B is better than C, then A must be better that C. While non-transitivity has been much discussed, we draw out the cost and benefits of recklessness and timidity when it comes to axiology, decision theory, and moral uncertainty.
Introduction
On you deathbed, God brings good news. Although, as you already knew, there's no afterlife in store, he'll give you a ticket that can be handed to the readper, good for an additional year of happy life on Earth. As you celebrate, the devil appears and asks, 'Won't you accept a small risk to get something vastly better? Trade that ticket for this one; it's good for 10 years of happy life, with probability 0.999.' You accept, and the devil hands you a new ticket. But then the devil asks again, 'Won't you accept a small risk to get something vastle better? Trade that ticket for this one: it is good for 100 years of happy life — 10 times as long — with probability 0.999^2 — just 0.1% lower.'
An hour later, you've made 30,000 trades. (The devil is a fast talker.) You find yourself with a ticket for 10^30,000 years of happy life that only works with probability 0.999^30,000, less than one chance in 10^21. Predictably, you die that very night.
Read the rest of the paper
AlexMennen @ 2022-08-17T04:23 (+4)
Timidity seems unobjectionable to me, and the arguments against it in section 3 seem unconvincing.
3.1: Marginal utility in number of lives already dropping off very steeply by 1000 seems implausible, but if we replace 1000 with a sufficiently large number, an agent with a bounded utility function would deny that these prospects keep getting better for the same (rational, imo) reasons they would eventually stop taking the devil's deals to get more years of happy life with high probability.
3.2: It seems perfectly reasonable to me to selectively create valuable things in situations in which value isn't already nearly maxed out, even if you can create valuable things much less efficiently in those situations, which is all that's going on here. Also disagree that it's strange for the value of creating lives to depend on what's happening far away to any significant degree at all; for an example that might seem more intuitive to some, what if the lives you're about to create already exist elsewhere? Wouldn't it be much better to create different ones instead?
3.3: If we replace 10^10 and 10^80 with 10^1000 and 10^8000, then I'd prefer A over B. I'm not sure what this is supposed to add that the initial example with the devil's deals for more years of happy life didn't.
3.4: Longshot bets like the "Lingering doubt" scenario are very different from longshot bets like Pascal's mugger, in ways that make them seem much more palatable to me (see 3.2). Furthermore, longshot bets as a criticism of recklessness can be seen as a finitistic stand-in for issues of nonconvergence of expected utility, which isn't a problem for timidity.
Furthermore, I think the paper seriously undersells (in section 4) how damning it is that recklessness violates prospect-outcome dominance. This implies vulnerability to Dutch books. After playing the St. Petersburg lottery, no matter what the outcome, it will only have finite value, less than the St. Petersburg lottery is worth in expectation, so if given the option, the agent with spend 1 util to try again, and replace their payout with whatever they get the next time around. They will do this no matter what the outcome is the first time, even though their prospects are no better on the second attempt.