EAs Should Use Less Bayesian Reasoning

By James Brobin @ 2026-08-02T21:06 (+52)

Edit: If I were to re-write this post, I would re-frame it as a list of reasons for why we should lower our confidence in Bayesian estimates rather than as an attack on Bayesianism itself.

This is a crosspost from my blog post.

Many EAs use extensive “bayesian reasoning.” The basic idea behind it is that you should do the following:

  1. Be willing to assign probabilities to the likelihood of anything occurring or being true.
  2. Update these probabilities when you learn new information.
  3. Act on these probabilities if they suggest that certain actions are higher in value than all other actions.

Bayesian reasoning is helpful for a lot of everyday decision-making. If you’re trying to figure out whether to take a job in Los Angeles or in New York, it makes sense to try to guess how happy you’d be in each respective city. And, if you’re trying to compare career paths, you should assign probabilities to how likely you are to succeed in them.

But, to me, EAs take this kind of reasoning too far. EAs have variously tried to predict how many future humans there will be, asked non-domain experts how likely they think a catastrophic pandemic will be, and even tried to determine whether we should work on improving the lives of people in the far future.

Probably the most common (and representative) example of this, though, is the idea of AI timelines, which means trying to predict when “AGI” will be developed. Given that we have no clue what it actually takes to develop AGI or how many breakthroughs are required, it seems pretty unreasonable to me to assign an exact year to when such an outcome would occur.

But, setting that aside, these are my five reasons EAs should use less Bayesian reasoning.

Reason #1: If you know very little about something, your guesses are completely arbitrary.

Have you ever asked a child how far away they think New York is and then heard them guess “a hundred miles?” This child’s guess is inaccurate because they don’t know very much about the relative distances of places around the world.

I think this same line of reasoning can be applied to a lot of EA’s predictions about the future. For instance, a lot of EAs make guesses about the likelihood of AI takeover, but I don't think we have enough knowledge to know when we'll develop AGI, whether we’ll develop AGI, or what the development of AGI will look like. So, given this, it seems pretty unlikely that we could come up with any kind of accurate prediction about how likely AI takeover is.

Reason #2: We're bad at making guesses in general.

Some people have undergone extensive training to be good at making forecasts over short durations of time in situations where trends can be roughly extrapolated forwards.

Most people have not undergone this training, and are, in fact, pretty bad at making guesses in general. 

Most people are bad at predicting how successful they will be, who will win elections, and how long it will take for them to finish projects.

Given this, it seems like we should expect our predictions on most things to be off by a reasonable extent.

Reason #3: If your guesses are completely arbitrary, updating won’t bring you to the correct probability.

Bayesians like to say that, if you have no clue how likely something is, you should just pick a random number and then update your beliefs from there. The problem, though, is that, if you’re updating relative to an arbitrary number, your arbitrary number might hold too much weight

For instance, if you think there’s a 10% chance that a pandemic this century will kill more than a billion people, you might be only willing to update by a single order of magnitude each time you learn new information.

Given this, if you learned that the Chinese government has decided to stop stockpiling masks, you might reduce your probability to 1%, but, if the real probability were 10^-7, you would need an overwhelming amount of information to update your beliefs to the correct probability.

Reason #4: We should expect most guesses about the future to be wrong.

People are generally familiar with the idea that we’re bad at predicting the future, but I think they fail to take seriously how significant of an issue this is.

The fact is that history has been determined by an extraordinarily complex interaction of social, political, environmental, economic, and circumstantial factors. And, as a result, historically, people were very bad at predicting the future. If you had someone in 1910 try to make predictions about how the century would go, they would probably be wrong in a vast myriad of ways. They likely wouldn’t have predicted two world wars and a cold war. They wouldn’t have guessed that we’d discover the existence of galaxies. And, they wouldn’t have been able to tell you that we’d become completely digitally interconnected. Given this, I think we should also consider our own predictions about the future to likely be very wrong.

Reason #5: If your guesses are based on other people’s guesses, you might all be wrong.

Humans experience an anchoring bias when it comes to making predictions, so if we hear someone make a prediction, we usually make ours relative to theirs. The problem with this is that, if one person makes a very prominent prediction that is completely off, everyone will be basing their prediction on that bad prediction.

I think this is particularly concerning in domains where only a few individuals have prominence, but there's very little information to go off of. If everyone is assuming those individuals know more than they do, then everyone will have very biased guesses.

Reason #6: People rarely offer extraordinarily low probabilities.

Whenever I ask someone how likely they think something is, they pretty much never give a probability less than .1% unless that something is religious in nature. Given this, it seems like people systematically over estimate low probabilities because they fail to consider probabilities such as 10^-7 or 10^-53.

Mo Putera @ 2026-08-03T05:41 (+14)

The post title somewhat confuses me since reasons 2, 4, and 5 (miscalibration, anchoring cascades, and granularity failure at the tails) are in-paradigm critiques.

I'm also confused by the link to the XPT tournament as if it illustrates the assertion that EAs take Bayesian reasoning too far, given it did in fact include domain experts and given its headline finding that the forecasts were discrepant between the groups and failed to converge after structured persuasion. Same confusion re: linked GPI paper. 

I do think many of your points are correct. The strongest argument to your post title I can think of is sparse evidence + multiple models and you're uncertain between them -> precise priors are unwarranted -> use an interval instead, possibly quite wide like [10⁻⁶, 0.5] (just to make something up) -> updating may not collapse this wide interval -> so EV-maxxing becomes undefined -> so switch to other decision criteria, e.g. maybe robustness to harms, which is less Bayesian as per your post title -> choose robustly good actions, maintain option value, build capacity etc. Which is basically what most meta interventions are about, no?

Anthony DiGiovanni 🔸 @ 2026-08-03T19:29 (+4)

-> choose robustly good actions, maintain option value, build capacity etc. Which is basically what most meta interventions are about, no?

Unfortunately I don't think there's really much of a case for "maintain option value, build capacity etc." being robustly good either, as argued here.

Mo Putera @ 2026-08-04T03:16 (+4)

Thanks Anthony. Would it be fair to interpret your unawareness series as your steelman of OP's post title, or as being relevant?

I couldn't find on a quick look what decision-making approach you would (at least provisionally) endorse instead, bracketing maybe? For my own reference later:

Bracketing says to base our decisions on those consequences we are – in a precise sense – not clueless about, “bracketing out” the others. The idea is that the effects we’re clueless about should not override the obligations given to us by the benefits we aren’t clueless about, like the immediate benefits of malaria nets to the global poor. Thus bracketing could provide action-guidance in the face of cluelessness and in particular support neartermism.

... Take Mogensen's (2020) example of deciding whether to donate to the Against Malaria Foundation (AMF) or the Make-a-Wish Foundation (MAWF). As Mogensen argues, you’re clueless about the overall effects of donating to AMF vs. MAWF, due to their highly ambiguous effects on population dynamics, economic growth, resource usage, etc.  You can come up with lots of asymmetrical effects each intervention has on total value, and you don’t have any principled way of weighing them up, but they still may swamp the immediate effects. Thus it seems that each of the available actions – Donate to AMF, Donate to MAWF, or Do Nothing – is permissible.

And yet, if you’re like me, you suspect that even an impartial consequentialist ought to choose AMF. For we aren’t clueless about the effects of our actions on the immediate beneficiaries! Restricting attention to those who would be prevented from contracting malaria by an AMF donation and the child who would be granted a wish by an MAWF donation, we can rank our actions by their expected total value: AMF  MAWF  Do Nothing. So my thought is, “Those immediately affected, who I’m not clueless about, give me a reason to Donate to AMF. It is true that, once I start accounting for more moral patients, I’ll become clueless about total value. But my cluelessness about this enlarged set of patients does not override the reasons given to me by the immediate beneficiaries. So, still, I’m required to Donate to AMF.” And that's the essence of bracketing.

I would be particularly interested in how you think meta- and/or longtermist-oriented grantmaking could be improved by your work. I have not been very impressed by the reasoning behind some of these (sometimes quite large) grants, at least on the rare occasions they've been shared publicly. 

Vasco Grilo🔸 @ 2026-08-05T19:06 (+2)

Hi Mo.

I do think many of your points are correct. The strongest argument to your post title I can think of is sparse evidence + multiple models and you're uncertain between them -> precise priors are unwarranted -> use an interval instead, possibly quite wide like [10⁻⁶, 0.5] (just to make something up) -> updating may not collapse this wide interval -> so EV-maxxing becomes undefined -> so switch to other decision criteria, e.g. maybe robustness to harms, which is less Bayesian as per your post title -> choose robustly good actions, maintain option value, build capacity etc. Which is basically what most meta interventions are about, no?

This argument works with sharp probabilities too? If the distributions for the cost-effectiveness are very wide, the expected cost-effectiveness of decreasing uncertainty or building capacity would tend to be higher than the highest expected cost-effectiveness of the interventions under evaluation, even if these are all sharp values?

I do not want to give up completeness because it follows from 3 super intuitive premises.

Behaviour norms are considered for decision trees which allow both objective probabilities and uncertain states of the world with unknown probabilities. Terminal nodes have consequences in a given domain [premise 1; unrestricted domain]. Behaviour is required to be consistent in subtrees [premise 2; dynamic consistency]. Consequentialist behaviour, by definition, reveals a consequence choice function independent of the structure of the decision tree [premise 3; consequentialism]. It implies that behaviour reveals a revealed preference ordering ["a complete, transitive, binary relation"] satisfying both the independence axiom and a novel form of surething principle.

simon @ 2026-08-06T21:21 (+1)

Bayesian inference and decision making are somewhat distinct steps. 
EV maxxing is a specific (and specifically simple) objective function that you can plug Bayesian estimates into for decision making. It removes the need to think about distributions.

I think it can be helpful to separate out the problems - eg I believe that there aren’t really any plausible alternatives to Bayesian inference (over future trajectories of the world as a function of your actions or similar) but I think there’s much more room for debate regarding the objective function. 

Mo Putera @ 2026-08-07T04:58 (+3)

Thanks for the corrections. Agree there's plenty of room for debate re: objective function, I generally think people don't take this seriously enough (Nuno Sempere's estimating value series is what I have in mind by "take it seriously"). 

Richard Ngo's Towards a Formal Scientific Epistemology sketches out the beginnings of an alternative to Bayesian epistemology in case you're interested. I am not the right person to field questions about it unfortunately, I'll just quote his intro:

In my post “Why I’m not a Bayesian”, I argued that the Bayesian approach of assigning credences to propositions with binary truth values only works in simple and restricted domains. Instead, I claimed, a better approach to epistemology is to assign degrees of truth to models of the world.

This approach is broadly inspired by science, which is the domain from which we have the most evidence about which epistemological practices allow us to solve very hard problems. We don’t currently have a complete theory of scientific epistemology, but we can identify some important differences between scientific epistemology and Bayesian epistemology. Central examples of Bayesian epistemology (such as Solomonoff induction) assume that the truth lies within the class of hypotheses being considered. Conversely, in central examples of scientific research, the truth is definitely not already under consideration: the main problem is to come up with any hypothesis that explains existing data.

Another way of putting this point: Bayesian epistemology is entirely about empirical updates, whereas science is mostly about the process of constructing new theories. In some cases, once you’ve constructed a theory, you can be confident that it’s close to the truth merely from how well it fits existing data. But scientific theories are only fully accepted after they make successful advance predictions. That’s another difference compared with Bayesian epistemology, which treats retrodictions as equivalent to predictions.

In general I think scientific epistemology is far superior as a guide for thinking about difficult problems (like AI alignment) than Bayesian epistemology. However, scientific epistemology has mostly been described informally—e.g. by Popper, Kuhn, Feyerabend, etc. Popper did attempt to formally define a metric for degrees of truth, but it wasn’t very successful. I’d like to be able to describe scientific epistemology as formally as we can describe Bayesian epistemology (and ideally to unify them in a single framework).

I interpret his follow-up essay Agents as webs of beliefs: Unifying beliefs, goals, and actions as sketching out the bigger picture.

simon @ 2026-08-07T20:51 (+1)

People in EA should definitely read more Feyerabend! (Or ask llms what Feyerabend would say about a topic etc). 

For example “a complete theory of scientific epistemology” is something he’d most likely reject even as an ideal. 

James Brobin @ 2026-08-04T17:32 (+1)

I was making in-paradigm critiques, since, if something is internally invalid, it gives us good reason to think it is also externally invalid. 

I linked to the XPT tournament just as an example of asking non-experts for predictions. I thought that pandemics were a good example since most people don't know much about pandemics at all so we should expect their guesses to be very off. I could see the argument that forecasting research has found forecasters to be better than experts so it makes sense to ask non-experts, but it's important to note that we haven't validated this over very long time periods.

I linked to the GPI paper since it seems to pretty absurd to me to try to guess how many future there will be. Even if we can come up with accurate estimates for carrying capacity for different regions of space, I have no clue how we could predict the likelihood of reaching carrying capacity in these regions of space.

In regards to your last point, I think my view is most basically that, by assigning probabilities to outcomes, we're giving ourselves excessive confidence when we often have too little knowledge to warrant the confidence, that correct decision theories should take into account our cluelessness to a much greater extent, and that people probably do Bayesian reasoning a lot worse than they think.

That said, I'm still figuring out my views on decisions theories so I probably should have researched this topic a lot more before making a post.

For your last point, I agree with you that we run into the problem of EV-maxxing being undefined, but I don't know where to go from there. It doesn't seem to make sense to me to do anything in regards to something with such a wide probability range because it seems like you'll just spend all your time chasing things that you know very little about but which suggest really high EV.

Thanks for the thoughtful comment.

titotal @ 2026-08-03T13:35 (+11)

I definitely agree that Bayesianism is overrated and overused. I do wonder if some of these pitfalls could be avoided if people started using actual Bayesian statistics, rather than this fake pop-bayes thing where people make single numerical estimates (like saying "10% chance of apocalypse") and nudge the number up and down a bit in response to news. 

Maybe I'm being a grumbly math nerd here, but outside of toy examples, actual bayesianism involves updating distributions, not single numbers.  And you need to actually run calculations using bayes formula, not guesstimate it. 

simon @ 2026-08-03T21:45 (+3)

Nice distinction!
I’d agree that pop-Bayesianism is overused while more rigorous Bayesianism is underused. 

James Brobin @ 2026-08-04T17:05 (+2)

Yeah, I think that articulates a lot of my criticism of it. It seems like people generally aren't very good at making guesses about probabilities of things in general so we shouldn't expect that we can engage in "Bayesianism" in our head and magically create really good predictions.

simon @ 2026-08-03T06:20 (+6)

Reason 2 (edit: now 3) is not a valid criticism of correctly applied Bayesian thinking. If your initial guesses are arbitrary and weak and you are aware, you have a very wide prior. So updating will bring your posterior arbitrarily close to the right answer. 
Of course it’s reasonable to criticise incorrectly applied Bayesian thinking or overconfidence in your prior, or too weak updating, or anchoring. 

James Brobin @ 2026-08-04T17:13 (+2)

Thanks for explaining that!

Guy Raveh @ 2026-08-03T15:49 (+1)

The question of whether we should expect sufficient evidence to bring us close enough to the truth still stands.

simon @ 2026-08-03T21:41 (+1)

Can you elaborate? Do you mean “should” in the sense that it possibly ought not to do that or do you mean it might not be close enough? (In the latter case it seems to be more about specific parameters in practice rather than the concept itself?)

Guy Raveh @ 2026-08-04T12:50 (+2)

I mean "do I expect to see, within the relevant time frame, enough information to make Bayesian updating with a very wrong (or just very wide) initial distribution useful rather than harmful?".

simon @ 2026-08-04T19:52 (+1)

I see, fair. 

What’s the alternative to “wide distribution stays wide?” in practice?

Separately, “very wrong and narrow prior” is a problem of course, but very much intra-Bayesian and not a criticism of Bayesian reasoning?

Guy Raveh @ 2026-08-05T14:55 (+2)

very much intra-Bayesian and not a criticism of Bayesian reasoning?

Not really. It's a criticism of a system that allows (and even encourages) you to pretend to know things when you don't.

Of course, I'm a mathematician and I think Bayesian reasoning is fundamentally correct and is useful in some contexts. Just not the contexts EA uses it for.

simon @ 2026-08-07T21:35 (+1)

Yeah I think the obvious Bayesian reply is: Just make your prior less informative if you know less? It’s fine to choose it to be close to uniform over a very wide range if you know close to nothing. 

I think the main concern is that people often use the expected value of such a wide distribution in utility maximisation. So the concerning part is the interaction between eg utilitarianism and Bayesian inference. 

Radical Empath Ismam @ 2026-08-11T03:03 (+2)

Someone should do this experiment one day. Compare the performance of people using Bayesian reasoning vs other reasoning methods, and see if there actually is an advantage to one heuristic over another.

Vasco Grilo🔸 @ 2026-08-05T19:00 (+2)

Hi James. Nice points.

Reason #6: People rarely offer extraordinarily low probabilities.

Whenever I ask someone how likely they think something is, they pretty much never give a probability less than .1% unless that something is religious in nature. Given this, it seems like people systematically over estimate low probabilities because they fail to consider probabilities such as 10^-7 or 10^-53.

Some people would argue models predicting a probability of at least 0.1 % should have significant weight, for example, at least 10 %, which would imply an expected probability of at least 0.01 % (= 1*10^-3*0.1). However, I have significant concerns about this kind of reasoning. I worry the weights of the models are close to arbitrary. For instance, in Bob Fischer's book about comparing welfare across species, there seems to be only 1 line about the weights. "We assigned 30 percent credence to the neurophysiological model, 10 percent to the equality model, and 60 percent to the simple additive model". People usually give weights that are at least 0.1/"number of models", which is at least 3.33 % (= 0.1/3) for 3 models, when it is quite hard to estimate the weights. However, giving weights which are not much smaller than the uniform weight of 1/"number of models" could easily lead to huge mistakes. As a silly example, if I asked random people with age 7 about whether the gravitational force between 2 objects is proportional to "distance"^-2 (correct answer), "distance"^-20, or "distance"^-200, I imagine I would get a significant fraction picking the exponents of -20 and -200. Assuming 60 % picked -2, 20 % picked -20, and 20 % picked -200, one may naively conclude the mean exponent of -45.2 (= 0.6*(-2) + 0.2*(-20) + 0.2*(-200)) is reasonable. Yet, there is lots of empirical evidence against this which the respondants are not aware of. The right conclusion would be that the respondants have practically no idea about the right exponent because they would not be able to adequately justify their picks.

James Brobin @ 2026-08-05T20:11 (+3)

Thanks! I really like that example you gave about asking seven year olds. This is definitely a major criticism of mine of Bob Fischer's book. It seems like different experts could easily have come to very different conclusions because of what models they choose or, as you point out, how they weigh them.

Vasco Grilo🔸 @ 2026-08-06T18:41 (+2)

Thanks. Relatedly, you may be interested in the comments from me and Wladimir on this post from Bentham's Bulldog arguing for the possibility of intense agony in many species.

Denis @ 2026-08-06T19:14 (+1)

Great post. I'm not sure most of it relates to Bayesian thinking, but just bad logic. I especially appreciate your point about low-probability events - but then the entire hedge-fund industry is based on the fact that people think 2% is a very small number, so we're not alone.

I think there is a lot of value in discounting estimates' influence on actions based on uncertainty.

James Brobin @ 2026-08-06T23:34 (+1)

Yeah, I agree. I think I should have framed it as like ways in which Bayesian thinking fails rather than as an attack on Bayesian reasoning.

simon @ 2026-08-06T21:06 (+1)

The hedge fund industry is not based on this. Most hedge fund investors are sophisticated and they often pay more than 2%. 

Denis @ 2026-08-07T10:17 (+1)

Sorry, I wasn't clear. I was referring to the fee structure, where many funds charge 2% of the investment as annual management fees. Two and Twenty: Explanation of the Hedge Fund Fee Structure. 

So hedge-fund managers can become absurdly rich despite adding very little value to society. IMHO if a fund manager had to set these out as numbers rather than percentages, clients would be more likely to complain that (for large investors) the management fees are ridiculous. ("How can you justify earning $30 m in one year for managing my money?"). But when they see "2%" maybe it feels cheap to objest to such a "small" number.

simon @ 2026-08-10T09:02 (+1)

Yes this was clear - imho you underestimate how sophisticated investors in hedge funds (what you call “clients”) are. The best hedge funds charge way more than 2% and it can still be a rational investment. “Value to society” is not a criterion that matters. “Value to investors” is. 

There are many mutual funds that charge over 1%, add little value and target unsophisticated investors. That’s the case where I buy your point. 

Denis @ 2026-08-10T10:13 (+1)

Fair, It seems you know more about this than I do! Thanks!