What are information hazards?

By MichaelA🔸 @ 2020-02-05T20:50 (+38)

This post was written for Convergence Analysis. Cross-posted to LessWrong.

The concept of information hazards is highly relevant to many efforts to do good in the world (particularly, but not only, from the perspective of reducing existential risks. I’m thus glad that many effective altruists and rationalists seem to know of, and refer to, this concept. However, it also seems that:

• People referring to the concept often don’t clearly define/explain it
• Many people (quite understandably) haven’t read Nick Bostrom’s original (34 page) paper on the subject
• Some people misunderstand or misuse the term “information hazards” in certain ways

Thus, this post seeks to summarise, clarify, and dispel misconceptions about the concept of information hazards. It doesn’t present any new ideas of my own.

In his paper, Bostrom defines an information hazard as:

A risk that arises from the dissemination or the potential dissemination of (true) information that may cause harm or enable some agent to cause harm.

He emphasises that this concept is just about how true information can cause harm, not how false information can cause harm (which is typically a more obvious possibility).

Bostrom’s paper outlines many different types of information hazards, and gives examples of each. The first two types listed are the following:

Data hazard: Specific data, such as the genetic sequence of a lethal pathogen or a blueprint for making a thermonuclear weapon, if disseminated, create risk.

[...] Idea hazard: A general idea, if disseminated, creates a risk, even without a data-rich detailed specification.

For example, the idea of using a fission reaction to create a bomb, or the idea of culturing bacteria in a growth medium with an antibiotic gradient to evolve antibiotic resistance, may be all the guidance a suitably prepared developer requires; the details can be figured out. Sometimes the mere demonstration that something (such as a nuclear bomb) is possible provides valuable information which can increase the likelihood that some agent will successfully set out to replicate the achievement.

He further writes:

Even if the relevant ideas and data are already “known”, and published in the open literature, an increased risk may nonetheless be created by drawing attention to a particularly potent possibility.

This leads him to his third type:

Attention hazard: mere drawing of attention to some particularly potent or relevant ideas or data increases risk, even when these ideas or data are already “known”.

Because there are countless avenues for doing harm, an adversary faces a vast search task in finding out which avenue is most likely to achieve his goals. Drawing the adversary’s attention to a subset of especially potent avenues can greatly facilitate the search. For example, if we focus our concern and our discourse on the challenge of defending against viral attacks, this may signal to an adversary that viral weapons—as distinct from, say, conventional explosives or chemical weapons—constitute an especially promising domain in which to search for destructive applications.

The significance of these and other types of potential information hazards is that it will sometimes be morally best to avoid creating or spreading even true information. (Exactly when and how to attend to and reduce potential information hazards is beyond the scope of this post; I begin to explore that topic here.)

Context and scale

Those quoted examples all relate to fairly large-scale risks (perhaps even existential risks). Some also relate to risks from advancing the development of potentially dangerous technologies (i.e., from going against the principle of differential progress). It seems to me that the concept of information hazards is most often used in relation to such large-scale, existential, and/or technological risks, and indeed that that’s where the concept is most useful.

However, it’s also worth noting that information hazards don’t have to relate to these sorts of risks. Information hazards can occur in a wide variety of contexts, and can sometimes be very mundane or minor. Some of Bostrom’s types and examples highlight that. For example:

Spoilers constitute a special kind of disappointment. Many forms of entertainment depend on the marshalling of ignorance. Hide-and-seek would be less fun if there were no way to hide and no need to seek. For some, knowing the day and the hour of their death long in advance might cast shadow over their existence.

[Thus, it is also possible to have a] Spoiler hazard: Fun that depends on ignorance and suspense is at risk of being destroyed by premature disclosure of truth.

Who’s at risk?

Spoiler hazards are a type of information hazards that risk harm only to the knower of the true information themselves, and as a direct result of them knowing the information. (In contrast, in Bostrom’s earlier examples, the knower might eventually be harmed, but this would be (1) along with many other people, and (2) a result of catastrophic or existential risks that were made more likely by the knowledge the knower spread, rather than as a direct result of them having that knowledge.)

Bostrom also lists other such types of information hazards where the risk of harm is to the knower themselves, and directly results from their knowledge. But there appears to be no established term for the entire subset of information hazards that fit that description. Proposed terms I’m partial to include “knowledge hazards” and “direct information hazards”. (Further discussion can be found in that comments section. If you have thoughts on that, please comment here or there.)

But I should emphasise that that is indeed only a subset of all information hazards; information hazards can harm people other than the knower themselves, and, as mentioned above, this will be the case in the contexts where information hazards are perhaps most worrisome and worth attending to. (I emphasise this because some people misunderstand or misuse the term “information hazards” as referring only to what we might call “knowledge hazards”; this misuse is apparent here, and is discussed here.)

Information hazards are risks

As noted, an information hazard is “A risk that arises from the dissemination or the potential dissemination of (true) information that may cause harm or enable some agent to cause harm” (emphasis added). Thus, as best I can tell:

• Something can be an information hazard even if no harm has yet occurred, and there’s no guarantee it will ever occur.

• E.g., writing a paper on a plausibly dangerous technology can be an information hazard even if it turns out to be safe after all.

• But if we don’t have any specific reason to believe that there’s even a “risk” of harm from some true information, and we just think that it’d be worth bearing in mind that there might be a risk, it may be best to say there’s a “potential information hazard”.

• So it’s probably not helpful to, e.g., slap the label of “information hazard” on all of biological research, for example.

Closing remarks

I hope you’ve found this post clear and useful. To recap:

• The concept of information hazards relates to risks of harm from creating or spreading true information (not from creating or spreading false information).
• The concept is definitely very useful in relation to existential risks and risks from technological development, but can also apply in a wide range of other contexts, and at much smaller scales.
• Some information hazards risk harm only to the knower of the true information themselves, and as a direct result of them knowing the information. But many information hazards harm other people, or harm in other ways.
• Information hazards are risks of harm, not necessarily guaranteed harms.

In my next post, I’ll attempt to similarly summarise and clarify the concept of differential progress (a generalisation of Bostrom’s differential technological development). And in a later post, Convergence will suggest some rules-of-thumb regarding why, when, and how one should deal with potential information hazards (aiming for more nuance than just “Always pursue truth!” or “Never risk it!”).

My thanks to David Kristoffersson and Justin Shovelain for feedback on this post.

Other sources relevant to this topic are listed here.