Reducing long-term risks from malevolent actors

By David_Althaus, Tobias_Baumann @ 2020-04-29T08:55 (+344)

Summary

What do we mean by malevolence?

Before we make any claims about the causal effects of malevolence, we first need to explain what we mean by the term. To this end, consider some of the arguably most evil humans in history—Hitler, Mao, and Stalin—and the distinct personality traits they seem to have shared.[1]

Stalin repeatedly turned against former comrades and friends (Hershman & Lieb, 1994, ch. 15, ch. 18), gave detailed instructions on how to torture his victims, ordered their loved ones to watch (Glad, 2002, p. 13), and deliberately killed millions through various atrocities. Likewise, millions of people were tortured and murdered under Mao’s rule, often according to his detailed instructions (Dikötter, 2011; 2016; Chang & Halliday, ch. 8, ch. 23, 2007). He also took pleasure in watching acts of torture and imitating in what his victims went through (Chang & Halliday, ch. 48, 2007). Hitler was not only responsible for the death of millions, he also engaged in personal sadism. On his specific instructions, the plotters of the 1944 assassination attempt were hung by piano wires and their agonizing deaths were filmed (Glad, 2002). According to Albert Speer, “Hitler loved the film and had it shown over and over again” (Toland, 1976, p. 818). Hitler, Mao, and Stalin—and most other dictators—also poured enormous resources into the creation of personality cults, manifesting their colossal narcissism (Dikötter, 2019). (The section Malevolent traits of Hitler, Mao, Stalin, and other dictators in Appendix B provides more evidence.)

Many scientific constructs of human malevolence could be used to summarize the relevant psychological traits shared by Hitler, Mao, Stalin, and other malevolent individuals in positions of power. We focus on the Dark Tetrad traits (Paulhus, 2014) because they seem especially relevant and have been studied extensively by psychologists. The Dark Tetrad comprises the following four traits—the more well-known Dark Triad (Paulhus & Williams, 2002) refers to the first three traits:

There is considerable overlap between the Dark Tetrad traits. In general, almost all plausible operationalizations of malevolence tend to positively correlate with each other and negatively with “benevolent” traits such as altruism, humility or honesty. (See the section Correlations between dark traits and other traits in Appendix B for more details.)

This suggests the existence of a general factor of human malevolence[2]: the Dark Factor of Personality (Moshagen et al., 2018)—analogous to g, the general factor of intelligence—characterized by egoism, lack of empathy[3] and guilt, Machiavellianism, moral disengagement, narcissism, psychopathy, sadism, and spitefulness. Like most personality traits (Johnson et al., 2008), malevolent traits seem relatively stable over the lifespan (Obradović et al., 2007) and influenced by genetic factors (Vernon et al., 2008), but more on this below.[4]

Throughout this article, we will assume a dimensional—rather than categorical, “black-or-white”—conception of malevolence. That is, we believe that malevolent traits exist on a continuum—just like most other human traits such as extraversion or intelligence (cf. Haslam et al., 2012; Plomin, 2019, ch. 5). Slight Machiavellian or sadistic tendencies, for example, are common. Many humans seem to flatter their superiors and enjoy seeing (non-tragic) mishaps of their political opponents. But only a few individuals will derive pleasure from witnessing human torture or will kill their former friends just to consolidate their power.

It is this latter type of human—showing clear signs of at least some highly elevated Dark Tetrad traits—who we have in mind when we use the term “malevolent”.

Malevolent humans in power pose serious long-term risks

In this section we discuss why and how malevolent individuals in highly influential positions—such as political leaders or CEOs of notable companies—could negatively affect humanity’s long-term trajectory, ultimately increasing existential risks (including extinction risks) and risks of astronomical suffering (s-risks).

Malevolent humans often rise to power

Malevolent humans are unlikely to substantially affect the long-term future if they cannot rise to power. But alas, they often do. The most salient examples are dictators who clearly exhibited elevated malevolent traits: not only Hitler, Mao, and Stalin, but also Saddam Hussein, Mussolini, Kim Il-sung, Kim Jong-il, Duvalier, Ceaușescu, and Pol Pot, among many others.

In fact, people with increased malevolent traits might even be overrepresented among business (Babiak et al., 2010; Boddy et al., 2010; Lilienfeld, 2014), military, and political leaders (Post, 2003; Lilienfeld et al., 2012), perhaps because malevolent traits—especially Machiavellianism and narcissism—often entail an obsession with gaining power and fame (Kajonius et al., 2016; Lee et al., 2013; Southard & Zeigler-Hill, 2016) and could even be advantageous in gaining power (Deluga, 2011; Taylor, 2019). Again, Appendix B provides more details.

History suggests that malevolent leaders have caused enormous harm

One reason for expecting malevolent humans in power to pose risks to the future is that they seem to have caused great harm in the past.

Hitler, Mao, and Stalin were directly involved in several of the greatest atrocities in history, such as World War II, the Holocaust, the Great Leap Forward, the Cultural Revolution, and the Great Terror. There thus seems to be a correlation between the malevolence of (autocratic) political leaders and the amount of harm that occurred under their rule; at least according to our understanding of history.[5] If the past is any guide to the future, individuals with highly elevated dark traits could again manage to rise to positions of extreme power and cause extraordinary harm.

However, correlation does not imply causation. Even if we grant that this correlation indeed exists[6], one could argue that there are better explanations for how these atrocities came about. In particular, it seems plausible that other factors—such as political instability or extremist ideologies—matter most. We discuss these issues in more detail in this section of Appendix A.

It’s also worth mentioning that individuals with malevolent personalities are more likely to adopt dangerous ideologies. Dark Tetrad traits are associated with political extremism generally, including supporting the use of violence to achieve political and other ideological goals (Duspara & Greitemeyer, 2017; Međedović & Knežević, 2018; Gøtzsche-Astrup, 2019; Jones, 2013).[7]

Thus, while we agree that history is largely shaped by economic, political, cultural, institutional, ideological and other systemic forces, we believe that the personality traits of individual leaders—at the very least in autocratic regimes—can plausibly make a substantial difference as well (see also Bertoli et al., 2019; Byman & Pollack, 2001; especially p. 115-121; Gallagher & Allen, 2014; Jones & Olken, 2005). After all, there were humans who rose to power within rather autocratic regimes but who nevertheless enacted relatively beneficial policies. Examples include Juan Carlos I of Spain, Mustafa Kemal Atatürk, Mikhail Gorbachev, Lee Kuan Yew, and Marcus Aurelius, who seemed to exhibit more benevolent personality traits than the likes of Hitler, Mao, and Stalin.

Malevolent leaders have the potential to corrupt humanity’s long-term future

One could question whether malevolent individuals can substantially influence the long-term trajectory of humanity for the worse, even from positions of extreme power. It is possible that they only cause short-term harm, in which case reducing malevolence may not be a priority from a longtermist perspective.

However, we believe malevolent leaders plausibly have a significant detrimental effect on the long-term future. Hitler, Stalin, and Mao, for instance, seemed to have had a profoundly negative influence on global affairs and international cooperation, some of which can arguably still be felt today, more than half a century after the atrocities they perpetrated.[8] That said, it is difficult—if not impossible—to assess long-term impacts, as we do not know what would have happened counterfactually.

Broad risk factors due to malevolent leaders

Beckstead (2013) asks whether there is “a common set of broad factors which, if we push on them, systematically lead to better futures”. It seems plausible that malevolent humans in power would push such factors in the wrong direction.[9]

Specifically, we conjecture that malevolent humans in power would affect the risk factors below in the following ways:

Such trends would plausibly lead to worse futures in expectation. They also plausibly increase existential risks (including extinction risks) and suffering risks (see the next section). However, the evidence linking these risk factors to malevolent humans in power is fairly weak, for various reasons. We are therefore only somewhat confident in these connections.

Existential and suffering risks due to malevolent leaders

In terms of more concrete scenarios, the most extreme risks to the long-term future would arguably result from malevolent humans with access to highly advanced technology, particularly transformative AI.

The following list outlines some (non-exhaustive) examples of how malevolent individuals could increase existential and suffering risks:

While specific scenarios are necessarily speculative, it seems clear that malevolent leaders pose a serious threat to humanity’s long-term future. Of course, malevolent leaders are not the root of all evil, and many conflicts, wars and atrocities would happen without them. Nevertheless, we believe that preventing malevolent individuals from rising to power is likely valuable and robustly positive, according to almost all moral perspectives (compare also Beckstead, 2013; Tomasik, 2013a, 2013b).

Interventions to reduce the influence of malevolent actors

Advancing the science of malevolence

Further research into the construct of malevolence and its consequences would allow us to make more rigorous statements about the links between malevolent leaders and bad outcomes.

A more established science of malevolence would also help raise awareness of malevolent personality traits and how to detect them among the general public, influencers, politicians, researchers, and academics. Generally, the more we know about malevolence, the easier it is to accomplish many of the interventions discussed below.

Developing better constructs and measures of malevolence

It seems worthwhile to develop constructs capturing more precisely the constellation of traits most worrisome from a longtermist perspective, as existing constructs will not always do so.

For example, one of the most commonly used scales to measure psychopathy, the Psychopathy Checklist-Revised by Hare et al. (1990), consists of 20 items, grouped into two factors. Factor 1—characterized by cruelty, grandiosity, manipulativeness, and a lack of guilt—arguably represents the core personality traits of psychopathy. However, scoring highly on factor 2—characterized by impulsivity, reactive anger, and lack of realistic goals—is less problematic from our perspective. In fact, humans scoring high on factor 1 but low on factor 2 are probably more dangerous than humans scoring high on both factors (more on this below). Generally, most measures of psychopathy include items related to increased impulsivity (e.g., Cooke & Michie, 2001; Levenson et al., 1995; Lilienfeld & Andrews, 1996).

The most dangerous individuals tend to go undiagnosed

Individuals officially diagnosed as malevolent—e.g. those diagnosed with psychopathy, antisocial or narcissistic personality disorder—are probably unrepresentative of the most dangerous individuals. This is because an official diagnosis is only made when somebody suffers from immediate and severe problems (relating to their malevolence) or was forced to seek therapy, e.g., because they committed a crime.

In contrast, malevolent humans with good impulse-control and otherwise decent mental health have no reason to seek out a therapist and will generally not be convicted of crimes. The most dangerous malevolent humans will realize that not being unmasked as malevolent is of the highest importance, and will have sufficient motivation, cunning, self-awareness, charisma, social skills, intelligence and impulse-control to avoid detection (Perina et al., 2020).[18] Such individuals might even deliberately display personality characteristics entirely at odds with their actual personality. In fact, many dictators did precisely that and portrayed themselves—often successfully—as selfless visionaries, tirelessly working for the greater good (e.g., Dikötter, 2019). It may therefore be very valuable to conduct more research on this hard-to-detect type of conscientious, strategic malevolence (cf., e.g., Gao & Raine, 2010; Lilienfeld et al., 2015; Mullins-Sweatt et al., 2010).

Manipulation-proof measures of malevolence

To prevent malevolent humans from reaching highly influential positions, we need to be able to reliably detect those traits.

Currently, most measures of dark traits take the form of either interviews or self-completed questionnaires. Smart malevolent humans can easily manipulate these types of instruments and evade detection by lying. It is key, therefore, that we develop manipulation-proof measures of malevolence, i.e., measures that cannot (easily) be gamed.

One possibility would be to ask peers and previous associates to evaluate the personality traits of the person in question.[19] Of course, this raises several problems. Malevolent humans could have charmed and fooled many of their (former) friends and colleagues. They could also bribe or manipulate others to lie. So, while other-report measures (e.g., 360 degree assessments) may be harder to manipulate than self-reported ones and are therefore valuable, they are unlikely to completely solve the problem.[20]

Physiological or neurobiological measures based on methods like EEG or fMRI might be particularly difficult to manipulate—though this would probably require substantial technological and scientific progress. Neuroimaging techniques might allow us to identify abnormal brain structures or detect suspicious behavior, such as showing neurological signs of pleasure and/or no distress when seeing other humans or animals in pain. Therefore, more neurobiological research on the neurological signatures of pleasure and displeasure (e.g., Berridge & Kringelbach, 2013), and on the neurobiology of sadism and psychopathy, might be very valuable.[21] (Note that we have not investigated this in detail, so it is probably best to start with a systematic literature review.) However, such methods also raise ethical questions about judging people by brain scans rather than their actual behavior.

Potential misuse and negative consequences

Manipulation-proof measures of malevolence could also be misused—like all technology. For instance, governments might falsely brand political opponents as psychopaths.

Another concern is that such tests may constitute an unfair form of discrimination against humans with certain traits. This is because they measure innate characteristics that are impossible to change, rather than exclusively considering the actual behaviour of individuals. Also, even if this is deemed acceptable in the case of malevolence, advocating for testing in this context might lead to the widespread adoption of personality testing in general, which some believe could have negative consequences. (On the other hand, existing selection procedures also implicitly or explicitly select based on innate traits such as intelligence, and also include various kinds of tests.)

Lastly, unless tests of malevolence have perfect validity and reliability, there will be measurement errors: Some people will be diagnosed as highly malevolent even though they aren’t, and some truly malevolent people will escape detection.

How valuable would manipulation-proof measures of malevolence be in practice?

Given the potentially enormous benefits, why has there been so little interest in the development of manipulation-proof tests of malevolence? First, doing so is likely difficult and, especially if it involves neuroscience research, expensive[22] (as an example, MRI machines cost between $0.3M and $3M). Second, malevolent humans might, in some cases, actually benefit individual companies or political parties: high levels of psychopathy and narcissism could be useful for things like negotiating, motivating employees, or winning public approval. Third, most people likely overestimate their ability to discern malevolent traits in others, making them less interested in such tests. Finally, it seems that tests in general are not used much in at least some contexts; for example, most elected or appointed positions in government do not require intelligence, knowledge, or personality tests.[23]

One might argue that it was obvious to most people that dictators such as Hitler, Mao, and Stalin were malevolent even before they gained power, and that manipulation-proof measures of malevolence would therefore have been useless. However, we are doubtful that people can easily detect malevolence, at least in the most dangerous types of individuals, as mentioned above. (See also the section How well can people detect malevolent traits in Appendix A for more details.)

Of course, many did suspect that Hitler, Mao, and Stalin were malevolent. However, this was not common knowledge—and without objective evidence, calling an individual malevolent can easily be dismissed as slander. Not to mention that anyone making such accusations risks serious reprisals. So even if a majority had realized early on that Hitler, Mao, and Stalin are malevolent, it might not have helped.

However, if manipulation-proof and valid measures of malevolence had existed—alongside strong norms to use them to screen political leaders and widespread trust in their accuracy—it could have been common knowledge that these individuals were malevolent, which would have significantly reduced their chance of rising to power. Essentially, manipulation-proof and valid measures of malevolence could serve as an objective arbiter of good intentions, analogous to the scientific use of experiments as objective arbiters of truth. Their role in a hiring process could then be compared to security clearances, for instance.

It is not clear whether even perfectly diagnostic measures of malevolence would ever become widespread—for example, because of the abovementioned ethical concerns. However, for highly influential positions, people are most willing to make use of all available evidence (and candidates for such positions have an incentive to provide credible signals of trustworthiness). For instance, receiving a top-secret security clearance involves extensive interviews with one’s (former) spouses, colleagues, friends and neighbors, alongside reviews of medical and psychiatric records, and sometimes even polygraph examinations. This elaborate and arguably privacy-violating process would be unacceptable for a routine job, but is considered appropriate given the stakes at hand. Last, it could already be valuable if only a few companies or government departments started using manipulation-proof measures of malevolence; near-universal adoption of such measures is by no means necessary.

Despite these caveats, we believe that work on manipulation-proof measures of malevolence is promising. Subject to personal fit, it may be worthwhile for some effective altruists to consider careers in psychology or neuroscience. This would allow them to advance the science of malevolence, contribute to the development of manipulation-proof measures of malevolence, and improve their chances to convince decision-makers to take such measures seriously.

Political interventions

Many factors determine whether an individual can rise to a position of power, and it is important to include (non-)malevolence as a criterion when selecting leaders. Ideally, we should establish strong norms against allowing highly malevolent leaders to rise to power—even in cases where elevated Dark Tetrad traits may be instrumental in advancing the interests of a company or nation.

While the notion of Dark Tetrad traits is not foremost in most people’s minds, one could argue that much political debate is about related concepts like the trustworthiness or honesty of candidates, and voters do value those attributes.[24]

Perhaps the key issue, then, is not a lack of awareness; rather the non-availability of reliable objective measures and the overestimation of people's ability to detect malevolence. In fact, humans seem too eager to view their political opponents as inherently malevolent and ill-intentioned. Conversely however, humans also tend to view members of their own tribe as inherently good and overlook their misdeeds. (See again Appendix A for more details.)

The media also tends to depict impulsive psychopaths—say, ruthless serial killers with a long history of violence or crime. These are relatively easy to detect, potentially leading to a false sense of security (compare also Babiak et al., 2010, p.174-175). As mentioned above, it may therefore be valuable to raise awareness that at least some types of malevolent humans are difficult to detect.

Alternatively, we could influence political background factors that make malevolent leaders more or less likely. It seems plausible that political instability, especially outright revolutions, enable malevolent humans to rise to power (Colgan, 2013, p. 662-665). Generally, democracies plausibly select for more trustworthy, predictable and benevolent leaders (Byman & Pollack, 2001, p.139-140). Thus, interventions to promote democracy and reduce political instability seem valuable—though this area seems rather crowded.

Even within established democracies, we could try to identify measures that avoid excessive polarization and instead reward cross-party cooperation and compromise. Mitigating the often highly combative nature of politics would plausibly make it harder for malevolent humans to rise to power.[25] (For example, effective altruists have discussed electoral reform as a possible lever that could help achieve this.)

Since elevated Dark Tetrad traits are significantly more common among men (Paulhus & Williams, 2002; Plouffe et al., 2017), it also seems beneficial to advance gender equality and increase the proportion of female leaders.

Other potential factors that might facilitate the rise of malevolent individuals include social and economic inequality, poverty, ethnic, military or religious conflicts, and a “widespread sense of grievance or resentment” (Glad, 2002, p. 4). Thus, identifying cost-effective interventions to improve these factors (as well as identifying factors we haven’t thought of) could be promising. A more thorough study of the history of malevolent humans rising to power would also be valuable to better understand which factors are most predictive.

Overall, it seems plausible that many promising political interventions to prevent malevolent humans from rising to power have already been identified and implemented—such as, e.g., checks and balances, the separation of powers, and democracy itself. After all, much of political science and political philosophy is about preventing the concentration of power in the wrong hands.[26] We nevertheless encourage interested readers to further explore these topics.

Future technologies and malevolence

In this section, we explore how possible future technologies could be used to reduce the influence of malevolent actors.

Whole brain emulation

Whole brain emulation is the hypothetical process of scanning the structure of a brain and replicating it on a computer. Hanson (2016) explores the possible implications of this technology. In his scenario, brain emulations (“ems”) will shape future economic, technological and political processes due to their competitive advantage over biological minds.

One key question is: which human brains will be uploaded? We believe that it would be crucial to screen potential ems for malevolence—particularly the first individuals to be uploaded. Considering the power that the first ems would likely have to shape this new “Age of Em”, it could be disastrous for humanity’s long-term future if a malevolent individual forms the basis for (some of) the first ems (cf. Bostrom, 2002, p. 12). Conversely, by screening for malevolence, using manipulation-proof measures, we could effectively reduce malevolence among ems. This offers an unprecedented opportunity to ensure that malevolent forces have significantly less influence over the long-term future.[27]

Transformative AI

Many longtermist effective altruists think that shaping transformative artificial intelligence, and in particular solving the alignment problem, is a particularly good lever to improve the long-term future. Some concrete proposals for alignment—such as Iterated distillation and amplification—involve a “human-in-the-loop” whose feedback is used to align increasingly capable AI.

In these scenarios, the “human-in-the-loop” plausibly has enormous responsibility and leverage over the long-term future. It is therefore extremely valuable to ensure that the relevant individual or individuals—if e.g. a jury or parliament fulfills the role of “human-in-the-loop”—do not exhibit malevolent traits. (Again, this requires or is at least facilitated by the availability of manipulation-proof measures.)

Even without human involvement, artificial agents may exhibit behaviour that resembles malevolence (to the extent that this notion makes sense in non-human contexts) if such heuristics prove useful in its training process. After all, the fact that malevolent traits such as psychopathy or sadism evolved in some humans suggests that those traits provided fitness advantages, at least in certain contexts (Book et al. 2015; McDonald et al., 2012; Nell, 2006; Jonason et al., 2015).

In particular, it is possible that domain-general capabilities will emerge via increasingly complex multi-agent interactions (Baker et al., 2019). In this case, it is crucial that the training environment is set up in a way that prevents the evolution of undesirable traits like malevolence, and instead rewards cooperative and trustworthy behaviour.

To the extent that artificial intelligence designs are inspired by the human brain (“neuromorphic AI”), it seems important to understand the neuroscientific basis of malevolence in humans to reduce the risk of neuromorphic AIs also exhibiting malevolent traits.

Genetic enhancement

A third class of relevant new technologies are those that make it possible to change the genetic makeup of future humans. This would offer unprecedented leverage to change personality traits and “human nature”, for better or for worse (cf. Genetic Enhancement as a Cause Area). In particular, selection against malevolent traits could significantly reduce the influence of malevolent individuals.

This is because most variance in adult personality is due to genetic influences (~30–50%) and nonshared environment effects (~35–55%), leaving comparatively little room for the shared environment (~5–25%)[28] (e.g., Knopik et al., 2018, ch. 16; Johnson et al., 2008; Plomin, 2019; Vukasović & Bratko, 2015). (See the sections “Broad-sense heritability estimates of dark traits” and “Is selecting for personality traits possible?” in Appendix B for more details.) By contrast, nonshared environmental influences—which include measurement error, chance life events, and de novo mutations—seem to be mostly unsystematic, idiosyncratic, and unstable, and therefore difficult to influence (Plomin, 2019, ch. 7).

Genetic enhancement technologies might also result in the creation of humans with extraordinary intelligence (see, e.g., Shulman & Bostrom, 2014, p.2-3). Such humans, if created, will likely be overrepresented in positions of enormous influence and would thus have an outsized impact on the long-term future. Reducing malevolence among those individuals is therefore especially important.

Overview of genetic enhancement technologies

There are various technologies that would make it possible to modify the genetic makeup of future humans. We think the following four are most relevant:

Note that these methods can interact with each other and should thus not be viewed as being completely separate. For more details, we highly recommend Gwern’s Embryo selection for intelligence.

How far away are these technologies? Gwern writes that “IES is still distant and depends on a large number of wet lab breakthroughs and finetuned human-cell protocols.” Nonetheless, he states that: “[...] it seems clear, at least, that it will certainly not happen in the next decade, but after that…?”. He concludes that “IES has been badly under-discussed to date.”

Regarding genome synthesis, Gwern writes that the “cost curve suggests that around 2035, whole human genomes reach well-resourced research project ranges of $10-30m” and that it “is entirely possible that IES will develop too slowly and will be obsoleted by genome synthesis in 10-20 years.”

Gwern gives the following summary:

“CRISPR & cloning are already available but will remain unimportant indefinitely for various fundamental reasons; [...] massive multiple embryo selection is some ways off but increasingly inevitable and the gains are large enough on both individual & societal levels to result in a shock; IES will come sometime after massive multiple embryo selection but it’s impossible to say when, although the consequences are potentially global; genome synthesis is a similar level of seriousness, but is much more predictable and can be looked for, very loosely, 2030-2040 (and possibly sooner).”

Dangers

Genetic enhancement is widely criticized. Numerous atrocities have been committed in the quest to forge a new, “better” kind of human. We would like to emphasize that we do not advocate for genetic enhancement per se. We only argue that if genetic enhancement happens, it seems prima facie important to select against malevolent traits—comparable to the rationale behind differential intellectual progress and differential technological progress.

Still, we are treading dangerous waters. Even just bringing up the possibility of selection for or against personality traits might inspire misuse of such methods. One particularly worrisome scenario is selection against all forms of rebellion and independence, branded as “antisocial tendencies”, which could enable extreme totalitarianism. Generally, the currently dominant individuals and classes could abuse these powerful technologies to cement their power.

It is also worth noting that very high levels of usually beneficial traits can be negative: too much trust, for example, might result in naïvety and an increased likelihood of being exploited. Similarly, completely eliminating usually harmful traits could backfire as well: for example, in certain situations, some degree of narcissism and Machiavellianism may benefit entrepreneurs and politicians. Generally, different personality traits are useful for different roles in society, so some diversity is beneficial.

For these and other reasons, it could be net negative to shift personality traits by more than one or two standard deviations.[29] However, we are primarily interested in shrinking the right tail of the distribution—e.g., by selecting against embryos with polygenic scores for Dark Tetrad traits above, say, the 99th percentile. This could be done while only marginally decreasing the mean. Generally, if we apply the (double) reversal test, current rates of dark traits—particularly highly elevated ones—appear very far from optimal.

Moreover, many dark traits appear to be genetically correlated with each other and negatively genetically correlated with benevolent personality traits (Vernon et al., 2008). Thus, selecting against one dark trait will tend to decrease other dark traits and increase benevolent traits. This plausibly makes selection efforts more robust, though this could also have some downsides.[30]

Lastly, we should arguably be especially cautious in scenarios that involve genetically enhanced humans of extraordinary intelligence. Extremely intelligent sadists and psychopaths would pose risks that outweigh any plausible benefits.

Interventions

Funding or otherwise encouraging more research on the genetic basis of malevolent traits would allow us to better select against these traits. Ideally, we would have a good understanding of the genetic basis of malevolent traits before technologies such as genome synthesis arrive. Thus, it is plausibly time-sensitive to do this research now, even if powerful genetic enhancement technologies will not be developed for the next several decades.

A particularly cost-effective intervention might be to convince personal genomics companies, such as 23andMe, to offer tests of Dark Tetrad traits. 23andMe has over 10 million customers, so even if only a small fraction of customers took these tests (e.g., out of curiosity), we would already achieve sample sizes surpassing those of large GWA studies.[31] Improved psychological measures of malevolence with higher reliability and validity, as discussed in previous sections, would also enable GWA studies to better identify genetic variants associated with such traits.

In general, increasing the social acceptability of screening for Dark Tetrad traits plausibly increases the probability that future projects involving more powerful technology will also do so. The more established and well-known Dark Tetrad traits are, and the less controversial their heritability, the easier it would be to accomplish many of the interventions mentioned above. It might be valuable, for instance, to persuade sperm banks or other institutions responsible for screening sperm (or egg) donors to add measures of Dark Tetrad traits to their screening process and display the results prominently to women choosing sperm donors.

As with non-genetic interventions, we could attempt to raise awareness of malevolent traits, their heritability, and their dangers. Rather than trying to make changes to the supply side, it might be easier to increase demand by popularizing Dark Tetrad traits.[32] Most parents want their children to be responsible, empathic, and kind. If they are willing to pay for screening for malevolent traits, then sperm banks or others will offer such services.[33]

However, considering the significant dangers outlined above, we believe that public advocacy of the idea of genetic selection against malevolence would likely be premature. Indeed, more research on how to best avoid negative consequences—such as increased inequality or dehumanization of (un)enhanced humans—of possible interventions in this area would be important.

Subject to personal fit, it may also be worthwhile for some effective altruists to consider careers in bioinformatics, social sciences relating to GWA studies, bioethics, or related fields, to be in a good position to later influence key players.

Concluding remarks

Many of the above interventions face serious technical challenges. It may be hard to develop manipulation-proof measures of malevolence, and selection on personality traits is probably difficult due to low additive heritability. In addition, many interventions—especially those related to genetic enhancement technologies—entail severe risks of misuse and unintended negative consequences.

However, some of the suggested interventions involve neither speculative future technology nor controversial ideas about genetic enhancement. Overall, we recommend a mix of different interventions, as well as further work aiming to find new types of interventions and checking the assumptions that underlie existing interventions.

Most parents, cultures, and religions feature some notion of "not being evil", so one could argue that reducing malevolence, broadly construed, is already quite crowded. However, we believe the interventions we have explored are more targeted, and are potentially more far-reaching and more neglected than, say, cultural norms or parenting.

Reducing the influence of malevolent actors is not a panacea, of course. Many of the world's biggest problems are not (primarily) due to malevolent intent per se, and instead are mostly caused by incompetence, irrationality, indifference, and our inability to coordinate the escape from undesirable equilibria.

That being said, we believe that reducing the chances of malevolent individuals rising to power would have substantially positive effects under a broad range of scenarios and value systems—whether they place primary importance on avoiding existential risks, reducing suffering, or improving the quality of the long-term future.

Appendix A

How important are situational factors and ideologies compared to personality traits?

In this section, we discuss the extent to which historical atrocities can be attributed to the personality traits of individuals versus structural factors.

First, it seems plausible that background conditions that enable dictatorships in the first place—such as political instability and an absent rule of law—also make it more likely that malevolent humans will rise to power. Individuals who are reluctant to engage in murder and betrayal, for example, will be at a considerable disadvantage under such conditions (also see Colgan 2013, especially p. 662-665).

Similarly, power tends to corrupt (e.g., Bendahan et al., 2015; Cislak et al., 2018) so it could be argued that most individuals who rise to the top within autocratic regimes, will become more malevolent. Generally, a wealth of social psychology research attests to the importance of situational factors in explaining human behavior (Milgram, 1963; Burger, 2009), though the understanding of modern psychology is that behavior depends on both situational factors and individual personality traits (Bowers, 1973; Endler & Magnusson, 1976).

One particularly relevant factor is the spread of extremist and fanatical ideologies such as fascism, violent communism, and fundamentalist religion, which have undoubtedly contributed to historical atrocities. In fact, such ideologies have plausibly had a much bigger impact on history than the personality traits of individuals and could pose even greater risks to the long-term future. So why focus on personality rather than ideology or structural factors?

For one, tens of millions of people are already combating the dangerous ideologies mentioned above, or work on ensuring political stability and rule of law. These efforts are laudable, but also seem very crowded, which suggests that many of the most cost-effective interventions have already been identified and carried out.

As mentioned above, there is also ample evidence that individuals with malevolent personalities are drawn to dangerous ideologies:[34] Dark Triad traits predict increased intention to engage in political violence (Gøtzsche-Astrup, 2019). Narcissism and psychopathy are associated with political extremism (Duspara & Greitemeyer, 2017). Sadistic and psychopathic traits predict endorsing a militant extremist mind-set, in particular the use of violence to achieve political and other ideological goals (Međedović & Knežević, 2018). Machiavellianism and psychopathy predict racist attitudes, including support for Neo-Nazis and the KKK (Jones, 2013). Dark Triad traits correlate with social-dominance orientation (Jones & Figueredo, 2013; Jones, 2013), a measure of an individual’s preference for economic and social inequality within and between groups (Pratto et al., 1994; Dallago et al. 2008).[35]

Most ideologies also seem open for interpretation, leaving sufficient room for the idiosyncratic beliefs and personality traits of leaders to make a difference. Khrushchev and Gorbachev, for example, while broadly sharing Stalin’s Marxist-Leninist ideology, have caused much less harm than Stalin. Conversely, as the examples of Stalinism, Maoism, and Juche show, malevolent individuals can develop an existing ideology further, making it even more harmful.

In the end, ideologies, belief systems, and personality traits appear inevitably intertwined. Narcissism, for example, entails inflated beliefs about one’s abilities and place in history, by definition. Generally, malevolent individuals tend to hold beliefs that serve as (un)conscious justifications for their behavior, such as a sense of entitlement and grandiosity, and seem more likely to endorse dangerous worldviews and “ideologies that favor dominance (of individuals or groups)” (Moshagen, 2018, p. 659).

Finally, it is instructive to compare large-scale atrocities to small-scale atrocities like murder or contract killing. While rates of violent crime surely depend on social background factors and culturally transmitted norms, psychopathy is also considered a strong predictor for homicide, including instrumental, calculated murder (Fox & DeLisi, 2019). If we accept that malevolent personality traits like psychopathy play a causal role in violent crime, it stands to reason that such traits also play at least some causal role in many large-scale atrocities.

How well can people detect malevolent traits?

Historical evidence suggests that even many of their political adversaries—at least for some time—did not realize that Hitler, Mao, and Stalin were malevolent, even after they were in power.

Chamberlain famously trusted Hitler’s sincerity for far too long. Churchill once remarked that ‘‘Poor Chamberlain believed he could trust Hitler. He was wrong. But I don’t think that I am wrong about Stalin’’ (Yergin, 1977, p. 65). Similarly, Truman believed that Stalin ‘‘could be depended upon…I got the impression Stalin would stand by his agreements’’ (Larson, 1988, p. 246).[36] At least until the 1940s, many Westerners and Chinese seemed to have been enamored with Mao, potentially partly due to the influential book ‘Red Star over China’ (Snow, 1937) which painted him in an extremely favorable light (Chang & Halliday, ch. 18, 2007).

Countless famous intellectuals—including G.B. Shaw, H.G. Wells, Sartre, Simone de Beauvoir, Beatrice Webb, Sidney Webb, Susan Sontag, Oswald Spengler, Carl Jung, Konrad Lorenz, and Martin Heidegger—praised authoritarian leaders like Mussolini, Hitler, Stalin or Fidel Castro (Hollander, 2016; 2017). Even today, most Russians and Chinese think highly of Stalin and Mao, respectively.

In summary, it seems that many humans fail to detect malevolent individuals, particularly when ideological, patriotic or other biases affect their judgment. Generally, Hitler, Mao, and Stalin—like many narcissists—seem to have been quite polarizing; some thought they were obviously malevolent, others viewed them as benevolent, nearly messianic figures.

Appendix B

See Appendix B: Reducing long-term risks from malevolent actors for additional details.

Acknowledgements

Thanks to Jesse Clifton, Jonas Vollmer, Lukas Gloor, Stefan Torges, Chi Nguyen, Mojmír Stehlík, Richard Ngo, Pablo Stafforini, Caspar Oesterheld, Lucius Caviola, Johannes Treutlein, and Ewelina Tur for their valuable comments and feedback. Thanks to Sofia-Davis Fogel for copy editing. All errors and views expressed in this document are our own, not those of the commenters.

David’s work on this post was funded by the Center on Long-Term Risk.

References

Azizli, N., Atkinson, B. E., Baughman, H. M., Chin, K., Vernon, P. A., Harris, E., & Veselka, L. (2016). Lies and crimes: Dark Triad, misconduct, and high-stakes deception. Personality and Individual Differences, 89, 34-39.

Babiak, P., Neumann, C. S., & Hare, R. D. (2010). Corporate psychopathy: Talking the walk. Behavioral sciences & the law, 28(2), 174-193.

Baker, B., Kanitscheider, I., Markov, T., Wu, Y., Powell, G., McGrew, B., & Mordatch, I. (2019). Emergent tool use from multi-agent autocurricula. arXiv preprint arXiv:1909.07528.

Baron-Cohen, S. (2012). The science of evil: On empathy and the origins of cruelty. Basic books.

Beckstead (2013). How to compare broad and targeted attempts to shape the far future.

Bendahan, S., Zehnder, C., Pralong, F. P., & Antonakis, J. (2015). Leader corruption depends on power and testosterone. The Leadership Quarterly, 26(2), 101-122.

Berridge, K. C., & Kringelbach, M. L. (2013). Neuroscience of affect: brain mechanisms of pleasure and displeasure. Current opinion in neurobiology, 23(3), 294-303.

Bertoli, A., Dafoe, A., & Trager, R. (2019). Leader Age and International Conflict.

Boddy, C. R., Ladyshewsky, R., & Galvin, P. (2010). Leaders without ethics in global business: Corporate psychopaths. Journal of Public Affairs, 10(3), 121-138.

Boddy, C. R. (2011). The corporate psychopaths theory of the global financial crisis. Journal of Business Ethics, 102(2), 255-259.

Book, A., Visser, B. A., & Volk, A. A. (2015). Unpacking “evil”: Claiming the core of the Dark Triad. Personality and Individual Differences, 73, 29-38.

Bostrom, N., & Ord, T. (2006). The reversal test: eliminating status quo bias in applied ethics. Ethics, 116(4), 656-679.

Bostrom, N. (2002). Existential risks: Analyzing human extinction scenarios and related hazards. Journal of Evolution and Technology, 9.

Bowers, K. S. (1973). Situationism in psychology: An analysis and a critique. Psychological review, 80(5), 307.

Bowler, S., & Karp, J. A. (2004). Politicians, scandals, and trust in government. Political Behavior, 26(3), 271-287.

Buckels, E. E., Jones, D. N., & Paulhus, D. L. (2013). Behavioral confirmation of everyday sadism. Psychological science, 24(11), 2201-2209.

Burger, J. M. (2009). Replicating Milgram: Would people still obey today?. American Psychologist, 64(1), 1.

Burkle, F. M. (2016). Antisocial personality disorder and pathological narcissism in prolonged conflicts and wars of the 21st century. Disaster medicine and public health preparedness, 10(1), 118-128.

Button, K. S., Ioannidis, J. P., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S., & Munafò, M. R. (2013). Power failure: why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14(5), 365-376.

Byman, D. L., & Pollack, K. M. (2001). Let us now praise great men: Bringing the statesman back in. International Security, 25(4), 107-146.

Campbell, W. K., Goodie, A. S., & Foster, J. D. (2004). Narcissism, confidence, and risk attitude. Journal of behavioral decision making, 17(4), 297-311.

Campbell, W. K., Rudich, E. A., & Sedikides, C. (2002). Narcissism, self-esteem, and the positivity of self-views: Two portraits of self-love. Personality and Social Psychology Bulletin, 28(3), 358-368.

Chang, J., & Halliday, J. (2007). Mao: The unknown story. Vintage.

Cislak, A., Cichocka, A., Wojcik, A. D., & Frankowska, N. (2018). Power corrupts, but control does not: What stands behind the effects of holding high positions. Personality and Social Psychology Bulletin, 44(6), 944-957.

Colgan, J. D. (2013). Domestic revolutionary leaders and international conflict. World Politics, 65(4), 656-690.

Cooke, D. J., & Michie, C. (2001). Refining the construct of psychopathy: Towards a hierarchical model. Psychological assessment, 13(2), 171.

Dallago, F., Cima, R., Roccato, M., Ricolfi, L., & Mirisola, A. (2008). The correlation between right-wing authoritarianism and social dominance orientation: The moderating effects of political and religious identity. Basic and applied social psychology, 30(4), 362-368.

Deluga, R. J. (2001). American presidential Machiavellianism: Implications for charismatic leadership and rated performance. The Leadership Quarterly, 12(3), 339-363.

Derefinko, K. J., & Lynam, D. R. (2006). Convergence and divergence among self-report psychopathy measures: A personality-based approach. Journal of Personality Disorders, 20(3), 261-280.

Dikötter, F. (2011). Mao's great famine: The history of China's most devastating catastrophe, 1958-1962. Bloomsbury Publishing.

Dikötter, F. (2016). The Cultural Revolution: A People's History, 1962—1976. Bloomsbury Publishing.

Dikötter, F. (2019). Dictators: The Cult of Personality in the Twentieth Century. Bloomsbury Publishing.

Duspara, B., & Greitemeyer, T. (2017). The impact of dark tetrad traits on political orientation and extremism: an analysis in the course of a presidential election. Heliyon, 3(10), e00425.

Endler, N. S., & Magnusson, D. (1976). Toward an interactional psychology of personality. Psychological bulletin, 83(5), 956.

Fox, B., & DeLisi, M. (2019). Psychopathic killers: a meta-analytic review of the psychopathy-homicide nexus. Aggression and violent behavior, 44, 67-79.

Gallagher, M. E., & Allen, S. H. (2014). Presidential personality: Not just a nuisance. Foreign Policy Analysis, 10(1), 1-21.

Gao, Y., & Raine, A. (2010). Successful and unsuccessful psychopaths: A neurobiological model. Behavioral sciences & the law, 28(2), 194-210.

Glad, B. (2002). Why tyrants go too far: Malignant narcissism and absolute power. Political Psychology, 23(1), 1-2.

Gluck, M., Heesacker, M., & Choi, H. D. (2020). How much of the dark triad is accounted for by sexism? Personality and Individual Differences, 154, 109728.

Gøtzsche-Astrup, O. (2019). Partisanship and violent intentions in the United States.

Hanson, R. (2016). The Age of Em: Work, Love, and Life when Robots Rule the Earth. Oxford University Press.

Hare, R. D. (2003). Manual for the Hare Psychopathy Checklist-Revised. Toronto, ON, Canada: Multi-Health Systems.

Hare, R D., et al. (1990). "The revised psychopathy checklist: reliability and factor structure." Psychological Assessment: A Journal of Consulting and Clinical Psychology 2, no. 3: 338.

Harris, G. T., & Rice, M. E. (2006). Treatment of psychopathy. Handbook of psychopathy, 555-572.

Haslam, N., Holland, E., & Kuppens, P. (2012). Categories versus dimensions in personality and psychopathology: a quantitative review of taxometric research. Psychological medicine, 42(5), 903-920.

Hershman, D. J., & Lieb, J. (1994). A brotherhood of tyrants: Manic depression and absolute power. Amherst, NY: Prometheus

Hollander, P. (2016). From Benito Mussolini to Hugo Chavez. Cambridge University Press.

Hollander, P. (2017). Political pilgrims: Western intellectuals in search of the good society. Routledge.

Hosker-Field, A. M., Molnar, D. S., & Book, A. S. (2016). Psychopathy and risk taking: Examining the role of risk perception. Personality and Individual Differences, 91, 123-132.

Jain, K., & Bearden, J. N. (2011). Machiavellianism and overconfidence.

Johnson, A. M., Vernon, P. A., & Feiler, A. R. (2008). Behavioral genetic studies of personality: An introduction and review of the results of 50+ years of research. The SAGE handbook of personality theory and assessment, 1, 145-173.

Jonason, P. K., Duineveld, J. J., & Middleton, J. P. (2015). Pathology, pseudopathology, and the Dark Triad of personality. Personality and Individual Differences, 78, 43-47.

Jonason, P. K., Slomski, S., & Partyka, J. (2012). The Dark Triad at work: How toxic employees get their way. Personality and individual differences, 52(3), 449-453.

Jones, B. F., & Olken, B. A. (2005). Do leaders matter? National leadership and growth since World War II. The Quarterly Journal of Economics, 120(3), 835-864.

Jones, D. N., & Figueredo, A. J. (2013). The core of darkness: Uncovering the heart of the Dark Triad. European Journal of Personality, 27(6), 521-531.

Jones, D. N. (2013). Psychopathy and machiavellianism predict differences in racially motivated attitudes and their affiliations. Journal of Applied Social Psychology, 43, E367–E378. doi:10.1111/jasp.12035

Kajonius, P. J., Persson, B. N., & Jonason, P. K. (2015). Hedonism, achievement, and power: Universal values that characterize the dark triad. Personality and Individual Differences, 77, 173-178.

Kline, P., & Cooper, C. (1984). A construct validation of the Objective-Analytic Test Battery (OATB). Personality and Individual Differences, 5(3), 323-337.

Knopik, Neiderhiser, DeFries, Plomin (2018). Behavioral genetics. 7th edition. Macmillan Learning. New York.

Koehn, M. A., Jonason, P. K., & Davis, M. D. (2019). A person-centered view of prejudice: The Big Five, Dark Triad, and prejudice. Personality and Individual Differences, 139, 313-316.

Larson, D. W. (1988) Problems of Content Analysis in ForeignPolicy Research. International Studies Quarterly 32 (2): 241–255.

Lee, K., Ashton, M. C., Wiltshire, J., Bourdage, J. S., Visser, B. A., & Gallucci, A. (2013). Sex, power, and money: Prediction from the Dark Triad and Honesty–Humility. European Journal of Personality, 27(2), 169-184.

Levenson, M. R., Kiehl, K. A., & Fitzpatrick, C. M. (1995). Assessing psychopathic attributes in a noninstitutionalized population. Journal of personality and social psychology, 68(1), 151.

Lilienfeld, S. O., & Andrews, B. P. (1996). Development and preliminary validation of a self-report measure of psychopathic personality traits in a noncriminal population. Journal of personality assessment, 66(3), 488-524.

Lilienfeld, S. O., Latzman, R. D., Watts, A. L., Smith, S. F., & Dutton, K. (2014). Correlates of psychopathic personality traits in everyday life: Results from a large community survey. Frontiers in psychology, 5, 740.

Lilienfeld, S. O., Waldman, I. D., Landfield, K., Watts, A. L., Rubenzer, S., & Faschingbauer, T. R. (2012). Fearless dominance and the US presidency: Implications of psychopathic personality traits for successful and unsuccessful political leadership. Journal of personality and social psychology, 103(3), 489.

Lilienfeld, S. O., Watts, A. L., & Smith, S. F. (2015). Successful psychopathy: A scientific status report. Current Directions in Psychological Science, 24(4), 298-303.

MacAskill, W. (2020). Will MacAskill on the moral case against ever leaving the house, whether now is the hinge of history, and the culture of effective altruism. 80000 Hours Podcast. Jan 24, 2020. https://80000hours.org/podcast/episodes/will-macaskill-paralysis-and-hinge-of-history/

Macenczak, L. A., Campbell, S., Henley, A. B., & Campbell, W. K. (2016). Direct and interactive effects of narcissism and power on overconfidence. Personality and Individual Differences, 91, 113-122.

Marcus, D. K., Preszler, J., & Zeigler-Hill, V. (2018). A network of dark personality traits: What lies at the heart of darkness?. Journal of Research in Personality, 73, 56-62.

Matthews, G., Reinerman-Jones, L. E., Burke, C. S., Teo, G. W., & Scribner, D. R. (2018). Nationalism, personality, and decision-making. Personality and Individual Differences, 127, 89-100.

McDonald, M. M., Donnellan, M. B., & Navarrete, C. D. (2012). A life history approach to understanding the Dark Triad. Personality and Individual Differences, 52(5), 601-605.

Međedović, J., & Knežević, G. (2018). Dark and Peculiar. Journal of Individual Differences.

Milgram, S. (1963). Behavioral study of obedience. The Journal of abnormal and social psychology, 67(4), 371.

Moshagen, M., Hilbig, B. E., & Zettler, I. (2018). The dark core of personality. Psychological Review, 125(5), 656.

Mullins-Sweatt, S. N., Glover, N. G., Derefinko, K. J., Miller, J. D., & Widiger, T. A. (2010). The search for the successful psychopath. Journal of Research in Personality, 44(4), 554-558.

Nai, A., & Toros, E. (2020). The peculiar personality of strongmen: comparing the Big Five and Dark Triad traits of autocrats and non-autocrats. Political Research Exchange, 2(1), 1-24.

Nai, A. (2019). Disagreeable narcissists, extroverted psychopaths, and elections: A new dataset to measure the personality of candidates worldwide. European Political Science, 18(2), 309-334.

Nell, V. (2006). Cruelty's rewards: The gratifications of perpetrators and spectators. Behavioral and Brain Sciences, 29(3), 211-224.

O'Connell, D., & Marcus, D. K. (2016). Psychopathic personality traits predict positive attitudes toward sexually predatory behaviors in college men and women. Personality and Individual Differences, 94, 372-376.

Obradović, J., Pardini, D. A., Long, J. D., & Loeber, R. (2007). Measuring interpersonal callousness in boys from childhood to adolescence: An examination of longitudinal invariance and temporal stability. Journal of Clinical Child and Adolescent Psychology, 36(3), 276-292.

Paulhus, D. L., & Williams, K. M. (2002). The dark triad of personality: Narcissism, Machiavellianism, and psychopathy. Journal of research in personality, 36(6), 556-563.

Paulhus, D. L. (2014). Toward a taxonomy of dark personalities. Current Directions in Psychological Science, 23(6), 421-426.

Perina, K., Hsu, S., Washington, C. (2020). Manifold Podcast. https://manifoldlearning.com/episode-036/.

Plomin, R. (2019). Blueprint: How DNA makes us who we are. MIT Press.

Plouffe, R. A., Saklofske, D. H., & Smith, M. M. (2017). The assessment of sadistic personality: Preliminary psychometric evidence for a new measure. Personality and individual differences, 104, 166-171.

Post, J. M. (2003). Assessing leaders at a distance: The political personality profile. The psychological assessment of political leaders: With profiles of Saddam Hussein and Bill Clinton, 69-104.

Pratto, F., Sidanius, J., Stallworth, L. M., & Malle, B. F. (1994). Social dominance orientation: A personality variable predicting social and political attitudes. Journal of personality and social psychology, 67(4), 741.

Preston, O. C., & Anestis, J. C. (2018). Psychopathic traits and politics: Examining affiliation, support of political issues, and the role of empathy. Personality and individual differences, 131, 142-148.

Ronson, J. (2012). The psychopath test: A journey through the madness industry. Riverhead Books.

Shulman, C., & Bostrom, N. (2014). _Embryo Selection for Cognitive Enhancement: Curiosity or Game-changer? Global Policy, 5(1), 85–92._doi:10.1111/1758-5899.12123.

Southard, A. C., & Zeigler-Hill, V. (2016). The dark triad traits and fame interest: Do dark personalities desire stardom?. Current Psychology, 35(2), 255-267.

Sparrow, R. (2013). In vitro eugenics. Journal of Medical Ethics, 40(11), 725-731.

Taylor, S. (2019). Pathocracy. When people with personality disorders gain power. Psychology Today.

Toland, J. (1976). Adolf Hitler. New York: Doubleday.

Tomasik, B. (2013a). Charity Cost-Effectiveness in an Uncertain World. Foundational Research Institute.

Tomasik, B. (2013b). Differential Intellectual Progress as a Positive-Sum Project. Foundational Research Institute.

Ullrich, S., Borkenau, P., & Marneros, A. (2001). Personality disorders in offenders: categorical versus dimensional approaches. Journal of Personality Disorders, 15(5), 442-449.

Vernon, P. A., Villani, V. C., Vickers, L. C., & Harris, J. A. (2008). A behavioral genetic investigation of the Dark Triad and the Big 5. Personality and individual Differences, 44(2), 445-452.

Visser, B. A., Pozzebon, J. A., & Reina-Tamayo, A. M. (2014). Status-driven risk taking: Another “dark” personality?. Canadian Journal of Behavioural Science, 46(4), 485.

Vukasović, T., & Bratko, D. (2015). Heritability of personality: a meta-analysis of behavior genetic studies. Psychological bulletin, 141(4), 769.

Yergin, D. (1977). Shattered peace. Boston: Houghton Mifflin Company.

  1. Of course, assessing other people’s personality is always fraught with uncertainty, especially if they are long dead. ↩︎

  2. Other researchers have suggested similar constructs aimed to represent the common core of “evil” (e.g., Book et al., 2015; Jones & Figueredo, 2013; Marcus et al., 2018). ↩︎

  3. Baron-Cohen (2012) argues that the defining feature of human evil is “zero degrees of empathy.” However, some psychopaths can read other people extremely well and would thus score highly on certain items of the empathy questionnaires Baron-Cohen describes in his book. Furthermore, as Baron-Cohen acknowledges, people on the autism spectrum tend to have less empathy—at least certain forms of it—but they are not more malevolent than the population average. Therefore, reducing malevolence to “zero degrees of empathy” could be problematic or at least crucially depends on how we define and operationalize empathy. ↩︎

  4. As of now, there is no established treatment of malevolence. Harris and Rice (2006) review the empirical findings on the treatment of psychopathy but are quite pessimistic about their effectiveness. ↩︎

  5. A more rigorous analysis would be valuable, though it would also be methodologically challenging—assessing the personality traits of historical figures, for example, is rather difficult. ↩︎

  6. One reason to be hesitant here is that it seems plausible, for instance, that we, as well as journalists and historians, see more signs of malevolent personality traits in leaders who have caused great harm, and will tend to overlook malevolent personality traits in leaders who have done more good. ↩︎

  7. Again, we refer to Appendix A for more details. ↩︎

  8. For example, without Mao and Stalin the probability of a communist China is smaller. A non-communist China may have better relations with the U.S., and the probability of great power wars and (AI) arms races may be reduced. However, such claims are necessarily very speculative. For instance, one could also argue that World War II may have increased longer-term stability by leading to the formation of the UN. ↩︎

  9. Or, as Robert Hare, one of the most well-known researchers of psychopathy, puts it: “Serial killer psychopaths ruin families. Corporate, political and religious psychopaths ruin economies [and] societies.” (Ronson, 2012, p. 117). ↩︎

  10. Gallagher & Allen (2014) found that U.S. presidents scoring higher on the Big Five facet “altruism” were less likely to employ military force. ↩︎

  11. Also compare MacAskill: “I still endorse the view of pushing resources into the future. The biggest caveat actually I’d have is about the rise of fascism and Stalinism as the thing to push on [...] even though you might not think that a particular ideology will last forever, well, if it lasts as long until you get like some eternal lock-in event, then it lasts forever. [...] I kind of think the rise of fascism and Stalinism was a bigger deal in the 20th century than the invention of nuclear weapons.” (MacAskill, 2020). ↩︎

  12. Dark Triad traits in political candidates correlate with more negative campaigns and fear appeals (Nai, 2019). ↩︎

  13. Since Dark Triad traits correlate with social dominance orientation (SDO, Jones & Figueredo, 2013), malevolent leaders will, on average, exhibit higher SDO and prefer policies resulting in higher social and economic inequality. ↩︎

  14. Azizli et al. (2016) find that psychopathy and Machiavellianism are associated with a greater propensity to lie and engage in high-stakes deception. ↩︎

  15. Bowler & Karp (2004) find that scandals involving politicians tend to lower political trust. It seems plausible that malevolent political leaders are more likely to be involved in scandals. ↩︎

  16. According to Ulrich et al. (2001), the rate of antisocial personality disorder among criminal offenders, 45% of whom were convicted of “robbery or extortion,” is more than 10 times higher than that of the control sample. Jonason et al. (2012) also find that Machiavellianism, narcissism, and psychopathy all correlate with the use of “hard tactics” in the workplace, including “threats of punishment” (see Table 1, p. 451). ↩︎

  17. MacAskill (2020, emphasis added): “[...] when you look at history of well what are the worst catastrophes ever? They fall into three main camps: pandemics, war and totalitarianism. Also, totalitarianism or, well, autocracy has been the default mode for almost everyone in history. And I get quite worried about that. So even if you don’t think that AI is going to take over the world, well it still could be some individual. And if it is a new growth mode, I do think that very significantly increases the chance of lock-in technology.” ↩︎

  18. Kaja Perina on the Manifold podcast (Perina, 2020): “[M]ost of the studies on psychopaths [...] are done on inmates. For that reason, we’re forced to conjecture about the really successful ones because I think the more successful, the more they evade detection, perhaps, lifelong. So there is this disconnect wherein a lot of them, the violent ones, the less intelligent ones, really end up in jail, and these are the ones who are studied, but these are not the ones who are highly Machiavellian, necessarily, these are not the ones who are brilliantly manipulative. These are the ones who are committing violent crimes and get caught.” ↩︎

  19. Extensive background checks, for example with the help of private investigators, would be another promising possibility. Intelligence agencies do this already for somewhat related purposes. Generally, the competitive nature of the political process can often uncover past immoral behavior—though swaying partisan views seems to require evidential strength that is difficult to achieve. Thanks to Mojmír Stehlík for raising these points. ↩︎

  20. Yet another possibility would be to use “objective” personality tests that don’t rely on self- or other-report but use actual performance tests to evaluate personality traits (without the test-taker knowing which trait is supposed to be measured). However, according to our cursory reading of the literature, the few “objective” personality tests that exist seem to have low validity (e.g., Kline & Cooper, 1984). ↩︎

  21. However, one needs examples to train such predictors in the first place. One could start by looking for differences in the brains of normal people and, say, diagnosed psychopaths, but this metric will be biased towards diagnosed psychopaths who are at least somewhat unrepresentative of non-diagnosed malevolent humans (as explained above). One needs to correct for this ascertainment bias. ↩︎

  22. Relatedly, neuroscience research is often underpowered, resulting in low reproducibility of the accumulated findings (Button et al., 2013). ↩︎

  23. However, the education system also involves a lot of tests and grading; and is at least somewhat related to career advancement. Such tests are also common for military entry and sometimes civil service. In the context of elections, the key question is why voters do not generally demand such tests (including related objective measures, such as tax returns). ↩︎

  24. However, there exists the frightening possibility that some voters want their political leaders to be at least moderately malevolent. Most Russians and Chinese, for example, seem to think highly of Stalin and Mao, respectively—though this is likely at least partly due to propaganda. Generally, many voters seem to like “strong men” like Putin and overlook or even appreciate elevated Dark Tetrad traits in their political leaders. Also, according to the (potentially biased) assessment of “experts”, politicians with autocratic tendencies—many of whom nonetheless received the majority of votes—score significantly higher on Dark Triad traits than the average politician (Nai & Toros, 2020). ↩︎

  25. It is also worth noting that in some forms of government, such as allocating political positions to randomly selected individuals or hereditary monarchy, those in positions in power are exactly as likely to be malevolent as the population at large. This may be better than fierce competition for positions of power if the latter advantages the most ruthless and malevolent individuals. On the other hand, good selection procedures could also reduce malevolence in positions of power below the baseline; and of course this is only one consideration among many when evaluating different forms of government. ↩︎

  26. Thanks to Richard Ngo for making this point. ↩︎

  27. However, initial distributions may change due to competitive pressures or other factors. Even if none of the first ems are malevolent, there is no guarantee that malevolence will remain absent in the long run. ↩︎

  28. Extreme events like severe abuse or violence can make a huge difference for the victims, but such events are relatively rare and therefore do not explain much variance in the general population (Plomin, 2019). ↩︎

  29. Also note that shifting personality traits by more than this would likely be very difficult even if one wanted to do this. ↩︎

  30. Some dark traits, such as Machiavellianism, can be beneficial under certain circumstances. It might be better if one could single out a dark trait, such as sadism, and only select against it while leaving other dark traits unchanged. ↩︎

  31. However, GWA studies of personality are still fairly weak even at such scales. Even higher sample sizes might be achieved by identifying proxy variables of malevolence, such as public records on crime. However, this could easily backfire and cause great harm in numerous ways, so one would have to be very careful. ↩︎

  32. Currently, Dark Tetrad traits seem to be neglected in many relevant areas. Most sperm banks measure traits such as height, attractiveness, physical and mental health but not Dark Tetrad traits. Services that offer pre-implementation diagnostics screen for all sorts of genetic diseases and some even for IQ but not for Dark Tetrad traits. Test batteries of enormous government projects like the UK Biobank measure thousands of variables—including physical health, height, and preferred coffee and cereal type—but they don’t measure Dark Tetrad traits or even most personality traits in any sort of rigorous manner. ↩︎

  33. Generally, reducing the influence of malevolent actors can be done in myriad (often mutually reinforcing) ways. For instance, the more widespread belief systems are that put great value on non-malevolent traits such as compassion and altruism, the more parents might demand screening for malevolent traits, and the more future (government) projects will include measures of malevolence in their test batteries. ↩︎

  34. Adopting certain ideologies could also make one more malevolent. However, we think it’s plausible that most of the correlation is explained by causation from malevolent traits to dangerous ideologies, partly because personality traits seem less amenable to change than beliefs. ↩︎

  35. The Dark Triad also predicts sexism (Gluck et al., 2020; O'Connell & Marcus, 2016), nationalism (Matthews et al., 2018) as well as cognitive and affective prejudice (Koehn et al., 2019). Psychopathic traits predict opposition towards free speech and animal rights as well as support for using war as a tool for diplomacy (Preston & Anestis, 2018, Table 3). Machiavellianism and narcissism also seem to correlate with overconfidence (Campbell et al., 2002; 2004; Macenczak et al., 2016; Jain & Bearden, 2011), and thus plausibly negatively correlate with epistemic humility which should serve a protective function against all kinds of extremism and fanaticism. (Note that we don’t intend to convey that all these associations are equally dangerous.) ↩︎

  36. Many of these and similar quotes could have been made solely for political reasons, e.g., to strengthen alliances. ↩︎

weeatquince @ 2020-05-03T08:50 (+53)

Hi, interesting article. Thank you for writing.

I felt that this article could have said more about possible policy interventions and that it dismisses policy and political interventions as crowded too quickly. Having thought a bit about this area in the past I thought I would chip in.

MORE ON POLICY INTERVENTIONS

Even within established democracies, we could try to identify measures that avoid excessive polarization and instead reward cross-party cooperation and compromise. ... (For example, effective altruists have discussed electoral reform as a possible lever that could help achieve this.)

There are many things that could be done to prevent malevolent leaders within established democracies. Reducing excessive polarization (or electoral reform) are two minor ones. Other ideas you do not discuss include:

ARE POLITICAL INTERVENTIONS CROWDED?

Alternatively, we could influence political background factors that make malevolent leaders more or less likely... interventions to promote democracy and reduce political instability seem valuable—though this area seems rather crowded.

You might be correct, but this feels a bit like saying the AI safety space is crowded because lots of groups are trying to develop AI. However it may not be the case that those groups are focusing as much on safety as you would like. Although there are many groups (especially nation states) that want to promote democracy there may be very specific interventions that prevent malevolent leaders that are significantly under-discussed, such as elections for parties rather than leaders, or other points listed above. It seems plausible that academics and practitioners in this space may be able to make valuable shifts in the way fledgling democracies are developing that are not otherwise being considered.

And as someone in the improving government institutions space in the UK is is not evident to me that there is much focus on the kinds of interventions that would limit malevolent leaders.

David_Althaus @ 2020-05-04T17:35 (+24)

Thank you, excellent points!

I will probably add some of your intervention ideas to the article (I'll let you know in that case).

I felt that this article could have said more about possible policy interventions and that it dismisses policy and political interventions as crowded too quickly.

Sorry about that. It certainly wasn’t our intention to dismiss political interventions out of hand. The main reason for not writing more was our lack of knowledge in this space; which is why our discussion ends with “We nevertheless encourage interested readers to further explore these topics”. In fact, a comment like yours—containing novel intervention ideas written by someone with experience in policy—is pretty much what we were hoping to see when writing that sentence.

Better mechanisms for judging individuals. Eg ensuring 360 feedback mechanisms are used routinely to guide hiring and promotion decisions as people climb political ladders. (I may do work on this in the not too distant future)

Very cool! This is partly what we had in mind when discussing manipulation-proof measures to prevent malevolent humans from rising to power (where we also briefly mention 360 degree assessments).

For what it's worth, Babiak et al. (2010) seemed to have some success with using 360 degree assessments to measure psychopathic traits in a corporate setting. See also Mathieu et al. (2013).

weeatquince @ 2020-08-06T11:30 (+4)

Relevant policy report from the UK Parliament on enforcing the Ministerial Code of good behaviour, (from 2006): https://publications.parliament.uk/pa/cm200506/cmselect/cmpubadm/1457/1457.pdf

(I wasn't sure what to do with this when I found it, I might add other policy reports I find to this thread too until I have the capacity to actually work on this in any detail)

Less directly relevant but somewhat interesting too: https://publications.parliament.uk/pa/cm200607/cmselect/cmpubadm/121/121i.pdf

MichaelA @ 2020-05-06T06:38 (+10)

Thanks for sharing these additional ideas and insights!

Also removing the extent to which dishonesty pays, for example with better fact-checking services.

I ran (and published a paper on) an experiment on fact-checking for my Psychology Honours in 2017, so I have a smidge of knowledge here, though it's a bit rusty. Some brief thoughts:

  • I do suspect "better fact-checking services", in the sense of more accuracy or checking more facts, would be somewhat beneficial
  • But I think there's some reason for pessimism about just how much people really "care" about the results of fact-checks, even when they see those fact-checks. Here's my study's abstract:
In the ‘post-truth era’, political fact-checking has become an issue of considerable significance. A recent study in the context of the 2016 US election found that fact-checks of statements by Donald Trump changed participants' beliefs about those statements—regardless of whether participants supported Trump—but not their feelings towards Trump or voting intentions. However, the study balanced corrections of inaccurate statements with an equal number of affirmations of accurate statements. Therefore, the null effect of fact-checks on participants’ voting intentions and feelings may have arisen because of this artificially created balance. Moreover, Trump's statements were not contrasted with statements from an opposing politician, and Trump's perceived veracity was not measured. The present study (N = 370) examined the issue further, manipulating the ratio of corrections to affirmations, and using Australian politicians (and Australian participants) from both sides of the political spectrum. We hypothesized that fact-checks would correct beliefs and that fact-checks would affect voters’ support (i.e. voting intentions, feelings and perceptions of veracity), but only when corrections outnumbered affirmations. Both hypotheses were supported, suggesting that a politician's veracity does sometimes matter to voters. The effects of fact-checking were similar on both sides of the political spectrum, suggesting little motivated reasoning in the processing of fact-checks.
    • This is also relevant to the following sentence from the original post: "While the notion of Dark Tetrad traits is not foremost in most people’s minds, one could argue that much political debate is about related concepts like the trustworthiness or honesty of candidates, and voters do value those attributes." I think this is true, but probably less true than many might think (or at least than they would've thought pre-2016).
  • And then there's also the matter of whether people come to encounter fact-checks in the first place. In my study's conclusion, I wrote that "participants were unable to avoid fact-checks or to select which ones they received. In reality, some people may not encounter any fact-checks at all [9], and the sample of fact-checks which others encounter is often influenced by selective exposure and selective sharing [65,66]." (I feel weird about quoting myself, but 2017 Michael knew more about this than 2020 Michael does!)
  • So I'd tentatively see more value in making fact-checking services "better" in the sense of being clearer, more attention-grabbing, better publicised, or things like that (as long as this doesn't cost too much accuracy, nuance, etc.), rather than in e.g. making more or more accurate fact-checks.
    • And there may be still more value in somehow "shifting norms" towards valuing truth more highly, or things like that, though I don't know how one would actually do that. (I'm guessing this post is relevant, but I haven't read it yet.)
Jc_Mourrat @ 2020-05-13T00:32 (+6)

I've just read the results of an interesting new study on the effect of red-flagging some information on social media, with flags such as "Multiple fact-checking journalists dispute the credibility of this news", and variations with "Multiple fact-checking journalists" replaced by, alternatively, "Major news outlets", "A majority of Americans", or "Computer algorithms using AI". The researchers tested the effect this had on the propensity of people to share the content. The effect of the "fact-checking" phrasing was the most pronounced, and very significant (a reduction of about 40% of the probability to share content; which jumps to 60% for people who identify as Democrats). Overall the effect of the "AI" phrasing was also very significant, but quite counterintuitively it has the effect of increasing the probability of sharing content for people who identify as Republicans! (By about 8%; it decreases that same probability by 40% for people who identify as Democrats.)
https://engineering.nyu.edu/news/researchers-find-red-flagging-misinformation-could-slow-spread-fake-news-social-media

weeatquince @ 2020-05-06T07:19 (+5)

Thank you for the insight. I really have no strong view on how useful each / any of the ideas I suggested were. They were just ideas.

I would add on this point that narcissistic politicians I have encountered worried about appearance and bad press. I am pretty sure that transparency and fact checking etc discouraged them from making harmful decisions. Not every narcissistic leader is like Trump.

MichaelA @ 2020-05-06T12:03 (+3)

Yeah, that sounds right to me. And reminds me of a paper I read when working on that experiment, the abstract of which was:

Does external monitoring improve democratic performance? Fact-checking has come to play an increasingly important role in political coverage in the United States, but some research suggests it may be ineffective at reducing public misperceptions about controversial issues. However, fact-checking might instead help improve political discourse by increasing the reputational costs or risks of spreading misinformation for political elites. To evaluate this deterrent hypothesis, we conducted a field experiment on a diverse group of state legislators from nine U.S. states in the months before the November 2012 election. In the experiment, a randomly assigned subset of state legislators was sent a series of letters about the risks to their reputation and electoral security if they were caught making questionable statements. The legislators who were sent these letters were substantially less likely to receive a negative fact-checking rating or to have their accuracy questioned publicly, suggesting that fact-checking can reduce inaccuracy when it poses a salient threat.

Relatedly, it could be that "more" or "better" fact-checking would lead to better actions by or discourse from politicians, even if voters "don't really care much" about fact-checks or never really see them, due to politicians overestimating what impact fact-checks would have on voters' perceptions.

(To be clear, I do think fact-checks probably have at least some impact via the more obvious route too; I wonder mostly about the magnitude of the effect, not whether it exists.)

Ramiro @ 2020-04-29T18:50 (+21)

Personally, I think this is gold.

On the other hand, I’m not so sure that sadism is particularly worse for the long-term future than the dark triad traits. Yes, sadistic leaders may pose a special S-risk, but I am not sure they increase other x-risks so much – e.g., Hitler was kind of cautious concerning the risk of nuclear weapons igniting the atmosphere, and Stalin was partially useful in WW II and avoided WW III.

Contrast:

Some dark traits, such as Machiavellianism, can be beneficial under certain circumstances. It might be better if one could single out a dark trait, such as sadism, and only select against it while leaving other dark traits unchanged.

with these parts of the Appendix A:

Dark Triad traits predict increased intention to engage in political violence (Gøtzsche-Astrup, 2019)…
Historical evidence suggests that even many of their political adversaries—at least for some time—did not realize that Hitler, Mao, and Stalin were malevolent

Actually, open sadism often raises opposition; and I think the big 3 above weren't salient because they were sadistic – would they have killed less people if they were just functional psychopaths?

It might be hard to spot dark tetrad individuals, but it’s not so hard to realize an individual is narcissistic or manipulative. I don’t think people contend that Stalin and Mao were so – they may argue that they were beneficial, or that they were excused because they were “great man”.

So why do such guys acquire power? Why do people support it? We often dislike acquaintances that exhibit one of the dark traits; then why do people tolerate it when it comes from the alpha male boss?

My point is that the problem might be precisely that we often claim things like “Some dark traits, such as Machiavellianism, can be beneficial under certain circumstances. It might be better if one could single out a dark trait, such as sadism, and only select against it while leaving other dark traits unchanged.” If we could just acknowledge that dark triad traits individuals are very dangerous, even if they’re sometimes useful (like a necessary evil), then perhaps we could avoid (or at least be particularly cautious with) malevolent leaders.

David_Althaus @ 2020-04-30T12:19 (+18)

Thank you, good points.

I agree that it’s not clear whether sadism is really worse than the Dark Triad traits.

e.g., Hitler was kind of cautious concerning the risk of nuclear weapons igniting the atmosphere,

I’m not sure whether Hitler and Stalin were more sadistic than they were Machiavellian or narcissistic. If I had to decide, I’d say that their Machiavellianism and narcissism were more pronounced than their sadism.

Regarding Hitler being “kind of cautious”: It seems plausible to me that Hitler was less cautious than a less malevolent counterfactual German leader.

What is more, it seems not too unlikely that Hitler would have used nuclear weapons in 1945 if he had had access to them. For instance, see this quote from Glad (2002, p. 32):

As the Allies closed in on Berlin, [Hitler] wanted churches, schools, hospitals, livestock, marriage records, and almost anything else that occurred to him to be destroyed. In April 1945 he wanted the entire leadership of the Luftwaffe to be summarily hanged. He considered bringing about the destruction of German cities by announcing the execution of all Royal Air Force war prisoners, so that there would be massive bombing in reprisal. He may also have given orders for all wounded German soldiers to be killed. His aim became the destruction of Germany in the greatest Gotterdaemmerung of history. As Albert Speer said about Hitler's obsession with architecture: "Long before the end I knew that Hitler was not destroying to build, he was building to destroy.

You write:

and Stalin was partially useful in WW II and avoided WW III.

I'm not sure about that. Stalin was kind of Hitler’s ally until Hitler invaded Russia. In other words, if it weren’t for Hitler (and his overconfidence), Stalin might have even helped Hitler to win WW II (though Hitler might have lost WW II even with Stalin's support, I don't know). I’m no historian but Stalin’s response to Hitler’s invasion also seems to have been rather disastrous (see also Glad, 2002, p.8) and might have ended up in Russia losing WW II.

Resources like Khlevniuk (2015) also suggest that Stalin, if anything, was detrimental to Russia’ military success, at least for the first years of the war.

It might be hard to spot dark tetrad individuals, but it’s not so hard to realize an individual is narcissistic or manipulative.

I generally agree but there also seem to exist exceptions. The most strategic individuals know that narcissism and manipulativeness can be very off-putting so they will actively try to hide these traits. Stalin, for example, often pretended to be humble and modest, in public as well as in his private life, at least to some success. My impression from reading the various books listed in the references is that Stalin and Mao managed to rise to the top, partly because they successfully deceived others into believing that they are more modest and less capable, ruthless, and strategic than they actually are.

Also, it’s often not clear where justified confidence ends and narcissism begins—certainly not to the admirers of the leader in question. Similarly, what some would see as benign and even necessary instances of being realistic and strategic, others describe as unprincipled and malicious forms of manipulativeness. (The Sanders supporters who don't want to vote for Biden come to mind here even though this example is only tangentially related and confounded by other factors.)

So why do such guys acquire power? Why do people support it? We often dislike acquaintances that exhibit one of the dark traits; then why do people tolerate it when it comes from the alpha male boss?

Good questions. Indeed, some people do not only seem to tolerate these traits in leaders, they seem to be actively drawn to such "strong men" who display certain Dark Tetrad characteristics. I have some theories for why this is but I don't fully understand the phenomenon.

If we could just acknowledge that dark triad traits individuals are very dangerous, even if they’re sometimes useful (like a necessary evil), then perhaps we could avoid (or at least be particularly cautious with) malevolent leaders.

I’m definitely sympathetic to this perspective.

One important worry here is that different movements/political parties seem to be in a social dilemma. Let’s assume—which seems relatively plausible—that leaders with a healthy dose of narcissism and Machiavellianism are, on average, really better at things like motivating members of their movement to work harder, inspiring onlookers to join their movement, creating and taking advantage of high-value opportunities, and so on. If one movement decides to exclude members who show even small signs of Dark Triad traits, it seems plausible that this movement will be at a disadvantage compared to other movements that are more tolerant regarding such traits. Conversely, movements that tolerate (or even value) Dark Triad traits might be even more likely to rise to power. It seems very important to avoid such outcomes.

I’m also somewhat worried about "witch hunts" fueled by simplistic conceptions of malevolence. I know of several individuals who exhibit at least some level of Machiavellian and narcissistic traits—many even say so themselves—but whom I’d still love to give more influence because I believe that they are “good people” who are aware of these traits and who channel these traits towards the greater good. (Admittedly, a supporter of, say, Mao might have said the same thing back in the days.)

Ramiro @ 2020-05-09T15:08 (+11)

Thanks for the remarks concerning Hitler and Stalin.

I think it might be quite valuable, for the project as a whole, to better understand why people are drawn to leaders with features they would not tolerate in peers, such as dark traits.

For one, it’s very plausible that, as you mentioned, the explanation is (a) dark traits are very useful – these individuals are (almost) the only ones with incentives enough to take risks, get things done, innovate, etc. Particularly, if we do need things like strategies of mutually assured destruction, then we need someone credibly capable of “playing hawk”, and it's arguably hard to believe nice people would do that. This hypothesis really lowers my credence in us decreasing x-risks by screening for dark traits; malevolent people would be analogous to nukes, and it’s hard to unilaterally get rid of them.

A competing explanation is that (b) they’re not that useful, they’re parasitical. Dark traits are uncorrelated with achievement, they just make someone better at outcompeting useful pro-social people, by, e.g., occupying their corresponding niches, or getting more publicity - and so making people think (due to representative bias) that bad guys are more useful than they are. That's plausible, too; for instance, almost no one outside EA and LW communities knows about Arkhipov and Petrov. If that’s the case, then a group could indeed unilaterally benefit from getting rid of malevolent / dark trait leaders.

(Maybe I should make clear that I don't have anything against dark triad traits individuals per se, and I'm as afraid of the possibility of witch hunts and other abuses as everyone else. And even if dark traits were uncorrelated with capacity for achievement, a group might deprise itself from a scarce resource by selecting against very useful dark trait individuals, like scientists and entrepreneurs)

Richard Möhn @ 2024-06-16T02:38 (+20)

Just a quick thought: Supporting (investigative) journalism is also a possible intervention. I think journalism is considered a pillar of democracy? Currently, Kelsey Piper's work on OpenAI/Sam Altman is a good example.

David_Althaus @ 2024-06-18T10:40 (+15)

Yes, I think investigative journalism (and especially Kelsey Piper's work on Altman & OpenAI) is immensely valuable. 

In general, I've become more pessimistic about technology-centric/ "galaxy-brained" interventions in this area and more optimistic about "down-to-earth" interventions like, for example, investigative journalism, encouraging whistleblowing (e.g. setting up prizes or funding legal costs), or perhaps psychoeducation / workshops on how to detect malevolent traits and what do when this happens (which requires, in part, courage / the ability to endure social conflict and being socially savvy, arguably not something that most EAs excel in). 

Richard Möhn @ 2024-06-22T09:35 (+6)

Down-to-earth interventions sounds better to me, too. I like all of the examples.

Kaj_Sotala @ 2020-04-30T11:01 (+18)
Malevolent humans with access to advanced technology—such as whole brain emulation or other forms of transformative AI—could cause serious existential risks and suffering risks.

Possibly relevant: Machiavellians Approve of Mind Upload Technology Directly and Through Utilitarianism (Laakasuo et al. 2020), though it mainly tested whether machiavellians express moral condemnation of mind uploading, rather than their interest directly.

In this preregistered study, we have two novel findings: 1) Utilitarian moral preferences are strongly and psychopathy is mildly associated with positive approval of MindUpload; and 2) that Machiavellianism – essentially a calculative self-interest related trait – is strongly associated with positive approval of Mind Upload, even after controlling for Utilitarianism and the previously known predictor of Sexual Disgust (and conservatism). In our preregistration, we had assumed that the effect would be dependent on Psychopathy (another Dark Triad personality dimension), rather than Machiavellianism. However, given how closely related Machiavellianism and Psychopathy are, we argue that the results match our hypothesis closely. Our results suggest that the perceived risk of callous and selfish individuals preferring Mind Upload should be taken seriously, as previously speculated by Sotala & Yampolskiy (2015)
Emanuele_Ascani @ 2020-05-01T15:39 (+17)

This is one of the best posts I've read here, wow.

One of the main things that concern me is that malevolent people could appropriate the concept of malevolence itself and start a witch hunt for people who have nothing to do with malevolence. This was passingly mentioned when acknowledging that political leaders could brand their opponents as malevolent. Overall I think this post makes a good job of outlining the pros and cons, but I just wanted to write this consideration in a comment because it has been somewhat prominent in my mind.

Tobias_Baumann @ 2020-05-02T16:14 (+8)

Thanks for the comment!

I would guess that having better tests of malevolence, or even just a better understanding of it, may help with this problem. Perhaps a takeaway is that we should not just raise awareness (which can backfire via “witch hunts”), but instead try to improve our scientific understanding and communicate that to the public, which hopefully makes it harder to falsely accuse people.

In general, I don’t know what can be done about people using any means necessary to smear political opponents. It seems that the way to address this is to have good norms favoring “clean” political discourse, and good processes to find out whether allegations are true; but it’s not clear what can be done to establish such norms.

sbehmer @ 2020-05-05T15:29 (+15)

Thanks, it's a very nice article on an important topic. If you're interested, there's a small literature in political economy called "political selection" (here's an older survey article) . As far as I know they don't focus specifically on the extreme lower tail of bad leaders, but they do discuss how different institutional features can lead to different types of people gaining power.

David_Althaus @ 2020-05-11T11:56 (+1)

Thanks for the pointer!

Ben_West @ 2020-05-05T19:15 (+14)

Thanks for this interesting article. Regarding malevolence among business leaders: my impression is that corporations have rewarded malevolence less over time.

E.g. in the early 1900s you had Frederick Taylor (arguably the most influential manager of the 20th century) describing his employees like:

one of the very first requirements for a man who is fit to handle pig iron as a regular occupation is that he shall be so stupid and so phlegmatic that he more nearly resembles in his mental make-up the ox than any other type.

Modern executives would never say this about their staff, and no doubt this is partly because what's said in the boardroom is different from what's said in public, but there is a serious sense in which credibly signaling prosocial behaviors towards your employees is useful. E.g. 80 years later you have Paul O'Neill, in almost exactly the same industry as Taylor, putting worker safety as his key metric, because he felt that people would work harder if they felt taken care of by the company.

My guess is that corporations which rely on highly skilled workers benefit more from prosocial executives, and that it's hard to pretend to be prosocial over a decades-long career, though certainly not impossible. So possibly one hard-to-fake measure of malevolence is whether you repeatedly succeed in a corporation where success requires prosociality.

David_Althaus @ 2020-05-08T11:33 (+5)

Thanks, that's a good example.

my impression is that corporations have rewarded malevolence less over time.

Yeah, I think that's probably true.

Just to push back a little bit, the pessimistic take would be that corporate executives simply have become better at signalling and public relations. Maybe also partly because the downsides of having bad PR are worse today compared to, say, the 1920s—back then, people were poorer and consumers didn't have the luxury to boycott companies whose bosses said something egregious; workers often didn't have the option to look for another job if they hated their boss, et cetera. Generally, it seems plausible to me that "humans seem to have evolved to emphasize signaling more in good times than in bad." (Hanson, 2009).

I wonder if one could find more credible signals of things like "caring for your employers", ideally in statistical form. Money invested in worker safety might be one such metric. Salary discrepancies between employees and corporate executives might be another one (which seems to have gotten way worse since at least the 1970s) though there are obviously many confounders here.

The decline in child labor might be another example of how corporations have rewarded malevolence less over time. In the 19th century, when child labor was common, some amount of malevolence (or at least indifference) was arguably beneficial if you wanted to run a profitable company. Companies run by people who refused to employ children for ethical reasons presumably went bankrupt more often given that they could not compete with companies that used such cheap labor. (On the other hand, it's not super clear what an altruistic company owner should have done. Many children also needed jobs in order to be able to buy various necessities—I don't know.)

Maybe this is simply an example of a more general pattern: Periods of history marked by poverty, scarcity, instability, conflict, and inadequate norms & laws will tend to reward or even require more malicious behavior, and the least ruthless will tend to be outcompeted (compare again Hanson's "This is the Dream Time", especially point 4).

Ben_West @ 2020-05-13T16:57 (+4)

I wonder if one could find more credible signals of things like "caring for your employers", ideally in statistical form. Money invested in worker safety might be one such metric.

That seems reasonable. Another possibility is looking at benefits, which have grown rapidly (though there are also many confounders here).

Something which I can't easily measure but seems more robust is the fraction of "iterated games". E.g. I would expect enterprise salespeople to be less malevolent than B2C ones (at least towards their customers), because successful enterprise sales relies on building relationships over years or decades. Similarly managers are often recruited and paid well because they have a loyal team who will go with them, and so screwing over that team is not in their self-interest.

caspar42 @ 2020-08-04T14:07 (+13)

Probably you're already aware of this, but the APA's Goldwater rule seems relevant. It states:

On occasion psychiatrists are asked for an opinion about an individual who is in the light of public attention or who has disclosed information about himself/herself through public media. In such circumstances, a psychiatrist may share with the public his or her expertise about psychiatric issues in general. However, it is unethical for a psychiatrist to offer a professional opinion unless he or she has conducted an examination and has been granted proper authorization for such a statement.

From the perspective of this article, this rule is problematic when applied to politicians and harmful traits. (This is similar to how the right to confidentiality has the Duty to Warn exception.) A quick Google Scholar search gives a couple of articles since 2016 that basically make this point. For example, see Lilienfeld et al. (2018): The Goldwater Rule: Perspectives From, and Implications for, Psychological Science.

Of course, the other important (more empirical than ethical) question regarding the Goldwater rule is whether "conducting an examination" is a necessary prerequisite for gaining insight into a person's alleged pathology. Lilienfeld et al. also address this issue at length.

Pivocajs @ 2020-04-29T15:16 (+13)

Some thoughts that occured to me while reading:

1) Research suggestion: From afar, malevolence-detection techniques seem like a better version of the already-existing tool of top-secret security clearance (or tests similar to it). I am not confident about this, but it already seems that if top-secret security clearance was a requirement for holding important posts, a lot of grief would be avoided (at least where I am from). Yet we generally do not use this tool. Why is this? I suspect that whatever the answer is, it will apply to malevolence-detection techniques as well.

2) Potential bottleneck: Suppose you succeed and develop 100% accurate malevolence-detection technique. I think that, by default, you would have trouble convincing people to use it. ("I mean, what if I score high on it? You know, I am keeping my dark side in check and I don't plan to become too influential either, so my malevolence doesn't really hurt anybody. But the other people don't know that! If I get branded as malevolent, nobody will talk to me ever, or hire me, or anything!") I conjecture that the impact of this agenda will be bottlenecked on figuring out how to leave the malevolent people a line of retreat; making sure that if you score high on this, the implications aren't that bad. I see three reasons for this:

a) non-malevolent people might not know they are non-malevolent, and hence be afraid of this,

b) malevolent-and-know-it people might have enough power to hinder this,

c) reasonable general concerns about any test like this getting out of hand.

3) Relatedly to (2), would it make sense to consider some alternative branding that more accurately suggests what you intend to do with the concept (and doesn't suggest other things)? Unwieldly suggestion, to illustrate what I mean: Being publicly known as "potentially too risky to be in a position of great power" indicates that you shouldn't be a president, but you might still have friends, a spouse, and a prestigeous job. Being publicly known as "malevolent", however, ... . (Also, it seems plausible that there are people who are malevolent, but do not endorse being so, similarly to how, I think, there are paedophiles who wish they weren't so.)

(Also, it might not be obvious from my nitpicking, but I really like the post, thanks for it :-).)

David_Althaus @ 2020-04-30T17:31 (+11)

(Also, it might not be obvious from my nitpicking, but I really like the post, thanks for it :-).)

Thank you. :) No worries, I didn't think you were nitpicking. I agree with many of your points.

[...] if top-secret security clearance was a requirement for holding important posts, a lot of grief would be avoided (at least where I am from). Yet we generally do not use this tool. Why is this? I suspect that whatever the answer is, it will apply to malevolence-detection techniques as well.

One worry with security clearances is that they tend to mostly screen for impulsive behaviors such as crime and drug use (at least, according to my limited understanding of how these security clearances work) and would thus often fail to detect more strategic malevolent individuals.

Also, your claim that “we generally do not use this tool [i.e., security clearances]” feels too strong. For example, 5.1 million Americans seem to have a security clearance. Sounds like a lot to me. (Maybe you had a different country in mind.)

I conjecture that the impact of this agenda will be bottlenecked on figuring out how to leave the malevolent people a line of retreat; making sure that if you score high on this, the implications aren't that bad.

Good point. I guess we weren’t sufficiently clear in the post about how we envision the usage of manipulation-proof measures of malevolence. My view is that their results should, as a general rule, not be made public and that individuals who are diagnosed as malevolent should not be publicly branded as such. (Similarly, my understanding is that if someone doesn’t get a top level security clearance because they, for instance, have a serious psychiatric disorder, they only don't get the job requiring the security clearance—it's not like the government makes their mental health problems public knowledge.)

My sense is that malevolent individuals should only be prevented from reaching highly influential positions like becoming senators, members of Congress, majors, CEOs of leading AGI companies, et cetera. In other words, the great majority of jobs would still be open to them.

Of course, my views on this issue are by no means set in stone and still evolving. I’m happy to elaborate on my reasons for preferring this more modest usage if you are interested.

Pivocajs @ 2020-04-30T22:33 (+1)
Of course, my views on this issue are by no means set in stone and still evolving. I’m happy to elaborate on my reasons for preferring this more modest usage if you are interested.

I think the more modest usage is reasonable choice.

Maybe you had a different country in mind. [regarding top-secret security clearance]

I am Czech. We do have the institute, and use it. But, as far as I know, our president doesn't have it, and a bunch of other people don't have it. (I.e., it seems that people who need secret information on a daily basis have it. But you don't need it for many other positions from which you could put pressure on people who have the clearance.)

Pablo @ 2022-01-06T12:35 (+12)

I'm surprised to see that this post hasn't yet been reviewed. In my opinion, it embodies many of the attributes I like to see in EA reports, including reasoning transparency, intellectual rigor, good scholarship, and focus on an important and neglected topic.

MichaelPlant @ 2020-05-07T10:03 (+9)

Thanks for this write-up, I thought it was really interesting and not something I'd ever considered - kudos!

I'll now hone in on the bit of this I think needs most attention. :)

It seems you think that one of the essential things is developing and using manipulation-proof measures of malevolence. If you were very confident we couldn't do this, how much of an issue would that be? I raise this because it's not clear to me how such measures could be created or deployed. It seems you have (1) self-reports, (2) other-reports, (3) objective metrics, e.g. brain scans. If I were really sneaky, I would just lie or not take the test. If I were really sneaky, I would be able to con others, at least for a long-time - perhaps until I was in power. Regarding objective measures, there will be 'Minority Report' style objections to actually using them in advance, even if they have high predictive power (which might be tricky as it relies on collecting good data, which seems to require the consent of the malevolent).

The area where I see this sort of stuff working best is in large organisations, such as civil services, where the organisations have control over who gets promoted. I'm less optimistic this could work for the most important cases, political elections, where there is not a system that can enforce the use of such measures. But it's not clear to me how much of an innovation malevolence tests are over the normal feedback processes used in large organisations. Even if they could be introduced in politics somehow, it's unclear how much of an innovation this would be: the public already try to assess politicians for these negative traits.

It might be worth adding that the reason the Myers-Brigg style personality tests are, so I hear, more popular in large organisations than the (more predictive) "Big 5" personality test is that Myers-Briggs has no ostensibly negative dimensions. If you pass round a Big-5 test, people might score highly on neuroticism or low on openness and get annoyed. If this is the case, which seems likely, I find it hard e.g. Google will insist that staff take a test they know will assess them on their malevolence!

As a test for the plausibility of introducing and using malevolence tests, notice that we could already test for psychopathy but we don't. That suggests there are strong barriers to overcome.

David_Althaus @ 2020-05-08T16:30 (+6)

Thank you, great points.

It seems you think that one of the essential things is developing and using manipulation-proof measures of malevolence. If you were very confident we couldn't do this, how much of an issue would that be?

I wouldn't say it's "essential"—influencing genetic enhancement would still be feasible—though it would certainly be a big problem.

Regarding objective measures, there will be 'Minority Report' style objections to actually using them in advance, even if they have high predictive power

Yes, that would be an issue.

The area where I see this sort of stuff working best is in large organisations, such as civil services, where the organisations have control over who gets promoted. I'm less optimistic this could work for the most important cases, political elections, where there is not a system that can enforce the use of such measures.

That seems true though it seems at least conceivable that voters will demand such measures in the future. (As an aside, you mention large organisations but it seems such measures could also be valuable when used in smaller (non-profit) organizations?)

But it's not clear to me how much of an innovation malevolence tests are over the normal feedback processes used in large organisations.

Yeah, true. I guess it's also a matter of how much (negative) weight you put on malevolent traits, how much of an effort you make to detect them, and how attentive you are to potential signs of malevolence—most people seem to overestimate their ability to detect (strategic) malevolence (at least I did so before reality taught me a lesson).

It might be worth adding that the reason the Myers-Brigg style personality tests are, so I hear, more popular in large organisations than the (more predictive) "Big 5" personality test is that Myers-Briggs has no ostensibly negative dimensions.

Interesting, that seems plausible! I've always been somewhat bewildered by its popularity.

If this is the case, which seems likely, I find it hard e.g. Google will insist that staff take a test they know will assess them on their malevolence!

True. I guess measures of malevolence would work best as part of the hiring process (i.e., before one has formed close relationships).

As a test for the plausibility of introducing and using malevolence tests, notice that we could already test for psychopathy but we don't. That suggests there are strong barriers to overcome.

I agree that there are probably substantial barriers to be overcome. On the other hand, it seems that many companies are using "integrity tests" which go in a similar direction. According to Sacket and Harris (1984), at least 5,000 companies used "honesty tests" in 1984. Companies were also often using polygraph examinations—in 1985, for example, about 1.7 million such tests were administered to (prospective) employees (Dalton & Metzger, 1993, p. 149)—until they became illegal in 1988. And this even though polygraph tests and integrity tests (as well as psychopathy tests) can be gamed (rather easily).

I could thus imagine that at least some companies and organizations would start using manipulation-proof measures of malevolence (which is somewhat similar to the inverse of integrity) if it was common knowledge that such tests actually had high predictive validity and could not be gamed.

Milan_Griffes @ 2020-05-07T19:21 (+2)
Regarding objective measures, there will be 'Minority Report' style objections to actually using them in advance, even if they have high predictive power (which might be tricky as it relies on collecting good data, which seems to require the consent of the malevolent).

You could imagine scenarios where we apply objective measures to children, embryos, and/or couples who are considering having children.

This would avoid some of the free-choice problem around who takes the test, though it doesn't really get around the Minority Report problem. Also there'd still be selection at the parent level (some would-be parents would decide to not get tested).

Interesting that we don't do anything like this for psychopathy currently, as far as I know. (Psychopathy appears to be somewhat genetic.)

JustinShovelain @ 2020-04-30T15:26 (+9)

Nice post!

Here are a couple additional posts that I think are worth checking out by Gwern:

https://www.lesswrong.com/posts/ktr39MFWpTqmzuKxQ/notes-on-psychopathy

https://www.lesswrong.com/posts/Ft2Cm9tWtcLNFLrMw/notes-on-the-psychology-of-power

David_Althaus @ 2020-04-30T16:15 (+7)

Thank you!

Agree, these posts are excellent. For what it's worth, I share Gwern's pessimistic conclusion about the treatment of psychopathy. Other Dark Tetrad traits—especially if they are less pronounced—might be more amenable to treatment though I'm not especially optimistic.

However, even if effective treatment options existed, the problem remains that the most dangerous individuals are unlikely to ever be motivated to seek treatment (or be forced to do so).

Milan_Griffes @ 2020-04-29T19:58 (+9)
In this section, we explore how possible future technologies could be used to reduce the influence of malevolent actors.

Psychedelics probably help with this as well (when carefully administered).

Anecdotally, a lot of Western contemplative teachers got started on that path because of psychedelic experiences (Zen, Tibetan Vajrayana, Vipassana, Advaita Vedanta, Kashmiri Shaivism). These traditions are extremely prosocial & anti-malevolent.

Less anecdotally, Hendricks et al. 2018 found lifetime psychedelic use correlated with reduced criminality (survey with n = 480,000). Lyons & Carhart-Harris 2018 found psilocybin decreased authoritarian political beliefs (small, open-label, prospective study).

There hasn't been a proper RCT yet though.

David_Althaus @ 2020-04-30T11:14 (+35)

There hasn't been a proper RCT yet though.

Pokorny et al. (2017) seems like a relevant RCT. They found that psilocybin significantly increased empathy.

However, even such results don’t make me very optimistic about the use of psychedelics for reducing malevolence.

The kind of individuals that seem most dangerous (people with highly elevated Dark Tetrad traits who are also ambitious, productive and strategic) seem less likely to be interested in taking psychedelics—such folks don't seem interested in increasing their empathy, becoming less judgmental or having spiritual experiences. In contrast, the participants of the Pokorny et al. study—like most participants in current psychedelics studies (I think)—wanted to take psychedelics which is why they signed up for the study.

Moreover, my sense is that psychedelics are most likely to increase openness and compassion in those who already started out with some modicum of these traits and who would like to increase them further. I’m somewhat pessimistic that giving psychedelics to highly malevolent individuals would make them substantially more compassionate. That being said, I'm certainly not confident in that assessment.

My intuition is partly based on personal experience and anecdotes but also more objective evidence like the somewhat disappointing results of the Concord Prison Experiment. However, due to various methodological flaws, I'd be hesitant to draw strong conclusions from this experiment.

Overall, I’d nevertheless welcome further research into psychedelics and MDMA. It would still be valuable if these pharmaceutical agents “only" increase empathy in individuals who are already somewhat empathic.

Anecdotally, a lot of Western contemplative teachers got started on that path because of psychedelic experiences (Zen, Tibetan Vajrayana, Vipassana, Advaita Vedanta, Kashmiri Shaivism). These traditions are extremely prosocial & anti-malevolent.

My guess is that most Western contemplative teachers who, as a result of taking psychedelics, got interested in Buddhism and meditation (broadly defined) were, on average, already considerably more compassionate, idealistic, and interested in spiritual questions than the type of practically oriented, ambitious, malevolent people I worry about.

As an aside, I'm much more optimistic about the use of psychedelics, empathogens, and entactogens for treating other issues such as depression or PTSD. For example, the early results on using MDMA for treatment-resistant PTSD seem extremely encouraging (and Doblin's work in general seems promising).

Aside from the obvious dangers relating to bad trips, psychosis, neurotoxicity (which seems only relevant for MDMA), et cetera[1], my main worry is that psychedelics sometimes seem to decrease people’s epistemic and instrumental rationality. I also observed that they sometimes seem to have shifted people’s interests towards more esoteric matters and led to “spiritual navel-gazing”—of course, this can be beneficial for people whose life goals are comparatively uninformed.

  1. Though my impression is that these risks can be reduced to tolerable levels by taking psychedelics only in appropriate settings and with the right safety mechanisms in place. ↩︎

Milan_Griffes @ 2020-04-30T16:27 (+9)

Thanks for this great comment!

I had forgotten about Pokorny et al. 2017 – that's definitely relevant work.


The kind of individuals that seem most dangerous (people with highly elevated Dark Tetrad traits who are also ambitious, productive and strategic) seem less likely to be interested in taking psychedelics—such folks don't seem interested in increasing their empathy, becoming less judgmental or having spiritual experiences.

A lot of Nazis were interested in the occult, and Mao wrote poetry.


I’m somewhat pessimistic that giving psychedelics to highly malevolent individuals would make them substantially more compassionate. That being said, I'm certainly not confident in that assessment.

More research needed here.

As far as I know, there hasn't been any work done on how psychedelics affect individuals with high Dark Tetrad scores.


... the Concord Prison Experiment. However, due to various methodological flaws, I'd be hesitant to draw strong conclusions from this experiment.

Yeah, the Concord Prison Experiment is pretty bad. Hendricks et al. 2014 is better work on this, though it's just a survey.


my main worry is that psychedelics sometimes seem to decrease people’s epistemic and instrumental rationality.

Do you know where this worry comes from?

Gabay et al. 2019 found that MDMA boosted people's cooperation with trustworthy players in an iterated prisoner's dilemma, but not with untrustworthy players. I take that as some evidence that MDMA doesn't acutely harm one's rationality. I don't think there's been similar work done for other psychedelics.


I also observed that they sometimes seem to have shifted people’s interests towards more esoteric matters and led to “spiritual navel-gazing”—of course, this can be beneficial for people whose life goals are comparatively uninformed.

I think spiritual bypassing is a real problem, and unwise psychedelic use can definitely facilitate it. Context & integration seems very important.

David_Althaus @ 2020-05-05T12:50 (+4)

A lot of Nazis were interested in the occult, and Mao wrote poetry.

Good point, my comment was worded too strongly. I’d still guess that malevolent individuals are, on average, less interested in things like Buddhism, meditation, or psychedelics.

Do you know where this worry [that psychedelics sometimes seem to decrease people’s epistemic and instrumental rationality] comes from?

Gabay et al. 2019 found that MDMA boosted people's cooperation with trustworthy players in an iterated prisoner's dilemma, but not with untrustworthy players. I take that as some evidence that MDMA doesn't acutely harm one's rationality.

Interesting paper! Though I didn’t have MDMA in mind; with “psychedelics” I meant substances like LSD, DMT, and psilocybin. I also had long-term effects in mind, not immediate effects. Sorry about the misunderstanding.

One reason for my worry is that people who take psychedelics seem more likely to believe in paranormal phenomena (Luke, 2008, p. 79-82). Of course, correlation is not causation. However, it seems plausible that at least some of this correlation is due to the fact that consuming psychedelics occasionally induces paranormal experiences (Luke, 2008, p. 82 ff.) which presumably makes one more likely to believe in the paranormal. This would also be in line with my personal experience.

Coming back to MDMA. I agree that the immediate, short-term effects of MDMA are usually extremely positive—potentially enormous increases in compassion, empathy, and self-reflection. However, MDMA’s long-term effects on those variables seem much weaker, though potentially still positive (see Carlyle et al. (2019, p. 15).

Overall, my sense is that MDMA and psychedelics might have a chance to substantially decrease malevolent traits if these substances are taken with the right intentions and in a good setting—ideally in a therapeutic setting with an experienced guide. The biggest problem I see is that most malevolent people likely won’t be interested in taking MDMA and psychedelics in this way.

Milan_Griffes @ 2020-05-21T16:22 (+3)
Overall, my sense is that MDMA and psychedelics might have a chance to substantially decrease malevolent traits if these substances are taken with the right intentions and in a good setting—ideally in a therapeutic setting with an experienced guide. The biggest problem I see is that most malevolent people likely won’t be interested in taking MDMA and psychedelics in this way.

Our estimates of the likelihood of malevolent people being interested probably hinge on our theory of where malevolence comes from.

e.g. if we think malevolence mostly arises as a maladaptive coping response to early trauma, you could imagine interventions that resolve the trauma and replace the maladaptive response with a more prosocial & equally fit response (and malevolent people being interested in those interventions).

But if we think malevolence is mostly a genetically-mediated trait, it's probably harder to change.

I haven't poked the literature on this yet.

Timothy_Liptrot @ 2020-06-01T20:59 (+8)

I am skeptical of this line of reasoning because I see little reason to believe that malevolence determined the policies in question. Game theory political scientists argue that different institutional structures make it rational or irrational for leaders to distribute public goods or targeted goods, practice repression, allow political parties. For a more in depth treatment, see the Dictator's Handbook by Bruce Bueno De Meqsuita and Alistair Smith. Their core argument is that because dictators must appease a very small group of powerful interest leaders (generals for Mussolini, members of the centcom for Stalin, tribal and military leaders for the Abdullah II) they can protect their power by rewarding only that small group at the expense of the masses.

Here is an illustrative example of political phenomena that is difficult to explain from the leaders personality. Torture is more common in multi-party autocracies than in one-party states. If the leaders narcissism strongly influences policies and narcissism and sadism are strongly correlated, then we would expect torture to be more common in states that ban dissent. Suppose that torture is not about satisfying the personal desire of the dictator and is instead about policing dissent. Now it makes sense that if some dissent (like resistance to a new "non-security" policy) is allowed, there must be some boundary into banned dissent. Then the occurrence of torture in multi-party states makes more sense and the rarity of torture in the most severe autocracies makes sense.

Opposing personality-of-dictator explanations to ideological explanations surprised me because it ignored the strongest explanations in institutional structures of states and in political cultures. Possibly you emphasized ideology because your samples are older. While the early modernist dictators were authentically ideological, most modern autocrats espouse a bland, centrist, syncretic corporatism. Dictators like Chavez and Castro are the exception today (although their ideology does influence behavior). Here is an article which argues most dictatorships are interest driven, not ideologically driven https://sci-hub.tw/10.1080/13510347.2017.1307823

MichaelA @ 2020-08-26T15:57 (+8)

I'd definitely agree that leaders' personalities are not the sole factor in determining a countries' policies, levels of repression, etc., even in autocracies. And I'd tentatively guess that leaders' personalities tend not to be a larger factor than other influences combined. And it's very plausible to me that institutional structures tend to be a larger factor. 

But it seems to me that leaders personalities are very likely at least sometimes a substantial factor. And, from memory, I don't think this post makes or requires stronger claims than that. I.e., I don't recall it making confident claims that other factors like institutional structures are usually smaller factors than personality. Are there passages you thought were overly strong?

I think this post was largely arguing that, on the margin, it could be quite valuable to think about reducing risks from malevolence - not that other factors like institutional structures don't also matter.

Torture is more common in multi-party autocracies than in one-party states.

This claim seems plausible to me, but I've never seen the claim before or seen specific evidence on it. Could you cite a source for that?

I also know of some data points that seem to push against that claim. For example, I believe there was a lot of torture during the one-party states of Hitler, Stalin, and Mao. (Here are some sources that seem to support this, though I only read the titles, as I was already pretty sure this was true: 1, 2, 3.) 

And a handful of historical accounts I've read make it seem quite plausible that the personality of the leaders played a role in this. In particular, those accounts suggest these leaders (or at least Hitler and Stalin; I know less about Mao) took pleasure in things like ordering tortures or gruesome deaths,  that they regularly personally ordered these things, and that they sometimes ordered these things at times when it seems it wasn't useful or logical to do so. Here's one paper that supports those claims (though I'm mildly skeptical of the paper in some ways): Why Tyrants Go Too Far: Malignant Narcissism and Absolute Power.

(I've just started doing research following on from this post, and collaborating somewhat with its lead author, but all of the above opinions are my own, as always.)

Timothy_Liptrot @ 2020-08-26T16:15 (+3)

Source: https://edisciplinas.usp.br/pluginfile.php/103226/mod_resource/content/1/James%20Vreeland%202008.PDF

Concerns with causation:

I worry about the underlying assumption that democracies don't encourage malevolent traits. So we observe less mass killings and rival killings in democracies than in dictatorships. One explanation is that democracies are selecting for anti-killing leaders. Another explanation is that malevolent leaders in democracies see little gain from killing while malevolent leaders in certain types of dictatorships see much gain from killing. For example, Abraham Lincoln and Richard Nixon seem to have a lot of malevolent traits, but they mostly refrained from political killings. So the fourth section suggests too much causation on the personalities of leaders.

But it seems to me that leaders personalities are very likely at least sometimes a substantial factor.

Fair - even if most of the difference in political killing is from institutional incentives, preventing malevolent actors from becoming dictators or presidents is big gains.

Secondly, this research agenda must recognize that authoritarian cultures may accept or encourage violence against non-comformists or "disloyal" people. It's a deeply sad fact, but important to understand. If torturing dissidents is an expected and approved behavior it is weaker evidence of a malevolent personality (fundamental attribution error).

Targeting your interventions:

You should pay more attention to how autocrats are actually selected. There are a few good models, my favorite is selectorate theory which (in dictatorships) emphasizes trust between top lieutenants. To continue with your preferred 40's references, Herman Goring, Himler, Keitel and Borman all hate and distrust one another. But they each trust Hitler because Hitler selected them to receive stolen wealth in exchange for support. So as long as the autocrat is alive the alliance is "stable".

In this model they care most about malevolence when the ruling coalition/launching org selects the next dictator. This famously happened after Stalin died, when Beria (head of secret police) got overthrown and replaced with Khrushchev who promised to stop political killings. So target regimes in or near transitions.

For example, the politburo of Cuba might be interested in this research. Saudi Arabia, now that MBS has solidified control, will be less fruitful.

MichaelA @ 2020-08-26T18:23 (+7)

https://edisciplinas.usp.br/pluginfile.php/103226/mod_resource/content/1/James%20Vreeland%202008.PDF

Thanks for sharing that source. I tried to skim through to find the part most relevant to your claim, and found this:

What is observed regarding patterns of torture under dictatorship? The Hathaway torture data include 967 country-year observations of from 1985 to 1996 covering 109 separate dictatorships. The mean rate of torture is 3.0 with a standard deviation of 1.1; the median is 3. My principal explanatory variable is Gandhi's "parties," a dummy variable coded 1 if more than one party exists legally and 0 otherwise. For the Hathaway data, nearly 52 percent of the observations have multiple political parties. Average rates of torture under dictatorships without political parties is 2.8 (standard deviation 1.1), but rates of torture under dictatorships with political parties is 3.1 (standard deviation 1.0). A t-test indicates that the difference in average rates of torture is statistically significant beyond the 0.01 confidence level (t = -4.44). Rates of torture are ironically higher under open dictatorships that allow multiple political parties

So I do think this source supports your claim. But I'd note that:

  • That data just covers 1985-1996. Hitler, Stalin, and Mao seem the most salient examples of potentially very malevolent individuals wielding great power and causing great harm, and each would be excluded from that data set.
    • I'm not saying that the study should've focused on an earlier period, or that there's some reason our predictions about the future would be better informed by an earlier 11 year period.
    • But I'd be more confident about extrapolations from the data set if it spanned a larger period of time.
    • And I think it would make sense for our predictions to draw on both that data and the observation of extreme harms under the one-party states of Hitler, Stalin, and Mao. (The latter is just a sample of 3, but I believe it accounted for a large portion of all deaths in mass murders, famines, etc.)
  • The difference in averages doesn't look hugely substantial - ~30% of 1 standard deviation - even if it was statistically significant.
    • This seems like a further reason to also pay substantial attention to other observations, like harms under Hitler, Stalin, and Mao.

(Again, this doesn't mean your claim was false, just that its evidence base and implications may be more limited than one might have thought.)

Timothy_Liptrot @ 2020-08-31T18:05 (+3)

I'm compelled by this. That the difference is only 30% of 1 standard deviation means that lots of variation could be explained by other factors. Personality of the dictator could still explain lots of variation, even a majority. There could also be a relationship between dictator personality and allowances for dissent. Thanks for explaining that!

Aside, you would be more compelling if you talked about autocrats other than Hitler, Stalin and Mao.

MichaelA @ 2020-09-01T07:28 (+3)

Aside, you would be more compelling if you talked about autocrats other than Hitler, Stalin and Mao.

Do you mean talking about other autocrats as well, or instead of, talking about Hitler, Stalin, and Mao?

If you mean "as well", I'd agree. I've already started looking at some others (as did David and Tobias, the authors of this post), and will hopefully do more of that. The reason for focusing mostly on those three so far is just that it takes time to learn about more, and those three were (I'd argue) huge factors in a large portion of all harms from political atrocities in the 20th century.

If you mean "instead of", could you explain further why you say that?

Timothy_Liptrot @ 2020-09-03T15:34 (+4)

Hmmm, that is a good question. Let me dig in more. Here are reasons to talk about others than Hitler Stalin and Mao

Coup Proofing is a common practice of dictators for political survival

Some behaviors of Hitler, Stalin and Mao have compelling institutional explanations that have become repeated behavior of long-ruling dictators. I'm thinking of coup proofing in particular. Coup proofing is a set of policies dictators enact to prevent a single small group from seizing power; rotatring or purging officers (Tukachevsky/Rommel) splitting the army into multiple factions/militias (NKVD/SS/Revolutionary Guard). We've since observed that lots of dictators (and coup-threatened democracies) practice coup-proofing. So I would be careful about attributing the *intra-elite* violence by particular regimes to the personalities of the leaders. Coup-proofing cannot explain violence against non-elites by those regimes.

Change in ideological motivation between early dictators and modern dictators

Secondly, Hitler, Stalin and Mao were much more ideological than most modern dictators. The Mussolini model of a "moderate" or "synarchic" authoritarianism spread more after WWII. By moderate I mean without a narrow, extreme vision of state-society interaction, not that dissent or economic freedom were allowed. Particularly today ideology structures the behavior of dictators much less. So one could argue that both Hitler and Stalin faithfully followed the visions laid out in their (terrible and warped) ideologies. If you buy that argument you would update downward on future dictators committing similar violence against non-elites.

For example, Mohammad Bin Salman has killed and tortured elite rivals, dissenters and starved many thousands of Yemenis to death. But he does not seem interested in any state project that would involve violence on the scale of Hitler/Stalin/Pol Pot.

All that said, I personally do not put much weight on my ideology argument. Firstly, the ideological explanation of Hitler and Stalin's behavior is not that strong. Secondly just because most dictators, like MBS and the Ethiopians, have "moderate" visions of state-society relations does not prevent future radical dictators from taking power (tail risk). Note that ISIS is a deathcult that took over half of two middle income countries, and that even in the 20's and 30's Mein Kampf was such ridiculous nonsense that Hitler should not have risen to power.

But on a first read this is a real hole in your argument. Almost all modern dictators do not look like Hitler, Stalin and Mao due to their weak ideoligical commitments. If you include a few more modern examples you can evade this argument. Off the top of my head I would suggest:

Khomeini and Ahmedinajad - they did not murder lots of people but their stubborn refusal to compromise with the world order has impoverished Iran without causing a regime change (they changed a bit recently but 30 years of bad decisions before).

Pol Pot - 1980's, only stopped by Vietnamese intervention

Modern Burma

MichaelA @ 2020-09-03T17:09 (+7)

Your comments about coup-proofing seem interesting and useful.

We've since observed that lots of dictators (and coup-threatened democracies) practice coup-proofing. So I would be careful about attributing the *intra-elite* violence by particular regimes to the personalities of the leaders.

I think the fact that more leaders engaged in violent coup-proofing (rather than it just being Hitler, Stalin, and Mao) should indeed provide weak evidence against the theory that the unusual (compared to the population as a whole) personalities of leaders plays a key role in whether violent coop-proofing occurs. This is because that theory would now need to claim that a larger number of leaders have personalities that are unusual in the relevant way, or that a lack of such unusual personalities was "made up for" by other factors in some cases.

But I think that this fact would only serve as weak evidence, because it doesn't seem very implausible to claim that a fairly large number of dictators, or leaders of coup-threatened democracies, have personalities that are unusual in the relevant way. These are people in unusual positions which are arguably easier to get into if one is ruthlessly self-interested, so it wouldn't seem surprising (prima facie) if their average levels of ruthlessness-relevant traits was notably above population averages.

Additionally, it seems worth distinguishing violent coup-proofing from coup-proofing as a whole. In terms of how well they might evidence malevolent traits, "rotating [...] officers" and "splitting the army into multiple factions/militias" seem quite different from the sorts of violent purges engaged in by e.g. Stalin. (It may well be that violent coup-proofing is very common as well; I'm just flagging that the distinction seems relevant for my purposes.)

You also seem to imply that (a) these coup-proofing behaviours may have been rational things for a self-interested leader to do in those situations, and (b) this is reason to be careful in assuming that this is about personality. I think (a) is a good point. And I think there's some merit to (b), in the sense that this pushes against thinking something like "These leaders are just crazy and evil." 

But overall, I don't think this is a question of personality vs incentive structures. I think neither determines a person's behaviour by itself. And I think whether or not one would do even horrific things if it's in one's self-interest is partly a matter of personality. E.g., I'd be willing to bet that a very large portion of people wouldn't engage in violent coup-proofing, even if they were in a situation where doing so would help them keep power. (Another framing of this is that a person's personality and values helps determine their incentives, in the sense that it influences what we like and dislike, including how much we dislike harming people.) 

Coup-proofing cannot explain violence against non-elites by those regimes.

I think this is key; I think the massive-scale violence against non-elites is probably a larger part of why Hitler, Stalin, and Mao seem like interesting case studies of potential harm from malevolence than is the potentially "rational" coup-proofing.

That said, it's good to independently consider the merits of a even non-central arguments/points, and to update incrementally, so your arguments against making naive inferences from coup-proofing behaviours are still useful.

Timothy_Liptrot @ 2020-09-03T18:30 (+6)

I have one criticism of the argument that coup-proofing prevalence is evidence for personality factors. Suppose that if people observe a game being played multiple times they are more likely to set aside their personal preferences and "play to win". So if I were the first dictator of Iraq I might say "no I'm not going to kill generals who come from different towns, that would be evil". And then get killed for it. And maybe the second dictator says the same thing. But by the time the third or fourth dictator rises to power he'll either be selected for willingness to use violence or he will decide his preference for living is stronger than his preference for not killing. While I agree that many people would not commit inter-elite violence as the first leader, I suspect a much larger number would as the 5th leader. So an argument for point B.

Saddam Hussein was the 5th Iraqi leader to take power by coup within 21 years.

But on the other hand, there are lots of leaders that just stepped down when they lost the support of their ruling coalitions. And those heroes do not become famous. This is strong evidence of the importance of personality.

MichaelA @ 2020-09-03T19:57 (+3)

I have one criticism of the argument that coup-proofing prevalence is evidence for personality factors.

To be clear, my argument was more like "coup-proofing prevalence doesn't seem like strong evidence against personality playing an important role". I.e., I don't think that it should reduce our belief that personality plays an important role.

It is true that I think I'd see these behaviours as evidence for personality playing an important role. But I'm not sure, and I'm not seeing it as key evidence.

While I agree that many people would not commit inter-elite violence as the first leader, I suspect a much larger number would as the 5th leader. So an argument for point B.

I'd agree that a much larger number would as the 5th leader than as the 1st leader, in the scenario you describe. And I think this is a valuable point.

But, in line with your final paragraph, I'd still bet that many people wouldn't; I think many people would simply step down, flee, or accept radical changes to the nature of their regime.

And perhaps more importantly, I think personality influences whether someone tries to become a leader in the first place, and whether they succeed in that. So I expect a lot of people to not want to "do horrible things", recognise that pursuing this leadership position may require them to "do horrible things" along the way or to stay in power, and thus just not pursue those positions.

(That said, I did say "I'd be willing to bet that a very large portion of people wouldn't engage in violent coup-proofing, even if they were in a situation where doing so would help them keep power." So there's a valid reason why you focused on how people would behave if they somehow landed in the leadership position, rather than how likely they are to enter those positions to begin with.)

MichaelA @ 2020-09-03T17:09 (+3)

Thanks for this comment.

Your points (and counterpoints!) about changes in ideological motivation are very interesting. And I think it'd probably be good for me to spend some time engaging with evidence/arguments about how much ideology influenced Hitler, Stalin, and Mao's most "extreme" behaviours and whether/how much the influence of ideology has waned. 

And it does seem wise to think about that, and about more modern examples, if one is planning to communicate with the public, policymakers, or academics about this topic in a way that leans substantially on historical examples of dictators. (I'm not sure if anyone will actually do such communications, or emphasise those cases when doing so. It may, for example, make more sense to just focus on the psychological studies, or on examples from business.)

MichaelA @ 2020-08-26T18:23 (+3)

Another explanation is that malevolent leaders in democracies see little gain from killing while malevolent leaders in certain types of dictatorships see much gain from killing. For example, Abraham Lincoln and Richard Nixon seem to have a lot of malevolent traits, but they mostly refrained from political killings. So the fourth section suggests too much causation on the personalities of leaders.

I did recently think it might be interesting to look into Nixon as a case study in how, and how well, democratic institutions can mitigate the harm caused by leaders with high levels of dark tetrad traits. (I just think it might be such a case study, because I haven't yet really looked into evidence on Nixon's personality - this is just a guess so far.) Thanks for highlighting Lincoln too - I wouldn't have guessed he had high levels of dark traits, but I'll look into it.

I'd definitely guess that the reasons there is less harm from malevolent leaders in democracies are both that democracies select for malevolence less, and that democracies don't allow/incentivise malevolent behaviours as much. 

In my head, I currently break intervention options in this cause area into:

  • Reducing how malevolent people are (via, e.g., very cautious and well thought-out and not-rushed genetic engineering)
  • Reducing the chances of malevolent people getting into positions where those traits create major risks (via, e.g., electoral reform, reducing instability and conflict)
  • Reducing the risks created when malevolent people get into those positions (via, e.g., checks and balances, maybe reducing centralisation of power)

I had felt like this post implied all three of those categories, not just the first two. But now that I re-skim the Political interventions section, I see that that might not have been made very explicit. So that critique of yours may be valid. (And I definitely agree with your point, separate from how it relates to potential oversights of this post.)

this research agenda must recognize that authoritarian cultures may accept or encourage violence against non-comformists or "disloyal" people. It's a deeply sad fact, but important to understand. If torturing dissidents is an expected and approved behavior it is weaker evidence of a malevolent personality (fundamental attribution error).

I definitely agree that:

  • it's important to consider multiple explanations of the various horrific or troubling behaviours
  • it can be easy to psychoanalyse/diagnose from a distance in a foolish way
  • the fundamental attribution error is worth keeping in mind here

I'm hoping to look a bit into how much we can trust speculative psychological profiles in general, maybe "best practices" for that shaky endeavour, and maybe how this relates to people like Hitler in particular. (Before getting into EA, I might've dismissed such psychological profiles. But I've moved towards thinking that, for many questions, it's hard to do better than weak evidence, and it can be best to just gather diverse kinds of weak evidence, update incrementally, and remain aware of how uncertain we are.)

(Also, by the way, I read The Dictator's Handbook last year and found it interesting. I agree that the models/ideas in that book seem relevant here.)

MichaelA @ 2020-05-06T06:42 (+8)

Great post, thanks for writing it!

Baron-Cohen (2012) argues that the defining feature of human evil is “zero degrees of empathy.” However, some psychopaths can read other people extremely well and would thus score highly on certain items of the empathy questionnaires Baron-Cohen describes in his book. Furthermore, as Baron-Cohen acknowledges, people on the autism spectrum tend to have less empathy—at least certain forms of it—but they are not more malevolent than the population average. Therefore, reducing malevolence to “zero degrees of empathy” could be problematic or at least crucially depends on how we define and operationalize empathy.

Wikipedia notes that "Empathy is generally divided into two major components":

And that article further notes:

Although science has not yet agreed upon a precise definition of these constructs, there is consensus about this distinction.[36][37] Affective and cognitive empathy are also independent from one another; someone who strongly empathizes emotionally is not necessarily good in understanding another's perspective.

And I remember being told during my psych undergrad that "psychopaths" have low levels of affective empathy but roughly average levels of cognitive empathy, while people on the autism spectrum have low levels of cognitive empathy but roughly average levels of affective empathy. (I haven't fact-checked this, though, or at least not for years.)

In the context of this post's arguments, I'd guess it's low affective empathy that is especially concerning, and perhaps especially when paired with high cognitive empathy.

(I assume you already know this stuff, but thought it might be useful for other readers.)

David_Althaus @ 2020-05-06T13:32 (+6)

Thank you! I agree that the distinction between affective and cognitive empathy is relevant, and that low affective empathy (especially combined with high cognitive empathy) seems particularly concerning. I should have mentioned this, at least in the footnote you quote.

And I remember being told during my psych undergrad that "psychopaths" have low levels of affective empathy but roughly average levels of cognitive empathy, while people on the autism spectrum have low levels of cognitive empathy but roughly average levels of affective empathy. (I haven't fact-checked this, though, or at least not for years.)

That sounds right. According to my cursory reading of the literature, psychopathy and all other Dark Tetrad traits are characterized by low affective empathy. While all Dark Tetrad traits except for narcissism also seem to correlate with low cognitive empathy, the correlation with diminished affective empathy seems substantially more pronounced (Pajevicc et al., 2017, Table 1; Wai & Tiliopoulos, 2012, Table 1).[1] As you write, people on the autism spectrum basically show the opposite pattern (normal affective empathy, lower cognitive empathy (Rogers et al., 2006; Rueda et al., 2015).

We focused on the Dark Tetrad traits because they overall seem to better capture the personality characteristics we find most worrisome. Low affective empathy seems a bit too broad of a category as there are several other psychiatric disorders which don’t seem to pose any substantial dangers to others but which apparently involve lower affective empathy: schizophrenia (Bonfils et al., 2016), schizotypal personality disorder (Henry et al., 2007, Table 2), and ADHD (Groen et al., 2018, Table 2).[2]

Of course, the simplicity of a unidimensional construct has its advantages. My tentative conclusion is that the D-factor (Moshagen et al., 2018) captures the most dangerous personalities a bit better than low affective empathy—though this probably depends on the precise operationalizations of these constructs. In any case, more research on (diminished) affective empathy seems definitely valuable as well.

  1. Though Jonason and Krause (2013) found that narcissism actually correlates with lower cognitive empathy and showed no correlation with affective empathy. ↩︎

  2. This list is not necessarily exhaustive. ↩︎

MichaelA @ 2020-05-06T23:52 (+2)

Interesting!

Reading your comment, I realised one statement of mine may have been unclear. I wrote:

In the context of this post's arguments, I'd guess it's low affective empathy that is especially concerning, and perhaps especially when paired with high cognitive empathy

By that I just meant that, in terms of those two components of empathy, that might be the most concerning pattern. I didn't meant to imply that that's the key thing we should be concerned about in relation to personality traits more broadly - the idea that Dark Tetrad traits are a better focus seems reasonable to me (though I don't bring much independent knowledge to the table on that point).

alexherwix @ 2020-06-10T18:03 (+7)

Thanks for writing this post, very interesting! I haven't read all of the comments but wanted to share one point that came to me over and over again while reading the post. Apologies if it has already been mentioned in another comment.

It seems like you assume a strong (and relatively simple) causal relationship from genetics to malevolent traits to bad behavior. I think this view might make the problem seem more tractable than it actually might be. Humans are complex systems that are nested in other complex systems and everything is driven by complex and interacting feedback loops. Thus, to me it seems very difficult to untangle causality here. To me it would be much more intuitive to think about malevolence as a dynamic phenomenon that is emergent based on a history of interactions rather than a static personality trait. If you accept this characterization as plausible the task of screening for malevolence in a valid and reliable way seems much more difficult than just designing a better personality test. I think the main difference between those two perspectives is that in the simple case you have a lot of corner cases to keep in mind (e.g., what is if people have malevolent traits but actually want to be good people?) whereas the complex case is more holistic but also much more, well, complex and likely less tractable.

Nevertheless, I agree with the general premise of the post that mental health is an important aspect in the context of X/S-risk related activities. I would go even further than this post and argue that mental health in the context of X/S-risk related activities in general is a very pressing cause area that would score quite well in terms of ITN-Analysis. Thus, I would really love to see an organization or network being set up dedicated to the serious exploration of this area because existing efforts in the mental health space seem only to be focus on happiness in the context of global development. If some interested in this topic reads this, don't hesitate to reach out, I would love to support such efforts.

Neil_Dullaghan @ 2020-08-26T14:39 (+6)

Thanks for this!
I see you referenced Matthews et al. (2018), which I haven't read, and wondered if you had also seen the Authoritarian Ruling Elites Database, compiled by  Matthews (2019): “a collection of biographical and professional information on the individuals who constitute the top elite of authoritarian regimes.” Each of the project’s 18 datasets focuses on a particular regime, such as the military dictatorship that ruled Chile from 1973 to 1990. The biographical data-points include gender, occupation, dates of birth and death, tenure among the elite, and more. (This came to me via the Data Is Plural mailing list fwiw). Apologies if you mentioned it and I missed it.
 

peter_janicki @ 2020-05-03T11:34 (+6)

applause. thx for this highly interesting (and important) article.

while reading i thought about a lot of commenting, but you already considered most of these things…

still some minor comments:

as to narcissism (maybe (?) the least important "dark tetrad trait"): as for narcistic personality disorder, there is a reason why (some) of these people are trying to gain power. a huge lack of self-esteem etc. and some narcistic people are trying to fill this lack with a successful career etc., still it does not gets filled this way, so (some of them) are trying to become even more successful... a vicious circle. but maybe even more important for narcissism usually has it´s origin in childhood/ growing up. i don´t believe, that genetic (epigenetic?) factors attribute a lot to this trait. (I only found that one here, and I don’t know, whether it is good or representative: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3973692/ - still, it seems, that those twins grew up with the same parents …).

as for scanning for dark tetrad-traits: smart malevolent people might deceive tests/scanning procedures, once they are smart enough, once they know what they want. they usually don´t do so, when still in kindergarten, in elementary school. And it is sure (unprecise) to scan for example for those kids, who torture animals (or peers) etc. i am not saying: “let´s do this”, only saying: “looking at kids might makes sense”.

as for proposing to cut f. e. only the 1% with the highest polygenetic score for dark tetrad-traits: it be interesting, if this would include persons like hitler, stalin, mao, … they sure had some luck with timing, being at the right place, living in instable times, … maybe they would only be in the top 10th percentile? still, it sure be a bad idea, to make big cuts without more reasoning and knowledge.

and: actually, even though (or because) i just read some things about dictators and sadism... i am not convinced. most journalists argue, that being responsible for the deaths of millions makes one a sadist. or because one makes up some torture-routines. but from my understanding, sadism is to gain direct benefit through direct torturing, humuliating (and seeing/getting feedback on their reaction) of humans (or animals). if someone lets millions of people getting killed or tortured - i guess this is for other reasons (economic profit, consolidating power - even if through terror or because someone simply does not care). having a lack of empathy, it might easily sound like a reasonable idea to send millions to deaths, if there are other things to gain. actually my understanding of sadism is not important. but if those dictators haven´t been sadists, then it might be a good idea, to exclude sadism out of this topic.

still: up until now in this comment, there are 3 lines of praise for this article, and some other 20 lines... It should be the other way round. thx again.

Tobias_Baumann @ 2020-05-07T07:49 (+3)

Thanks for commenting!

I agree that early detection in children is an interesting idea. If certain childhood behaviours can be shown to reliably predict malevolence, then this could be part of a manipulation-proof test. However, as you say, there are many pitfalls to be avoided.

I am not well versed in the literature but my impression is that things like torturing animals, bullying, general violence, or callous-unemotional personality traits (as assessed by others) are somewhat predictive of malevolence. But the problem is that you'll probably also get many false positives from those indicators.

Regarding environmental or developmental interventions, we write this in Appendix B:

Malevolent personality traits are plausibly exacerbated by adverse (childhood) environments—e.g. ones rife with abuse, bullying, violence or poverty (cf. Walsh & Wu, 2008). Thus, research to identify interventions to improve such environmental factors could be valuable. (However, the relevant areas appear to be very crowded. Also, the shared environment appears to have a rather small effect on personality, including personality disorders (Knopik et al., 2018, ch. 16; Johnson et al., 2008; Plomin, 2019; Torgersen, 2009).)

Perhaps improving parenting standards and childhood environments could actually be a fairly promising EA cause. For instance, early advocacy against hitting children may have been a pretty effective lever to make society more civilised and less violent in general.

EdoArad @ 2020-05-01T06:12 (+6)

Thanks for a very thorough and interesting report! 

It seem plausible that institutional mechanisms that prevent malevolent use of power may work well today in democracies. I think that the comparison is very important for understanding the value of the suggested interventions. You have briefly touched this - 

Overall, it seems plausible that many promising political interventions to prevent malevolent humans from rising to power have already been identified and implemented—such as, e.g., checks and balances, the separation of powers, and democracy itself. After all, much of political science and political philosophy is about preventing the concentration of power in the wrong hands.[26] We nevertheless encourage interested readers to further explore these topics.

If these mechanisms are actually working quite well today, this somewhat lowers the importance of the suggested interventions. The analysis given above is mostly for non-modern institutions, but perhaps the court system, democracy and transparency has evolved so that malevolent actors can not really do much harm (or that it will be harder for them to get in power). 

Also, the major alternative to reducing the influence of malevolent actors may be in the institutional decision making itself, or some structural interventions. AI Governance as a field seems to mostly go in that route, for example. 

That said, I think that efforts going into your suggested interventions are largely orthogonal to these alternatives (and might actually be supportive of one another). Also, I intuitively find your arguments quite compelling.

David_Althaus @ 2020-05-01T11:49 (+10)

It seem plausible that institutional mechanisms that prevent malevolent use of power may work well today in democracies.

I agree that they probably work well but there still seems to be room for improvement. For example, Trump doesn't seem like a beacon of kindness and humility, to put it mildly. Nevertheless, he got elected President. On top of that, he wasn't even required to release his tax returns—one of the more basic ways to detect malevolence.

Of course, I agree that stable and well-functioning democracies with good cultural norms would benefit substantially less from many of our suggested interventions.

Also, the major alternative to reducing the influence of malevolent actors may be in the institutional decision making itself, or some structural interventions. AI Governance as a field seems to mostly go in that route, for example.

Just to be clear, I'm very much in favor of such "structural interventions". In fact, they overall seem more promising to me. However, it might not be everyone's comparative advantage to contribute to them which is why I thought it valuable to explore potentially more neglected alternatives where lower-hanging fruits are still to be picked.

That said, I think that efforts going into your suggested interventions are largely orthogonal to these alternatives (and might actually be supportive of one another).

Yes, my sense is that they should be mutually supportive—I don't see why they shouldn't. I'm glad you share this impression (at least to some extent)!

Milan_Griffes @ 2020-04-29T19:29 (+6)

Somewhat related – US college students have been growing more narcissistic (or maybe just gaining higher self-esteem) since the 1980s: https://flightfromperfection.com/college-students-more-narcissistic.html

maxh94 @ 2020-07-27T18:32 (+5)

Here is an interesting paper related to using machine learning to identify language patterns in psychopaths if people are interested. https://egrove.olemiss.edu/cgi/viewcontent.cgi?article=1445&context=etd

aaronb50 @ 2020-05-13T00:55 (+5)

Another type of intervention that could plausibly reduce the influence of malevolent actors is to decrease intergenerational transfer of wealth and power. If competent malevolence both (i) increases one's capacity to gain wealth and/or power and (ii) is heritable, then we should expect malevolent families amass increasing wealth and power. This could be one reason that the global shift away from hereditary monarchies is associated with global peace (I sense that both of these things are correct, but am not positive).

For example, North Korea's Kim family is almost certainly malevolent in the way that this post describes, and the country's political structure enables this family to continually keep power.

On a broader scale, larger estate taxes and other economic policies to decrease wealth transfer might have a similar effect.

edcon @ 2020-04-30T20:01 (+4)

Thanks for the post! Really well reasoned on the broad impact of malovelence.

1. It seems that any research on manipulation proof measures for detection for malevolence, would help the development of tools that would be useful for a totalitarian state.

2. I'm sceptical of further research on malevolence being helpful in stopping these people being in positions of power. At first glance I don't think a really well developed literature on malevolence, would of changed leaders coming to power in 20th century.

3. In terms of Public engagement, I am also sceptical (eg. jon ronson the psychopath test) as I suspect that making people want more altruistic people in charge, is hard to move needle on. (Interesting piece on malevelonce in leaders https://theconversation.com/narcissists-and-psychopaths-how-some-societies-ensure-these-dangerous-people-never-wield-power-118854)

4. In terms of development of genetically engineered individuals, I think it really matters more who is in charge. I doubt bio-ethicists saying dark triad traits are bad will have much of an effect. In terms of further GWAS studies, I suspect by the time this becomes feasible more GWAS on desirable personality traits will have been undertaken. And it seems that work done ahead of time will be less useful, due to earlier studies being superseded by studies with more advanced technology .

David_Althaus @ 2020-05-06T14:07 (+8)

Thanks, these are valid concerns.

  1. It seems that any research on manipulation proof measures for detection for malevolence, would help the development of tools that would be useful for a totalitarian state.

My guess is that not all research on manipulation-proof measures of malevolence would pose such dangers but it’s certainly a risk to be aware of, I agree.

  1. I'm sceptical of further research on malevolence being helpful in stopping these people being in positions of power. At first glance I don't think a really well developed literature on malevolence, would of changed leaders coming to power in 20th century.

In itself, a better scientific understanding of malevolence would not have helped, agreed. However, more reliable and objective ways to detect malevolence might have helped iff there also had existed relevant norms to use such measures and place at least some weight on them.

I think it really matters more who is in charge. I doubt bio-ethicists saying dark triad traits are bad will have much of an effect.

Bioethicists sometimes influence policy though I generally agree with your sentiment. This is also why we have emphasized the value of acquiring career capital in fields like bioinformatics.

In terms of further GWAS studies, I suspect by the time this becomes feasible more GWAS on desirable personality traits will have been undertaken.

I agree that this is plausible—though also far from certain. I’d also like to note that (very rudimentary) forms of embryo selection are already feasible, so the issue might be a bit time-sensitive (especially if you take into account that it might take decades to acquire the necessary expertise and career capital to influence the relevant decision makers).

L.F.D Olivares @ 2024-06-27T22:50 (+3)

Amazing article, It is a great insight into the malevolent person.

Yet I don't see the problem in the malevolent, but the system that allows their rise by making those malevolent traits praisable in every ideology, even in commerce.

There is no definition of ideology in the text, yet one is needed, because every ideology is dangerous. For an ideology seeks to impose a single maxim over other human endeavours, nazism was about racial supremacy, marxism positions the class struggle as an idea over all, religious fundamentalism puts god above everything, capitalism puts commerce above all, logic positivist wanted to reduce everything to epistemology as such classical physics uber alles.

Ideologies, then have a tendency to be holistic and reductionist, they attempt to put everything inside their maxim and reduce all other bodies of knowledge to such maxims, they are dogmatic, thus making it favorable for those with malevolent personalities to thrive as they are the most dogmatic of all.

The reason why some autocratic rulers were no malevolent such as Marcus Aurelius, Atatürk, and others is because they followed no ideology. They were not idealist but realists, in the case of Aurelius we have his meditations and as such we can see that he holded no maxim, he made no reduction to a singular knowledge, Stoicism was a physicalist philosophy, a realist belief system.

How can we fight ideologies then? Well Ideology are always opposed by other ideologies  and we can keep that fight eternally, but also by science, as the relentless pursuit of operational knowledge always ends up destroying the maxims' ideologies hold. Examples of these are; Galileo when incarcerated by proving the copernican system, a maxim of a race above all of nazims is absurd when biology proves there are only phenotypic differences between humans, the materialist beginning of civilization proposed by marxist and social darwinist both erroneos by the excavations of Gobekli Tepe and the other Tepe cites, etc.

In essence to stop the malevolent is by generating a system that does not allow for their rise, a system without ideologies or at least where they are contained into their own battle ground, internet forums. Such system then must disregard arguments by maxims, in essence make the validation by science alone.

If the malevolent can rise still in the system of science it is to be seen, but for that to happen they would need to transform science into an ideology, but that would be no science.

David_Althaus @ 2024-06-29T10:19 (+5)

Thanks.

I guess I agree with the gist of your comment. I'm very worried about extremist / fanatical ideologies but more on this below.

because every ideology is dangerous


I guess it depends on how you define "ideology". Let's  say "a system of ideas and ideals". Then it seems evident that some ideologies are less dangerous than others and some seem actually beneficial (e.g., secular humanism, the Enlightenment, or EA). (Arguably, the scientific method itself is an ideology.)

I'd argue that ideologies are dangerous if they are fanatical and extreme. The main characteristics of such fanatical ideologies include dogmatism (extreme irrationality and epistemic & moral certainty), having a dualistic/Manichean worldview that views in-group members as good and everyone who disagrees as irredeemably evil, advocating for the use of violence and unwillingness to compromise, blindly following authoritarian leaders or scriptures (which is necessary since debate, evidence and reason are not allowed), and promising utopia or heaven. Of course, all of this is a continuum. (There is much more that could be said here; I'm working on a post on the subject).

The reason why some autocratic rulers were no malevolent such as Marcus Aurelius, Atatürk, and others is because they followed no ideology. [...] Stoicism was a physicalist philosophy, a realist belief system.

Sounds like an ideology to me but ok. :)

 

Dave Cortright @ 2023-05-02T14:11 (+3)

Big Think just posted a video about improving leader selection by screening for psychopathy including narcissism… 

Dvir Caspi @ 2022-03-22T21:25 (+3)

Regarding neurological measurements - there is actually really significant fMRI work which demonstrates lack of empathy and fearlessness. I would recommend reading some of the work by Prof. Abigail Marsh  

Nathan_Barnard @ 2021-12-03T19:56 (+3)

It seems like a strange claim that both the atrocities committed by Hitler, Stalin and Mao were substantially more likely because they had dark triad traits and that when doing genetic selection we're interested in removing the upper tail, in the article it was the top 1%. To take this somewhat naively, if we think that the Holocaust, and Mao and Stalin's terror-famines wouldn't have happened unless all three leaders exhibited dark tetrad traits in the top 1%, this implies we're living in a world world that comes about with probability 1/10^6, i.e 1 in a million, assuming the atrocities were independent events. This implies a need to come up with a better model. 

Edit 2: this is also wrong. Assuming independence the number of atrocious should be binomially distributed with p=1/100 and n=#of leaders in authoritarian regimes with sufficiently high state capacity or something. Should probably be a markov-chain model. 

If we adjust the parameters to top 10%  and say that the atrocities were 10% more likely to happen if this condition is met, this implies we're living in a world that's  come about with probability (p/P(Dark triad|Atrocity)^3, where p is the probability of that the atrocity would have occurred without Hitler, Stalin and Mao having dark triad traits. The interpretation of P(Dark triad|Atrocity) is what's the probability that a leader has a dark triad traits given they've committed an atrocity.  If you have p as 0.25 and P(Dark|Atrocity) as 0.75 this means we're living in a 1/9 world, which is much more reasonable. But, this makes this intervention look much less good. 

Edit: the maths in this section is wrong because I did a 10% probability increase of p as 1.1*p rather than p having an elasticity of 0.1  with respect to the resources put into the intervention or something. I will edit this later. 

Excluding 10% of population from politcal power seems like a big ask. If the intervention reduced the probability that someone with dark triad traits coming to power (in a system where they could commit an atrocity) by 10%, which seems ambitious to me, this reduces the probability of an atrocity by 1% (if the above model is correct). Given this requires excluding 10% of the population from politcal power, which I'd say is generously 10%, this means that EV of the intervention is reducing the probability of an atrocity by 0.1%. Although this would increase if the intervention could be used multiple times, which seems likely. 

Thomas Kwa @ 2021-12-03T22:24 (+3)

I think an implicit assumption of the article is that people with dark triad traits are more likely to gain power. If unstable politics can create a 99:1 selection effect toward the Hitlers and Maos in the top %ile of dark triad, then they come to power in half of the nations with unstable politics. I can imagine this being true if people with dark triad traits are way more power-seeking and this is what matters in unstable political climates.

Nathan_Barnard @ 2021-12-04T00:36 (+1)

So the model is more like, during the Russian revolution for instance it's a 50/50 chance that whichever leader came out of that is very strongly selected to have dark traid traits, but this is not the case for the contemporary CCP.  

Yeah seems plausible. 99:1 seems very very strong. If it were 9:1  means we're in a 1/1000 world, 1:2 means an approx 1/10^5. Yeah, I don't have a good enough knowledge of rulers before they gained close to absolute power to be able to evaluate that claim. Off the top of my head, Lenin, Prince Lvov (the latter led the provisional govt' after Feb revolution) were not dark triady. 

The definition of unstable also looks important here. If we count Stalin and Hitler, both of whom came to power during peacetime, then it seems like also should count Soviet leaders who succeeded Stalin, CCP leaders who succeeded Mao,  Bashar al-Assad, Pinochet, Mussolini. Sanity check from that group makes it seem more much like a 1:5 than 1:99. Deng definitely not Dark Triad, nor Bashar, don't know enough about the others but they don't seem like it? 

If we're only counting Mao, then the selection effect looks a lot stronger off the top of my head, but should also probably be adjusted because the mean of sadism seems likely much higher after a period of sustained fighting given the effect of prison guards for instance becoming more sadistic over time, and gennerally violence being normalised. 

Don't know enough about psychopathy or machivallianism. 

It's also not completely clear to me that Stalin and Mao were in the top 10% for sadism at least. Both came from very poor peasant societies. I know at least Russian peasant life in 1910 was unbelievably violent and they reguarly did things which we sort of can't imagine. My general knowledge of European peasant societies - e.g crowds at public executions - makes me think that it's likely that the average Chinese peasant in 1910 would have scored very highly on sadism. If you look at the response of the Chinese police/army to the 1927  Communist insurgency it was unbelievably cruel. 

Makes screening for malicious actors seem worse and genetic selection seem better.  

Apologies that this is so scattered. 

ben.smith @ 2021-01-20T20:18 (+3)

I think the main solution is to develop strong and resilient institutions. Areas for improvement could be:

MikeJohnson @ 2020-05-05T09:40 (+3)

A core 'hole' here is metrics for malevolence (and related traits) visible to present-day or near-future neuroimaging.

Briefly -- Qualia Research Institute's work around connectome-specific harmonic waves (CSHW) suggests a couple angles:

(1) proxying malevolence via the degree to which the consonance/harmony in your brain is correlated with the dissonance in nearby brains;
(2) proxying empathy (lack of psychopathy) by the degree to which your CSHWs show integration/coupling with the CSHWs around you.

Both of these analyses could be done today, given sufficient resource investment. We have all the algorithms and in-house expertise.

Background about the paradigm: https://opentheory.net/2018/08/a-future-for-neuroscience/

Vasco Grilo @ 2024-02-06T13:46 (+2)

Hi David and Tobias,

I just came across Who would destroy the world? Omnicidal agents and related phenomena, which relates to your analysis. For reference, here is the abstract:

The capacity for small groups and even single individuals to wreak unprecedented havoc on civilization is growing as a result of dual-use emerging technologies. This means that scholars should be increasingly concerned about individuals who express omnicidal, mass genocidal, anti-civilizational, or apocalyptic beliefs/desires. The present article offers a comprehensive and systematic survey of actual individuals who have harbored a death wish for humanity or destruction wish for civilization. This paper thus provides a strong foundation for future research on “agential risks” and related issues. It could also serve as a helpful resource for counterterrorism experts and global risk scholars who wish to better understand our evolving threat environment.

However, the analysis is quite qualitative. Are you aware of research trying to quantify the number of omnicidal actors?

David_Althaus @ 2024-02-09T10:21 (+6)

I'm not aware of quantitative estimates of omnicidal actors. Personally, I'm less interested in omnicidal actors and more interested in actors that would decrease the quality of the long-term future if they had substantial levels of influence. This is partly because the latter type of category is plausibly much larger (e.g., Hitler, Mao, and Stalin wouldn't have wanted to destroy the world but would have been bad news regardless). 

FWIW, I've done a few pilots on how common various "s-risk conducive" values and attitudes are (e.g., extreme retributivism, sadism, wanting to create hell) and may publish these results at some point. 

Ben_West @ 2020-05-05T19:59 (+2)

A minor copyediting suggestion (adding the words in bold):

Factor 1—characterized by cruelty, grandiosity, manipulativeness, and a lack of guilt—arguably represents the core personality traits of psychopathy. However, scoring highly on factor 2—characterized by impulsivity, reactive anger, and lack of realistic goals—is less problematic from our perspective. In fact, humans scoring high on factor 1 but low on factor 2 are probably more dangerous than humans scoring high on both factors (more on this below).

It's not a big deal, but it took me a minute to understand why you were saying it's both less problematic and more dangerous.

David_Althaus @ 2020-05-06T10:37 (+3)

Thanks! Added your suggestion.

brb243 @ 2020-04-29T12:49 (+2)

Thank you very much for the breathtaking analysis. I think that malevolent humans will best be prevented from gaining influence if benevolent humans outcompete them at influential posts.

David_Althaus @ 2020-04-30T11:12 (+4)

Thank you.

The section “What about traits other than malevolence?” in Appendix B briefly discusses this.

brb243 @ 2020-05-01T20:16 (+1)

Thank you. I see. Then, I wish you the very best of luck in making an effective impact with your actions. I hope that you will succeed in preventing many decision-makers from promoting malevolent leaders.

Phil Tanny @ 2022-08-27T13:16 (+1)

Presumably, the same argument applies to banning scientific progress.

In case it's needed, I haven't made that argument, but have instead made this argument.

https://www.facebook.com/phil.tanny/posts/pfbid028hLRWeuJsDymwb3sgn4m8i1HLiA6YGbqAPNLgirFWEx4aBjXUp8t5E9pqBEYEjpsl

Phil Tanny @ 2022-08-27T11:21 (+1)

It's easy to say you're not calling for violence, but I don't see any way you could adopt this without violence, even if you had the technology to allow for it.

 

Ok, I accept your report that YOU don't see this.   That's completely normal.   There's a lot of it I don't see either.    But once we TRY to see, some possibilities can emerge.

I agree that within the current group consensus, a "world without men" proposal will be widely, if not universally, rejected.   I've been writing about this for years now, and that is what the evidence does show.  But the current group consensus is not locked in concrete, and I propose that it will be edited by large real world events which are probably coming.   

The problem is that the potential collapse of civilization is currently an abstraction, and most people are unable to engage abstractions emotionally.    Real world events are likely to transform such an abstraction in to a form that large numbers of people can engage emotionally.  

To pick an example randomly out of the hat, imagine even a limited nuclear exchange between Pakistan  and India.    More people will die in a few minutes than died in all of WWII.   Global media will ABSOLUTELY OBSESS over this event for months or longer.   Graphic scenes of mass destruction and vast suffering will be pumped in to every home across the civilized world, day after day after day after day.   And then...

Large numbers of people across  the world will finally grasp that this could happen to them too.  They'll grasp it where it matters, emotionally, not just intellectually.  

What happens next is anybody's guess, but it seems fair to claim the group consensus will then be up for grabs.  

Human beings don't learn things this large through reason alone.  We learn primarily through pain.  And lots of pain is most likely coming.

Phil Tanny @ 2022-08-27T11:01 (+1)

Many longtermist effective altruists think that shaping transformative artificial intelligence, and in particular solving the alignment problem, is a particularly good lever to improve the long-term future.

 

What will happen is what always happens.  The good people will use emerging technologies for well intended  purposes, and the bad  people will use emerging technologies in service to their agendas.

What seems to be missing from commentary on these subjects is  the insight that as the scale of such emerging powers grows, the room for error steadily shrinks, and the bad (or stupid) people are increasingly in a position to bring down the entire system, thus erasing whatever benefits the well intended good people have managed to create.

Shouldn't we have learned this in 1945??   

  1. Nuclear power is a powerful energy generation technology which, when perfected, can go a long way in addressing climate change challenges.
  2. But that doesn't really matter, because sooner or later a very long standing pattern of all out fight  to the death conflict which has been routine for 10,000 years will reappear (as it's trying to now in Ukraine) and nuclear weapons will erase the benefits of nuclear power.

Right now, as we speak, we may be on the very edge of the largest nuclear power plant in Europe poisoning all of Europe.    And that's because Putin can invade his neighbor with impunity.  And that's because he has nuclear weapons which protect him from any truly effective response to his invasion.

There is no science magic which will transform every psychopathic power tripper on the planet (almost all of whom are men) in to nice guys who aren't a threat.   

Hitler understood the laws of nature, and was very honest about it.   He said, "We are the predator, and you are the prey."    Hitler didn't invent that, he just understood that this has been the law of nature for a billion years.  The big fish eat the little fish.

The science clergy is working as hard as they can, to make the big fish bigger.  From the perspective of we the little fish, not such a great plan!

brb243 @ 2022-01-19T21:53 (+1)

Prima facie, this looks like a thoroughly researched innovative piece recommending an important focus area. The notions of preventing malevolent actors from causing harm if they rise to power by developing robust institutions, advancing benevolent frameworks to enable actors join alternatives as they are gaining decisionmaking leverage, and creating narratives which could inspire malevolent actors to contribute positively while keeping true to their beliefs are not discussed. Thus, using this framework for addressing malevolence can constitute an existential risk.

MichaelStJules @ 2022-01-19T22:23 (+4)

Could you expand on this? Are you suggesting that it could take attention away from solutions ("inspiring malevolent actors to contribute positively while keeping true to their beliefs") that would reduce x-risk more?

By "keeping true to their beliefs", do you mean their malevolent views? E.g. we want a non harmful or less harmful outlet for their malevolence? Or, just working with their beliefs without triggering their malevolence on a large scale? If the latter, what kinds of beliefs do you have in mind?

brb243 @ 2022-01-19T23:33 (+1)

Yes, if the community focuses on preventing people with the specified personality traits from gaining influence, it may become unpopular among prominent circles (not because leaders would necessarily possess these traits but because they may seek to present in agreement with the thinking, in order to keep their authority). Non-decisionmakers could be hesitant to publicly support frameworks which target actors which can be understood as threats, due to the fear of negative consequences for them. Thus, I am suggesting that while it is important to advance global good, this should be presented in a less conflicting manner.

By "keeping true to their beliefs" I mean fundamental righteousness/honorable behavior convictions. Yes, we would want an opportunity where they can be truly welcome and acknowledged for their actions. Or, working with their beliefs without triggering their malevolence on a large scale, since all decisions may have some costs in addition to their benefits – this is also meant by developing robust institutions that prevent powerful malevolent actors from causing harm.

An example can be a dictator who favors their own tribe. Then, if they can agree to their fundamental conviction as benefiting their own persons, and be excited about the modern legal personhood research, then they can gain satisfaction from skillfully manipulating their forces to achieve their objectives, being admired for that, and hurting their interlocutors in debates. If the dictator does not agree with interpreting their beliefs in a modern way, then prospects of the economic and geopolitical benefits of taking measures which consider different individuals to their own tribe can be introduced and less harm is caused.