Centre for the Study of Existential Risk: Six Month Report May-October 2018

By HaydnBelfield @ 2018-11-30T20:32 (+26)

We have just prepared a Six Month Report for our Management Board. This is a public version of that Report.  We send short monthly updates in our newsletter – subscribe here.

Contents:

  1. Overview
  2. Policy and Industry Engagement
  3. Academic Engagement
  4. Public Engagement
  5. Recruitment and research team
  6. Expert Workshops and Public Lectures
  7. Upcoming activities
  8. Publications

1. Overview

The Centre for the Study of Existential Risk (CSER) is an interdisciplinary research centre within the University of Cambridge dedicated to the study and mitigation of risks that could lead to civilizational collapse or human extinction. We study existential risk, develop collaborative strategies to reduce them, and foster a global community of academics, technologists and policy-makers working to tackle these risks. Our research focuses on Global Catastrophic Biological Risks, Extreme Risks and the Global Environment, Risks from Artificial Intelligence, and Managing Extreme Technological Risks.

Our last Management Board Report was in May 2018. Over the last six months we have continued to advance existential risk research and grow the community working in the field:

2. Policy and Industry Engagement

We have had the opportunity to speak directly with policymakers, industry-leaders and institutions across the world who are grappling with the difficult and novel challenge of how to unlock the socially beneficial aspects of new technologies while mitigating their risks. Through advice and discussions, we have the opportunity to reframe the policy debate and to hopefully shape the trajectory of these technologies themselves.

3.    Academic Engagement:

As an interdisciplinary research centre within the University of Cambridge, we seek to grow the academic field of existential risk research, so that this important topic receives the rigorous and detailed attention it deserves.

4.    Public Engagement:

We’re able to reach far more people with our research:
- Since our new site launched in Aug 2017, we’ve had 53,726 visitors.
- 6,394 newsletter subscribers, up from 4,863 in Oct 2016.
- Facebook followers have tripled since Dec 2016, from 627 to 2,049.
- Twitter followers have sextupled since Dec 2016, from 778 to 5,184.

5.    Recruitment and research team:

We have just appointed a new Research Project Administrator – Clare Arnstein, who will start in early December, and is currently Executive Assistant to the Vice Chancellor (on secondment from the School of Arts and Humanities). We have also just recruited an additional Senior Research Associate as an Academic Programme Manager.

New Postdoctoral Research Associates:

Visiting researchers:

New CSER Research Affiliates:

6.     Expert Workshops and Public Lectures:

Our events over the last few months have included:

7.     Upcoming activities

Four books will be published in early 2019:

Upcoming events:

Timing to be confirmed:

8.    Publications

Adrian Currie (ed.) (2018) Special Issue: Futures of Research in Catastrophic and Existential Risk. Futures.

Many of the fifteen papers in the Special Issue were originally presented at our first Cambridge Conference on Catastrophic Risk in 2016, and it includes three papers by CSER researchers:

“We present a novel classification framework for severe global catastrophic risk scenarios. Extending beyond existing work that identifies individual risk scenarios, we propose analysing global catastrophic risks along three dimensions: the critical systems affected, global spread mechanisms, and prevention and mitigation failures. The classification highlights areas of convergence between risk scenarios, which supports prioritisation of particular research and of policy interventions. It also points to potential knowledge gaps regarding catastrophic risks, and provides an interdisciplinary structure for mapping and tracking the multitude of factors that could contribute to global catastrophic risks.”

“There has been much discussion of the moral, legal and prudential implications of geoengineering, and of governance structures for both the research and deployment of such technologies. However, insufficient attention has been paid to how such measures might affect geoengineering in terms of the incentive structures which underwrite scientific progress. There is a tension between the features that make science productive, and the need to govern geoengineering research, which has thus far gone underappreciated. I emphasize how geoengineering research requires governance which reaches beyond science’s traditional boundaries, and moreover requires knowledge which itself reaches beyond what we traditionally expect scientists to know about. How we govern emerging technologies should be sensitive to the incentive structures which drive science.”

The rest of the papers are:

Scientific communities and existential risk

“Scientific freedoms are exercised within the context of certain responsibilities, which in some cases justify constraints on those freedoms. (Constraints that may be internally established within scientific communities and/or externally enacted.) Biosecurity dimensions of work involving pathogens are one such case and raise complex challenges for science and policy. The central issues and debates are illustrated well in the development of responses to publication of (‘gain of function’) research involving highly pathogenic avian influenza, by a number of actors, including scientists, journal editors, scientific academies, and national and international policy groups.”

“The special issue Creativity, Conservatism & the Social Epistemology of Science collects six papers which, in different ways, tackle 'promotion questions' concerning scientific communities: which features shape those communities, and which might be changed to promote the kinds of epistemic features we desire. In this introduction, I connect these discussions with more traditional debate in the philosophy of science and reflect upon the notions of creativity which underwrite the papers.”

“Existential risks, particularly those arising from emerging technologies, are a complex, obstinate challenge for scientific study. This should motivate studying how the relevant scientific communities might be made more amenable to studying such risks. I offer an account of scientific creativity suitable for thinking about scientific communities, and provide reasons for thinking contemporary science doesn't incentivise creativity in this specified sense. I'll argue that a successful science of existential risk will be creative in my sense. So, if we want to make progress on those questions we should consider how to shift scientific incentives to encourage creativity. The analysis also has lessons for philosophical approaches to understanding the social structure of science. I introduce the notion of a ‘well-adapted’ science: one in which the incentive structure is tailored to the epistemic situation at hand.”

Government reactions to disasters

“In East Asia, disasters have been regarded as events which uncover the mistakes of the past as much as they provide opportunities for building a more just society. In Japan, this phenomenon was captured through the concept of “world rectification” (yonaoshi) in the past and continues to lead to the improvement of disaster preparedness to this day. In the same way, disasters in historical China were not only interpreted as expressions of heavenly wrath for a ruler’s mistakes, but also as an opportunity for better governance. Taking into account the way in which disasters simultaneously mirror existing trajectories and open up space for new ones, this chapter compares the protection of disaster victims in China and Japan by looking at two recent catastrophes, the 2008 earthquake in Wenchuan and the earthquake, tsunami and nuclear meltdown of 11 March 2011 in eastern Japan. We pay particular attention to the framing of both disasters as either man-made or natural, which carries significant social and political implications. Both governments made use of this distinction to shrug off responsibility and to influence mobilisation processes among the victims. The distinction between man-made and natural disasters also had a significant influence on the resulting institutionalisation processes.”

Environmental assessment of high-yield farming

“How we manage farming and food systems to meet rising demand is pivotal to the future of biodiversity. Extensive field data suggest that impacts on wild populations would be greatly reduced through boosting yields on existing farmland so as to spare remaining natural habitats. High-yield farming raises other concerns because expressed per unit area it can generate high levels of externalities such as greenhouse gas emissions and nutrient losses. However, such metrics underestimate the overall impacts of lower-yield systems. Here we develop a framework that instead compares externality and land costs per unit production. We apply this framework to diverse data sets that describe the externalities of four major farm sectors and reveal that, rather than involving trade-offs, the externality and land costs of alternative production systems can covary positively: per unit production, land-efficient systems often produce lower externalities. For greenhouse gas emissions, these associations become more strongly positive once forgone sequestration is included. Our conclusions are limited: remarkably few studies report externalities alongside yields; many important externalities and farming systems are inadequately measured; and realizing the environmental benefits of high-yield systems typically requires additional measures to limit farmland expansion. Nevertheless, our results suggest that trade-offs among key cost metrics are not as ubiquitous as sometimes perceived.”

Issues in decision theory relevant to advanced artificial intelligence

“Can an agent deliberating about an action A hold a meaningful credence that she will do A? ‘No’, say some authors, for ‘deliberation crowds out prediction’ (DCOP). Others disagree, but we argue here that such disagreements are often terminological. We explain why DCOP holds in a Ramseyian operationalist model of credence, but show that it is trivial to extend this model so that DCOP fails. We then discuss a model due to Joyce, and show that Joyce’s rejection of DCOP rests on terminological choices about terms such as ‘intention’, ‘prediction’, and ‘belief’. Once these choices are in view, they reveal underlying agreement between Joyce and the DCOP-favouring tradition that descends from Ramsey. Joyce’s Evidential Autonomy Thesis is effectively DCOP, in different terminological clothing. Both principles rest on the so-called ‘transparency’ of first-person present-tensed reflection on one’s own mental states.”

Theoretical mapping of artificial intelligence

“We present nine facets for the analysis of the past and future evolution of AI. Each facet has also a set of edges that can summarise different trends and contours in AI. With them, we first conduct a quantitative analysis using the information from two decades of AAAI/IJCAI conferences and around 50 years of documents from AI topics, an official database from the AAAI, illustrated by several plots. We then perform a qualitative analysis using the facets and edges, locating AI systems in the intelligence landscape and the discipline as a whole. This analytical framework provides a more structured and systematic way of looking at the shape and boundaries of AI.”

“We analyze and reframe AI progress. In addition to the prevailing metrics of performance, we highlight the usually neglected costs paid in the development and deployment of a system, including: data, expert knowledge, human oversight, software resources, computing cycles, hardware and network facilities, development time, etc. These costs are paid throughout the life cycle of an AI system, fall differentially on different individuals, and vary in magnitude depending on the replicability and generality of the AI solution. The multidimensional performance and cost space can be collapsed to a single utility metric for a user with transitive and complete preferences. Even absent a single utility function, AI advances can be generically assessed by whether they expand the Pareto (optimal) surface. We explore a subset of these neglected dimensions using the two case studies of Alpha* and ALE. This broadened conception of progress in AI should lead to novel ways of measuring success in AI, and can help set milestones for future progress.”

“New types of artificial intelligence (AI), from cognitive assistants to social robots, are challenging meaningful comparison with other kinds of intelligence. How can such intelligent systems be catalogued, evaluated, and contrasted, with representations and projections that offer meaningful insights? To catalyse the research in AI and the future of cognition, we present the motivation, requirements and possibilities for an atlas of intelligence: an integrated framework and collaborative open repository for collecting and exhibiting information of all kinds of intelligence, including humans, non-human animals, AI systems, hybrids and collectives thereof. After presenting this initiative, we review related efforts and present the requirements of such a framework. We survey existing visualisations and representations, and discuss which criteria of inclusion should be used to configure an atlas of intelligence.”

undefined @ 2018-12-03T22:53 (+8)

Thanks for writing up such a thorough report!

I was interested to see Yasmine Rix on your list of research affiliates; this is the first time I've heard of an X-risk organization working with an artist on public outreach, and it's a neat idea.

Has this kind of work been on CSER's agenda for a while/did you reach out to Yasmine? Did she reach out to you with the suggestion before you'd considered art? I'm curious about how the collaboration came to exist, and which benefits CSER thinks might arise from public outreach through art. (Is there a particular type of audience/media outlet you'd like to reach in this way that wouldn't be reachable through publications?)

undefined @ 2018-12-04T15:05 (+5)

Thanks Aaron! We got to know Yasmine during our time based at the Cambridge Conservation Initiative. She was working there at the time, became a regular at our talks and discussions, and has remained involved with us since.

Ground Zero Earth is her initiative, although she has sought intellectual input from us throughout. She secured funding for it through an arts council grant. So from our side, it was a low-cost way to be involved with a pretty cool project.

This hasn't been an explicit part of our agenda, although it fits with our broader aim of engaging across society on existential risk and long-term thinking. With that said, we've engaged with and included a number of artists and cultural scholars who have reached out to us (e.g. Kathryn Brimblecombe-Fox: https://kathrynbrimblecombeart.blogspot.com/2018/06/existential-risk-research-art.html and a few of our researchers including Adrian Currie and myself have spoken at cultural events of different sorts. E.g. recently, CSER's Luke Kemp took part in an event organised by the Long Term Inquiry in London (https://medium.com/@thelongtimeinquiry/the-long-time-3383b43d42ab) which aims to be a cultural equivalent of CSER, considering topics such as our responsibility to future generations, etc.

Re: audiences and benefits of this sort of engagement, I don't think I can give an answer of EA-level rigour here (as a scientist, this isn't my area of comparative competence) so I'll wait for someone else from our group to chime in. But I'd make the broad (and I imagine, obvious) point that there's probably a lot to be said for engaging with arts and culture when it comes to engaging broader communities (and probably also scientific and policy experts) in thinking about somewhat abstract ideas such as existential risk and our responsibility to the future. I see this as complementary to the sort of scientific and governance-focused research that makes up the mainstay of our work.