Concrete actionable policies relevant to AI safety (written 2019)

By weeatquince @ 2022-12-16T18:41 (+48)

This is a Draft Amnesty Day draft. That means it’s not polished, it’s probably not up to my standards, the ideas are not thought out, and I haven’t checked everything. I was explicitly encouraged to post something unfinished! 
Commenting and feedback guidelines: I’m going with the default — please be nice. But constructive feedback is appreciated; please let me know what you think is wrong. Feedback on the structure of the argument is also appreciated. 


[Below is an unedited copy and past of some old notes I made on AI policy in 2019. This was written as a personal project as a way of testing my fit before I started working on on future generations policy advocacy. I don’t know if I ever really planned to publish it but am publishing now for Draft Amnesty Day.

It is not well explained in the text but all I was doing here was taking some "best practices" from nuclear safety policy and applying them to AI safety. Which was an interesting exercise but does not quite reach the standard I would expect policy research.

I also think most of what is here (insofar as it can be applied at all) could be applied to biotech safety.

I no longer agree with a all of what is written here, and leave a few comments where this is the case. All comments and additions are in italics and square brackets. I also redact some links to other draft documents.]

 

A policy professional’s perspective on preventing TAI risks

Artificial intelligence is the future, not only for Russia, but for all humankind.
It comes with colossal opportunities, but also threats that are difficult to predict.

– Russian President Vladimir Putin

 

Introduction

Scope and audience of this paper

Audience.

This draft of this paper is written for individuals who are already concerned about the catastrophic, existential and extreme suffering risks associated with the development of Transformative AI (TAI). The question of how serious these risks are is not discussed here.

Aim.

This paper makes the case that there are concrete policies that can be implemented now by governments that would help mitigate TAI risks, and to set out a number of such policies. This could be used if opportunities arise to engage sympathetic policy makers on AI risks.

Scope.

This paper is focused only domestic AI safety policy, those that could be implemented by a single country and do not require international agreement. The policies come from examining what can be learned about AI safety policy from best practice in civil nuclear safety policy in the UK context. The policy suggestions are therefore within a UK policy context, although are hopefully general enough to be applicable in other countries as well.

 

Key recommendations for government

 

Further down the line, government should: 

[2022: Not that sure about any of these now]

Additionally, this paper recommends that academics concerned about AI safety should be looking to build or demonstrate academic consensus regarding the scale of AI risks.

 

Background 

Background to AI

Transformative AI (TAI)[1] is an AI that could lead to rapid radical changes in human welfare or power. Potentially an Artificial General Intelligence (AGI) that can do tasks of any type or even an Artificial Super Intelligence (ASI) that is smarter than humans as we are to animals. This concept is borrowed from and explored in depth in Professor Allan Dafoe’s AI Governance: A Research Agenda. Experts predicts a 50% chance that high-level AI will be developed around 2040-2050[2].

TAI risks. This paper is focused on what policies could be adopted now that could mitigate some of the tail end risks associated with TAI, in particular the risk of TAI being misused by bad actors or the risk of an ASI that is not controllable or aligned with human goals. Both of these situations could be potentially existential risks for humanity.  For the purpose of this paper this topic is referred to as “AI safety policy”. 

 

Background to AI policy

The growth of AI has captured the attention of corporates, the media, the public and state actors. States keen to bring AI start-ups and innovative research to their shores and reap the potential economic and societal benefits. This has led many states from China to Tunisia to adopt AI policies or national AI strategies.[i] These strategies cover topics such as supporting the AI industry, developing or capturing AI skills, improving digital infrastructure, funding R&D, data use, safeguarding public data, AI use by government and AI regulation. In each case these strategies sit alongside work on related topics such as supporting the tech industry, software regulation, privacy and data use.

Within this process states have recognised the need to understand and prepare for the sociological, economic, ethical and legal implications that may stem from the widespread adoption of AI technology. There has been minimal action at the state level to address the implications of TAI.

The UK context

So far in the UK the Government has:

 

The policy context

AI policy making sit within a wider context. It is important to remember that a government’s policies on AI will be driven largely by the more general approach a government takes to business data and technology policy. Similarly, AI safety policy will sit within the context of wider AI policy.  (It is possible AI policy will sit under defence rather than business but this does not appear to be the current trend).

It is also helpful to distinguish two kids of AI safety policy a government can adopt

This paper focuses on domestic AI safety policy

Data and tech policy

 
         
 

AI policy

 

  
   

Risk management

   
  

 

 

 

    
  Domestic

TAI Safety policy

    
  

 

 

 

    
  

 

 

 

    
  International     
         
         
         
This paper is on domestic AI safety policy. It is important to understand the broader context.

 

Domestic AI safety policy

Best practice

Drawing from nuclear safety, we can identify seven key principles that policymakers should consider in order to reduce the chances of a technological disaster, the 6 ‘R’s & ‘S’:

This chapter looks at all of these except for regulation which is dealt with in the next section.

https://www.iaea.org/publications/10883/governmental-legal-and-regulatory-framework-for-safety

 

Recognition and Responsibility

[Update 2022: My views on this now are a bit more nuanced. I don’t think that governments need to directly or explicitly recognise AI existential risks. What I do think is that any polices introduced such as regulations need to be robust to future worlds where various extreme AI risks become a possibility. This could be done by mapping out the short term AI societal risks and the range of possible long term AI risks and mapping across and ensuring the categorisation of and ways of responding to short term risks is not likely to pose barriers in any future more extreme risk scenarios.] 

Governments could recognise the implications that future technologies and global trends might have and recognise that some emerging technologies, such as the AI, biotech or geoengineering, may pose threat of catastrophic and potentially existential scale. Such recognition should be based as much as possible on a comprehensive scientific understanding of the subjects at hand and ideally informed public debate.

Recognition of a policy area or potential risk should be followed by assigning responsibility for mitigating and preparing for the risk to a Minister, Department, or position. That responsible political figure would be accountable to the public should doubts arise as to the policy or safety regime in place and they or their Department would be the point of contact for concerned parties.

Factors that affect how likely a country is to recognise and assign responsibility for a risk include: academic consensus, recognition of the risk by other countries, clear actions that can be taken after a risk is recognised, public and pressure group concern, political ideology and political will. It is worth noting that there is currently a lack of academic and expert consensus on the risks of TAI. From a policy perspective it is not clear that it would currently be sensible for a Government to recognise TAI risks.

Moving away from nuclear safety best practice, another government

The UK has a number of Horizon scanning processes spread across departments that look into future trends. It is likely that CDEI will be looking into the future trends associated with AI.

Risks are manged by the Cabinet Office. They collate risk into the National Risk Register of Civil Emergencies[3]. The register risks that “have at least a 1 in 20,000 chance of occurring in the UK in the next five years”[vii]. These risks are then distributed to relevant departments and the such that the Secretary of State of that department is responsible for risk mitigation and emergency preparedness policy. For example, the risk of a radiological release from a nuclear site is the responsibility of the Secretary of State of the Department for Business Energy and Industrial Strategy and staff in that department ensure that satisfactory polices are in place. There is also a separate occasional National Security Risk Assessment carried out by the MoD.[4]

Recommendations

Further down the line, government’s should

The UK context and UK recommendations

Some recognition of AI risks has already taken place within the UK. The House of Lord’s “AI in the UK: Ready, Willing, and Able?”[5] endorses the Centre for the Study of Existential Risk’s testimony on cyber-security, dual use technology, and the possible AI arms race – but stops short from recognising the risks stemming from superintelligence.

We recommend the UK

Additionally, academics concerned about AI safety should be looking to build or demonstrate academic consensus regarding the scale of AI risks, as without this it is unclear the extent to which good policy decision making processes should be picking up on these risks.

 

Research

[Update 2022: Less keen on this one. I would still support government funding to work on AI governance and ethics and maybe also on technical safety and misuse risk, but would worry about government AI funding fueling capabilities. I think any ask for government research funding for should be very carefully thought out with this in mind, for example restricting such funding in various ways.]

Countering risks stemming from AI, whether short term (e.g. deep fakes and fake news) or long term (e.g. existential risk) requires significant research, both technical research and policy research.

Funding should be provided to academia and the third sector to ensure that technical AI safety research is carried out, especially to incentivise researchers to look at topics that may not necessarily lead to short term gains or profits. This would have an added benefit of helping to boost the UK’s AI skill base.

Government officials are rarely experts in particular areas, but they do make up a large body of the expertise in policymaking, statecraft, international relations, and so forth. For good AI policy to be developed it is imperative that civil servants are directed to work on AI policy issues.

It is also important that there are good relationships between civil servants and academic experts. In particular cross cutting research will be needed to improve estimates of the potential risks better so that resources can be allocated appropriately.

Government funded R&D may be to support and encourage industry, where there is a market failure to produce a public good, or simply exploratory and academic purposes

Currently the UK Government funds the nuclear research though

At present the nuclear the main focus of research funding is not currently on safety concerns, although this is researched.

Recommendations

The UK context and UK recommendations

Research Councils UK have an annual budget of £3bn.[12] Just 0.1% of this could fully fund existing global technical AI research[13]. The Government’s Industrial Strategy Challenge Fund’s earmarked £93m for safer AI.[14] (In compassion the UK government are investing around £460 million in nuclear research and innovation).

Civil Servants are being ask to research issues about ethics and technology and AI within CDEI.

The UK government should fund AI safety and ethics research including funding efforts seeking to better estimate AI risks. Bodies such as CDEI should also ensure there are open communication streams between policymakers and technical researchers and the third sector.

 

Record

[Update 2022: Seems like a nice to have but not a key ask]

Best practice & existing precedent

In nuclear safety, when a serious nuclear or radiological problem occurs, countries will rate this event on the International Nuclear and Radiological Event Scale (INES) and if it is serious enough (and does not threaten national security to do so) the country will report the event to the International Atomic Energy Agency’s (IAEA) Unified System for Information Exchange in Incidents and Emergencies (USIE). The event details are shared globally by the Nuclear Events Web-based System (NEWS). There are also strong working relationships between nuclear operators (and regulators) in different countries. So when evidence of a malfunction was discovered in a nuclear power plant in the UK it quickly became apparent that a plant in the US had dealt with similar issues and the lessons from that plan could be applied to the UK.

In cybersecurity there are various vulnerability disclosure processes in place to ensure rapid information sharing and appropriate responses from corporate actors. Governments have played a role in developing these processes and ensuring best practice even across international boundaries, for example the Dutch Coordinated Vulnerability Disclosure Guidelines.

There have been numerous reports of AI systems behaving unexpectedly and unpredictably.[15] Whilst this is not necessarily problematic for games of Go, it may lead to danger should it occur at a large scale, or in scenarios where human lives are at risk (e.g. self-driving cars) or in the development of TAI.  Technical AI safety researchers are already looking into setting up honeypots[16] to help identify unusual behaviour by advanced AIs. It could be useful to ensure that information on unexpected behaviour is shared across AI labs.

No data sharing system for AI safety currently exists. Such as system could be set up unilaterally by a state or non-state actor, although international agreement would be ideal to ensure it’s use. 

Recommendation

 

 

Resilience: AI defence and safety

[Update 2022: Not at the stage where I think there is much to do here yet, except maybe general support for research and projects aimed at civilizational resilience and recovery.]

In 1961, “a bomber was flying over North Carolina, when it lost a wing, and two of its nuclear bombs fell to the ground […]. One of the bombs broke on impact after its parachute failed. The other landed unharmed, but five of its six safety devices also failed. Defence Secretary Robert McNamara had this to say: "by the slightest margin of chance, literally the failure of two wires to cross, a nuclear explosion was averted.””[17]

Best practice in risk prevention is ‘defence in depth’. This means that a system is protected by a series of independent layers of defensive mechanisms (or ‘fail-safes’) such that if one mechanism fails, another will already be in place to protect against unexpected failures. To prevent TAI safety risks, AI researchers are already looking at how they can institute fail-safes. 

Governments have capabilities and expertise in defence, security and cyber security. One way the Government can help find more fail-safes to help defend against AI risks is to increase the space in which we are looking for options: ideas such as manual internet/electricity shut-offs, or investments in general computer security to complicate the creation of botnets, should be studied further. [18]

As well as ASI safety concerns there may be a host of policy political and governance issues that could arise very rapidly from the development of TAI that, further down the line, it may be helpful for governments to prepare for. 

Recommendation

Further down the line the Government and the MoD may want engage with AI safety researchers to put in place additional layers defence in depth and other measures to mitigate risks and support the resilience to shocks from the potential development of TAI.

 

Safeguarding

[Update 2022: No strong views on this. I expect it is a bit over the top for now which risks escalating race dynamics, but an idea to bare in mind. Maybe some good policies on supply chains here. Not really my area of expertise.]

“As the best AI will be expensive, digital theft will pay.26 We can expect to see sophisticated cyber-attacks against companies like Google’s DeepMind, IBM or Facebook if they are perceived to have the best AI code. Defending such civil/commercial assets may become an issue of national security”

Ministry of Defence, UK[19]

In the nuclear industry “safeguarding” is the term used to refer to the variety of mechanisms put in place nationally and internationally to prevent nuclear material intended for peaceful purposes being used for the development of nuclear weapons, as in line with the Nuclear Non-Proliferation Treaty (NPT). In practice in the UK this mean security clearances for staff in sensitive positions and high fences and the armed Civil Nuclear Constabulary at nuclear sites.

There is significant consensus that the misuse of AI, especially TAI by hostile actors poses a significant threat to global security. As such government’s who have industries that are developing AI should look to protect these AI assets from hostile actors who might use them for nefarious ends. Government defence and security capabilities should be used to support the cybersecurity of researchers and academics working on AI who might lack the ability to defend the code they develop from malicious threats. Government could also provide background personnel checks on individuals to make sure they are not influenceable by outside forces (or eventually by an AGI itself).

Recommendation

The government should consider if the UK has high risk AI assets and if so look to protect those assets: for example, by requiring and providing security clearances for staff at Deepmind and by MoD or the National Cyber Security Centre (NCSC) ensuring that DeepMind has excellent cybersecurity protection.

+ supply chain 

 

Regulation

[Update 2022: I am still very very keen on AI regulatory policy, if we can do it well. I think we can construct a world whereby there are skilled risk aware actors (regulators) who can step in at an emergency crunch point to pull the plug but otherwise do minimal except support safety work, prevent AI misuse, have some oversight and ensure AI companies have good governance structures. I think this world would be great. I think the community of people concerned about AI should be thinking much more about this.]

Based largely on memory but mostly draws from the International Atomic Energy Agency’s (IAEA) Safety Standards, in particular Governmental, Legal and Regulatory Framework for Safety, General Safety Requirements Part 1

Regulate

There is a lot written by governments and academics on best practice in regulation. Within the UK Government best practice is set out in the Regulators Code[20] and more recently the work of the Regulatory Futures Review[21]. For the uninitiated, Annex A looks to dispel some of the preconceptions about regulation. Including dispelling the ideas that regulation means having a detailed rulebook, being a burden on industry, watching everything industry does, enforcing rules using legal powers and so on. 

In the UK the nuclear Regulator is the Office for Nuclear Regulation (ONR). The (ONR) is responsible for regulation of the UK’s nuclear industry. They are required under the Energy Act to ensure nuclear safety, conventional safety, security and safeguarding on civil nuclear sites. They are an arm’s length Government body that reports to Parliament and are 97% funded by fees charged to dutyholders (nuclear sites). They use outcome focused regulation and dutyholders need to demonstrate that they are meeting the licence conditions, but the way they meet them is up to the dutyholder. The ONR are comprised of a body of experts, known as inspectors, who tend to have considerable experience in the nuclear industry. Inspectors monitor actions of dutyholders inspections, site self-reporting and site safety cases or other independent reports. The inspectors have a good relationship with the sites and 90 of the time soft enforcement (eg. a quick constructive conversation) is sufficient to address an issue. There is a separate regulator for Defence sites, the Defence Nuclear Safety Regulator (DNSR).

Another example of best practice UK regulation is the acclaimed Human Fertilisation and Embryology Authority (HFEA).

Drawing from best practice in regulation and especially from nuclear industry we can draw a number of conclusions about what good regulation of AI should look like.

Ideally the remit of an AI regulator should include:

The rules should be:

This tech regulator should be set up such that it is:

Recommendation

The UK Government should set up a regulator in the form of a well-funded independent body of technical experts is given very high-level goals to ensure the safe and ethical behaviour of the technology industry (similar to HEFA or the ONR).

+ Put in place whistleblowing mechanisms

+ Look also at the corporate structures and how responsibility is assigned to individuals within the organisations 

 

Concluding thoughts 

Recommendations

In the UK context

The paper recognises the positive steps that the UK Government has made to date in particular:

 

International agreements

The author of this paper is not an expert in international treaties, or international security mechanisms but it certainly seems likely that international policy and the international stage has a role to play in ensuring that if TAI is developed the transition goes well for humanity.

In the first instance regulatory expertise is often shared and developing a good regulatory system in the UK will pave the way for good AI regulation globally.

Additionally, international agreements could be reached on the ethical use of AI (an idea suggested by former UK digital and culture secretary Matt Hancock[22]), on the development of AI weapons, on international safeguarding and non-proliferation processes and on the right level of international data sharing and reporting of errors. Potentially even an international body for sharing regulator best practice could be set up similar to the IAEA. 

 

Moving forwards

That said, there may not be much choice. Governments are already discussing these issues, creating specialised statutory bodies, and providing finance to early stage projects. And all it needs now is for a serious event, which could be a massive cyber-attack using a machine learning algorithm, or could be the hacking of a self-driving car to murder a person, and the public may start shouting for regulation. It would be good to have the above policy suggestions at hand for when that happens.

 

Annex A : introduction to Regulation

For the uninitiated, Annex A looks to dispel some of the preconceptions about regulation. Including:

  1. Regulators are a part of Government. There is a scale from total industry self-regulation to government run regulators. Most regulatory bodies have some degree of independence from government and the Regulatory Futures Review recommended that government regulators be funded by dutyholders (the bodies being regulated) rather than government grants.
  2. Regulation does not mean having a detailed rulebook. Best practice is to have outcome focused regulations and to allow the dutyholders to work out how to achieve the outcomes. 
  3. Regulation does not mean being a burden on industry. Having a body of experts check that a system is safe provides a benefit to the dutyholder. Furthermore, a regulator may have a secondary responsibility to support industry growth. More perversely big corporates may like regulation as it keeps smaller organisations out of the market. 
  4. Enforcing regulation does not mean watching everything industry does. Dutyholders may self-report. Inspectors might only go to sites if they already expect there is a problem, or may use sampling methods to test the waters and look for evidence of poor process or risk taking.
  5. Enforcing regulation does not mean using legal powers. Although they will have legal enforcement powers, most of the work of a regulator is to keep dutyholders within the law without having to resort to legal powers.
  6. Regulation does not mean letting government off the hook. Regulators can have oversight of Government activities as well as corporate activities.
  7. Different regulators specialise in different topics or type of risk. This is somewhat true, and certainly topic expertise is important. But more important is the relationship formed with dutyholders, and different regulators regulate different parts of industry / government. (Eg the nuclear regulator regulates the nuclear industry, including conventional health and safety matters)

 

 

[1] https://www.fhi.ox.ac.uk/wp-content/uploads/GovAIAgenda.pdf 

[2] Nick Bostrom. (2014). Future Progress in Artificial Intelligence: A Survey of Expert Opinion. [online] Available at: https://nickbostrom.com/papers/survey.pdf [Accessed 15 Jul. 2018].

[3] https://www.gov.uk/government/publications/national-risk-register-of-civil-emergencies-2017-edition

[4] There was a National Security Risk Assessment in 2015 that fed into the National Security Strategy 2015. See: https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/478936/52309_Cm_9161_NSS_SD_Review_PRINT_only.pdf and https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/62484/Factsheet2-National-Security-Risk-Assessment.pdf

[5] House of Lords, (2018). AI in the UK: Ready, willing and able? Report of Session 2017-19. [online] Available at: https://publications.parliament.uk/pa/ld201719/ldselect/ldai/100/100.pdf [Accessed 15 Jul. 2018].

[6] https://publications.parliament.uk/pa/ld201719/ldselect/ldintfair/329/329.pdf

[7] A suggestion of Lord John Bird https://www.london.edu/news-and-events/news/big-issue-founder-dismantle-poverty-1133

[8] https://www.gov.uk/government/publications/green-book-supplementary-guidance-discounting

[9] https://www.cser.ac.uk/resources/representation-future-generations/

[10] https://www.gov.uk/government/news/announcement-on-the-national-nuclear-laboratory

[11] Eg. https://www.fnc.co.uk/news-and-events/news-archive/2018/frazer-nash-secures-contract-to-support-the-government-s-nuclear-research-and-development-programme.aspx

[12] Gov.UK. (2017). Funding for nuclear innovation. [online] Available at: https://www.gov.uk/guidance/funding-for-nuclear-innovation [Accessed 15 Jul. 2018].

[13] Based on the budgets of MIRI and CSER and GCRI.

[14] https://www.ukri.org/innovation/industrial-strategy-challenge-fund/robots-for-a-safer-world/ 

[15] Metz, C. (2018). The Sadness and Beauty of Watching Google's AI Play Go. [online] WIRED. Available at: https://www.wired.com/2016/03/sadness-beauty-watching-googles-ai-play-go/ [Accessed 15 Jul. 2018]. Key citation: “It's not a human move.”

[16] https://en.wikipedia.org/wiki/Honeypot_(computing)

[17] Future of Life Institute. (2018). Accidental Nuclear War: A Timeline of Close Calls - Future of Life Institute. [online] Available at: https://futureoflife.org/background/nuclear-close-calls-a-timeline/ [Accessed 15 Jul. 2018].

[18] Duettmann, A., Miller, M. and Peterson, C. (2018). Cyber, Nano, and AGI Risks: Decentralized Approaches to Reducing Risks. [online] Available at: https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/46290.pdf  [Accessed 15 Jul. 2018].

[19] P19 https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/709359/20180517-concepts_uk_human_machine_teaming_jcn_1_18.pdf

[20] https://www.gov.uk/government/publications/regulators-code 

[21] https://www.gov.uk/government/publications/regulatory-futures-review

[22] Government Computing Network. (2018). Hancock wants international regulatory framework on AI - Government Computing Network. [online] Available at: https://www.governmentcomputing.com/brexit-eu/news/hancock-wants-international-regulatory-framework-ai [Accessed 15 Jul. 2018].


 

[i] An Overview of National AI Strategies, Tim Dutton, https://medium.com/politics-ai/an-overview-of-national-ai-strategies-2a70ec6edfd

[ii] p39, Industrial Strategy, HM Government, https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/664563/industrial-strategy-white-paper-web-ready-version.pdf

[iii] The AI Council was a recommendation of the Independent report on Growing the artificial intelligence industry in the UK, https://www.gov.uk/government/publications/growing-the-artificial-intelligence-industry-in-the-uk

[iv] https://www.gov.uk/guidance/emerging-technology-for-defence-programme

[v] https://www.gov.uk/government/organisations/centre-for-connected-and-autonomous-vehicles

[vi] https://www.parliament.uk/documents/lords-committees/science-technology/life-sciences-industrial-strategy/Govt-response-Life-Sciences-Industrial-Strat.pdf

[vii] https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/644968/UK_National_Risk_Register_2017.pdf 

[viii] If a risk materialises but originates from another country the responsible Minister can avoid the blame for not acting. For example, consider the inability of the EU to progress with Bank Structural Reform, regulations for financial stability, plausibly as a result of self-interested state actors refusing to work together. https://ec.europa.eu/info/business-economy-euro/banking-and-finance/financial-supervision-and-risk-management/managing-risks-banks-and-financial-institutions/structural-reform-eu-banking-sector_en https://www.bloomberg.com/news/articles/2017-10-24/eu-buries-its-plans-to-split-up-retail-and-investment-banking