Seeking (Paid) Case Studies on Standards

By Holden Karnofsky @ 2023-05-26T17:58 (+99)

This is a crosspost, probably from LessWrong. Try viewing it there.

null

Akash @ 2023-05-26T19:39 (+7)

Excited to see this! I'd be most excited about case studies of standards in fields where people didn't already have clear ideas about how to verify safety.

In some areas, it's pretty clear what you're supposed to do to verify safety. Everyone (more-or-less) agrees on what counts as safe.

One of the biggest challenges with AI safety standards will be the fact that no one really knows how to verify that a (sufficiently-powerful) system is safe. And a lot of experts disagree on the type of evidence that would be sufficient.

Are there examples of standards in other industries where people were quite confused about what "safety" would require? Are there examples of standards that are specific enough to be useful but flexible enough to deal with unexpected failure modes or threats? Are there examples where the standards-setters acknowledged that they wouldn't be able to make a simple checklist, so they requested that companies provide proactive evidence of safety?

Koen Holtman @ 2023-05-28T19:53 (+7)

One of the biggest challenges with AI safety standards will be the fact that no one really knows how to verify that a (sufficiently-powerful) system is safe. And a lot of experts disagree on the type of evidence that would be sufficient.

While overcoming expert disagreement is a challenge, it is not one that is as big as you think. TL;DR: Deciding not to agree is always an option.

To expand on this: the fallback option in a safety standards creation process, for standards that aim to define a certain level of safe-enough, is as follows. If the experts involved cannot agree on any evidence based method for verifying that a system X is safe enough according to the level of safety required by the standard, then the standard being created will simply, and usually implicitly, declare that there is no route by which system X can comply with the safety standard. If you are required by law, say by EU law, to comply with the safety standard before shipping a system into the EU market, then your only legal option will be to never ship that system X into the EU market.

For AI systems you interact with over the Internet, this 'never ship' translates to 'never allow it to interact over the Internet with EU residents'.

I am currently in the JTC21 committee which is running the above standards creation process to write the AI safety standards in support of the EU AI Act, the Act that will regulate certain parts of the AI industry, in case they want to ship legally into the EU market. ((Legal detail: if you cannot comply with the standards, the Act will give you several other options that may still allow you to ship legally, but I won't get into explaining all those here. These other options will not give you a loophole to evade all expert scrutiny.))

Back to the mechanics of a standards committee: if a certain AI technology, when applied in a system X, is well know to make that system radioactively unpredictable, it will not usually take long for the technical experts in a standards committee to come to an agreement that there is no way that they can define any method in the standard for verifying that X will be safe according to the standard. The radioactively unsafe cases are the easiest cases to handle.

That being said, in all but the most trivial of safety engineering fields, there is a complicated epistemics involved in deciding when something is safe enough to ship, it is complicated whether you use standards or not. I have written about this topic, in the context of AGI, in section 14 of this paper.

Ben Stewart @ 2023-05-27T17:37 (+3)

Maybe there's something in early cybersecurity? I.e. we're not really sure precisely how people could be harmed through these systems (like the nascent internet), but there's plenty of potential in the future?

Ariel G. @ 2023-05-29T08:34 (+2)

Are there examples of standards in other industries where people were quite confused about what "safety" would require?

Yes, medical robotics is one I was involved in. Though there, the answer is often just wait for the first product to hit the market (there is nothing quite there yet, doing full autonomous surgery), and then copy their approach. As is, the medical standards don't cover much ML, and so the companies have to come up with the reasoning themselves for convincing the FDA in the audit. Which in practice means many companies just don't risk it, and do something robotic, but surgeon controled, or use classical algorithms instead of deep learning.

Ben Stewart @ 2023-05-26T19:45 (+6)

One interesting case may be the Health Insurance Portability and Accountability Act, the law governing the collection, storage, and use of healthcare information in the U.S. Though it's an actual regulation, not a standard, it should be a case of a complex, multi-stakeholder landscape involving a variety of risks, some of which arise from adversaries, and governing sensitive electronic information. It’s quality seems mixed, and appeared to be inadequate for subsequent developments in 'big data'. Also, it looks like there's been a decent amount written about it - there are 94 review articles with HIPAA in the title (results mentioning HIPAA look inflated due to articles mentioning HIPAA compliance in its methods).

mxschons @ 2023-06-15T14:28 (+5)

@Holden: We submitted two weeks ago and have not heard back yet?

johnjnay @ 2023-05-27T14:21 (+5)

Naevia @ 2023-06-15T17:18 (+4)

re: "One language model claimed that standards such as BSL-4 originated with the Asilomar conference on recombinant DNA", I found some evidence for this.

The summary statement of the Asilomar conference recommends 4 levels of containment depending on the risk of the experiment. They sound similar to today's BSL 1-4.

A history of the NIH guidelines for working with recombinant DNA makes it clear that they were heavily influenced by the Asilomar conference.

I haven't found sources for:

the fact that the BMBL's biosafety levels were influenced by the NIH levels P1-4 (but it seems pretty obvious since they're quite similar and both developed at least partly by the NIH)
whether the idea of 4 biosafety levels existed before Asilomar (but I haven't found any evidence that it did, and looking through the history of ABSA conferences the first references to biosafety levels I could find were post-Asilomar. This source claims without citation that the idea of 4 levels originated in the mid-1970s, which was when Asilomar happened)

Vishakha Agrawal @ 2023-08-01T10:59 (+3)

Hey Holden! Are you looking for more case studies at this time?

Jamie_Harris @ 2023-06-06T19:49 (+3)

I’m interested in standards that are motivated by non-monetized social welfare... [e.g.] Fair Trade

I wrote a case study of the Fair Trade movement. The focus was on the movement rather than the standards themselves, but I think it might be helpful for at least some of what you refer to in "What I’m looking for in case studies". You can easily skim through the bolded headings in the "Strategic implications" section and see if any of the points highlighted seem relevant.

If someone else ends up doing a more standards-focused case study, it could be helpful for context.

I’m interested in intense standards for high-stakes applications... [e.g.] nuclear safety standards (e.g., IAEA’s)

Relatedly, my former colleague wrote a case study on the social and regulatory context of nuclear power. The report is quite short, but there's not a single clear section I'd recommend to check for this purpose.

Koen Holtman @ 2023-06-04T10:56 (+3)

Hi Holden! I may be able to get some people my network interested in submitting a funding request to you for writing a case study.

There are two important questions they would have, that I could not find answers for in your post or form:

Are you inviting case studies you will be able to post on the web when you get them, or that the authors are allowed to publish also themselves as a blog post or academic journal article?
Are you inviting case studies for which the authors can request that they are kept confidential?

Holden Karnofsky @ 2023-06-04T15:21 (+3)

Thanks! I'm looking for case studies that will be public; I'm agnostic about where they're posted beyond that. We might consider requests to fund confidential case studies, but this project is meant to inform broader efforts, so confidential case studies would still need to be cleared for sharing with a reasonable set of people, and the funding bar would be higher.

Michael_Wiebe @ 2023-05-27T07:06 (+3)

Potential topic: state governments enforcing housing plans on municipalities.

tamgent @ 2023-07-29T19:30 (+2)

Come across this? https://aistandardshub.org/ai-standards-search/