Recommendation to Apply ISIC and NAICS to AI Incident Database

By Ben Turse @ 2024-07-21T07:25 (+1)

I wrote this piece as my capstone for the June 2024 BlueDot AI Safety Fundamentals: Governance course. Thank you Mac Jordan for facilitating the course and providing feedback on this post.

Introduction and Recommendation

The AI Incident Database (AIID) tracks detailed information on harms caused by the behavior of AI systems. The database schema is based on the Center for Security and Emerging Technology AIID Harm Taxonomy (AHT).  AHT element Sector of Deployment captures high level industry information. This element partially implements the UN International Standard Industrial Classification of All Economic Activities (ISIC) and the North American Industry Classification System (NAICS). 

Recommendation: AHT should fully implement both ISIC and NACIS.

ISIC and NAICS Overview

ISIC is "the international reference classification of productive activities" (United Nations,  p. iii). 

• Many countries have adopted ISIC to categorize industrial activity. 
  • The EU Industry Classification System (NACE) is derived from ISIC.
  • ISIC is the conceptual foundation of NAICS.

ISIC and NAICS use a hierarchical structure to organize industrial activity. Each level of the hierarchy provides a more specific level of categorization.

ISIC and NAICS Benefits

Adopting ISIC and NAICS will provide the following benefits:

• Enable more consistent incident reporting by relying on carefully defined categories and classification guidelines.
  • ISIC and NAICS documentation provide definitions and examples for all domain values.
• Allow AI incidents to be categorized at more granular industry levels. 
  • According to the United Nations: "[f]or analytical purposes, it is important to implement ISIC at its lower levels of detail..." (United Nations, p.5).
• Allow analysts to easily combine AI incident data with other datasets to facilitate robust analyses.
  • Many U.S. open datasets categorize information by NAICS (ie: Injury Tracking Application). Fully adopting NAICS would allow analysts to study relationships between trends in work-related injury and AI incidents by specific sub-industries.

Implementing two related standards creates additional labeling burden. However, each standard provides a unique benefit.

• ISIC supports international AI incident classification interoperability.
• NAICS increases analytics value of North American incidents.
  • Over half of reported AI incidents with a Location Region are in North America.

ISIC and NAICS Migration and Labeling

Recommended migration activities by level of effort:

• Low: remove Sector of Deployment values "unclear" and "other."
• Medium: derive ISIC and NAICS wherever a one-to-one mapping exists with a Sector of Deployment domain value.
• High: Review all existing incidents and apply the the appropriate ISIC and NAICS categories.
  • Refer to official examples and definitions when processing incident reports. 
  • Apply the most granular discernible ISIC and NAICS categories.

Additional Recommendations

Critical Infrastructure Sector (CIS) segments:

• AHT element Infrastructure Sectors resembles CIS categories.
• Some CIS categories contain segments not reflected by Infrastructure Sectors. These segments should be added to the schema.
  • Ex: Energy CIS sector includes electricity, oil, and natural gas.

European Institute of Innovation and Technology (EIIT) AI Capabilities:

• AHT should capture capabilities demonstrated by the AI system relevant to the harm based upon the EIIT AI capabilities taxonomy.
• This taxonomy could provide structure to AHT element AI Task.

Entity Type

• AHT element Entity Type should be replaced with the Federal Integrated Business Framework (FIBF) Business Type values.  

Image attribution: https://unsplash.com/photos/a-row-of-smoke-stacks-in-front-of-a-building-Xe-4WEJ80vQ