Designing Incident Reporting Systems for Harms from General-Purpose AI
Pith reviewed 2026-05-18 00:13 UTC · model grok-4.3
The pith
A seven-dimensional framework drawn from other safety industries can guide the design of systems for reporting harms from general-purpose AI.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
We introduce a conceptual framework and provide considerations for the institutional design of AI incident reporting systems, i.e., processes for collecting information about safety- and rights-related events caused by general-purpose AI. Through a literature review, we develop a framework for understanding the institutional design of AI incident reporting systems, which includes seven dimensions: policy goal, actors submitting and receiving reports, type of incidents reported, level of risk materialization, enforcement of reporting, anonymity of reporters, and post-reporting actions. We then examine nine case studies of incident reporting in safety-critical industries to extract design for
What carries the argument
The seven-dimension framework that organizes institutional choices for incident reporting across policy goal, submitting and receiving actors, incident types, risk materialization level, enforcement, reporter anonymity, and post-reporting actions.
If this is right
- Regulatory agencies may enforce reporting thresholds more consistently than non-regulatory bodies when applied to AI systems.
- Including near-miss events can surface potential AI harms before they fully materialize into damage.
- Combining mandatory thresholds with voluntary channels can increase report volume while lowering barriers for participants.
- Focusing post-report actions on safety learning and information sharing can improve subsequent AI deployments.
- Clarifying legal protections around reporting can raise participation rates without creating new liability concerns.
Where Pith is reading between the lines
- The same seven dimensions could be used to compare emerging AI reporting rules across different countries.
- A small-scale pilot of an AI incident system based on this framework could show which dimensions require modification for AI's fast pace of change.
- Linking reported incidents to technical evaluations of model capabilities might create a stronger feedback loop for risk reduction.
Load-bearing premise
That design lessons extracted from nine case studies in other safety-critical industries will transfer effectively to general-purpose AI systems despite differences in scale, opacity, and rapid capability change.
What would settle it
A systematic review of actual AI incidents that finds no measurable safety improvements or policy changes traceable to reporting systems built on these seven dimensions would challenge the framework's usefulness.
Figures
read the original abstract
We introduce a conceptual framework and provide considerations for the institutional design of AI incident reporting systems, i.e., processes for collecting information about safety- and rights-related events caused by general-purpose AI. As general-purpose AI systems are increasingly adopted, they are causing more real-world harms and displaying the potential to cause significantly more dangerous incidents - events that did or could have caused harm to individuals, property, or the environment. Through a literature review, we develop a framework for understanding the institutional design of AI incident reporting systems, which includes seven dimensions: policy goal, actors submitting and receiving reports, type of incidents reported, level of risk materialization, enforcement of reporting, anonymity of reporters, and post-reporting actions. We then examine nine case studies of incident reporting in safety-critical industries to extract design considerations for AI incident reporting in the United States. We discuss, among other factors, differences in systems operated by regulatory vs. non-regulatory government agencies, near miss reporting, the roles of mandatory reporting thresholds and voluntary reporting channels, how to enable safety learning after reporting, sharing incident information, and clarifying legal frameworks for reporting. Our aim is to inform researchers and policymakers about when particular design choices might be more or less appropriate for AI incident reporting.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript develops a conceptual framework for the institutional design of incident reporting systems for harms from general-purpose AI systems. Drawing on a literature review, it identifies seven dimensions (policy goal, actors submitting and receiving reports, type of incidents reported, level of risk materialization, enforcement of reporting, anonymity of reporters, and post-reporting actions). It then analyzes nine case studies from safety-critical industries to extract design considerations for AI incident reporting in the United States, discussing topics such as regulatory versus non-regulatory systems, near-miss channels, mandatory thresholds, voluntary reporting, safety learning, information sharing, and legal frameworks.
Significance. If the extracted considerations prove transferable, the work could offer timely, structured guidance for policymakers and researchers seeking to build effective AI incident reporting mechanisms. It fills a gap in AI governance literature by synthesizing cross-domain lessons into a seven-dimension framework and highlighting practical design trade-offs, potentially supporting better accountability and safety learning as general-purpose AI capabilities advance.
major comments (1)
- [Section 4] Section 4: The mapping of features from the nine non-AI case studies (e.g., mandatory thresholds, near-miss reporting, anonymity provisions) onto AI does not supply explicit adaptations for AI-specific properties such as model opacity and rapid capability change. The text acknowledges these differences at a high level but leaves unaddressed whether opacity would undermine reporting incentives or whether post-reporting learning loops would function similarly; without such adaptations or a clear invariance argument, the central claim that the extracted considerations inform AI system design rests on an unvalidated transfer assumption.
minor comments (1)
- The abstract and introduction would benefit from a brief enumeration of the nine case-study industries to allow readers to assess domain coverage at a glance.
Simulated Author's Rebuttal
We thank the referee for their constructive comments on our manuscript. The feedback on the transferability of design considerations from non-AI domains to AI incident reporting systems is particularly valuable. We address this point in detail below and have made revisions to strengthen the manuscript.
read point-by-point responses
-
Referee: [Section 4] Section 4: The mapping of features from the nine non-AI case studies (e.g., mandatory thresholds, near-miss reporting, anonymity provisions) onto AI does not supply explicit adaptations for AI-specific properties such as model opacity and rapid capability change. The text acknowledges these differences at a high level but leaves unaddressed whether opacity would undermine reporting incentives or whether post-reporting learning loops would function similarly; without such adaptations or a clear invariance argument, the central claim that the extracted considerations inform AI system design rests on an unvalidated transfer assumption.
Authors: We appreciate the referee pointing out this gap in our analysis. The manuscript does acknowledge differences at a high level in the introduction and when discussing the case studies, but we agree that explicit adaptations for AI-specific properties like model opacity and rapid capability change are needed to better support the claim. In the revised version, we will add a new subsection in Section 4 titled 'Adapting Considerations for AI-Specific Challenges.' This subsection will explicitly discuss how model opacity could potentially undermine reporting incentives by complicating incident identification and attribution, and propose mitigations such as requiring reports on observable behaviors or using third-party audits. Regarding post-reporting learning loops, we will argue that while rapid capability change may shorten the relevance of learned lessons, the principle of safety learning remains invariant and can be supported by mechanisms like continuous monitoring and updating of reporting thresholds. We will also clarify that the extracted considerations are intended as a foundation for context-specific adaptation rather than a direct mapping, thereby addressing the transfer assumption. revision: yes
Circularity Check
No circularity; framework and considerations synthesized from external literature and non-AI case studies
full rationale
The paper's central derivation consists of a literature review to identify seven design dimensions followed by extraction of considerations from nine external case studies in safety-critical industries. No equations, fitted parameters, self-definitional loops, or load-bearing self-citations are present. The framework is constructed from independent external sources rather than reducing to the paper's own inputs by construction. This is the most common honest finding for conceptual literature-review papers that remain self-contained against external benchmarks.
Axiom & Free-Parameter Ledger
axioms (1)
- domain assumption Design lessons from incident reporting systems in aviation, nuclear, and healthcare industries transfer to general-purpose AI
Lean theorems connected to this paper
-
IndisputableMonolith/Foundation/RealityFromDistinction.leanreality_from_one_distinction unclear?
unclearRelation between the paper passage and the cited Recognition theorem.
Through a literature review, we develop a framework for understanding the institutional design of AI incident reporting systems, which includes seven dimensions: policy goal, actors submitting and receiving reports, type of incidents reported, level of risk materialization, enforcement of reporting, anonymity of reporters, and post-reporting actions.
What do these tags mean?
- matches
- The paper's claim is directly supported by a theorem in the formal canon.
- supports
- The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
- extends
- The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
- uses
- The paper appears to rely on the theorem as machinery.
- contradicts
- The paper's claim conflicts with a theorem or certificate in the canon.
- unclear
- Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.
Reference graph
Works this paper leans on
-
[1]
Identifying and Mitigating the Security Risks of Gen- erative AI. arXiv:2308.14840. Beers, K. 2025. Beyond Corporate Promises: How Gov- ernment Can Follow Through on AI Preparedness. https:// perma.cc/B46A-RM4L. Bengio, Y .; Fox, B.; Carlos Ponce de Leon Ferreira de Car- valho, A.; Nemer, M.; Rivera, R. P .; Zeng, Y .; Heikkil¨ a, J.; Avrin, G.; Kr¨ uger,...
-
[2]
Senate Committee on the Judiciary, Subcommit- tee on Privacy, Technology, and the Law)
Oversight of A.I.: Principles for Regulation (Hearin g of the U.S. Senate Committee on the Judiciary, Subcommit- tee on Privacy, Technology, and the Law). https://perma.cc / PY5S-ZUB6. Boine, C.; and Rolnick, D. 2023. General Purpose AI Sys- tems in the AI Act: trying to fit a square peg into a round hole. In Proceedings of the 2023 W e Robot Conference . ...
-
[3]
Defending Against Unforeseen Failure Modes with Latent Adversarial Training. arXiv:2403.05030. Cattell, S.; Ghosh, A.; and Kaffee, L.-A. 2024. Coordi- nated Disclosure for AI: Beyond Security Vulnerabilities. arXiv:2402.07039. CDPH. 2021. DPH-11-023 Adverse Events Reporting. Cali- fornia Department of Public Health (CDPH) . https://perma. cc/KF4N-KDRE. Ac...
-
[4]
In 2023 ACM Conference on Fairness, Accountability, and Transparency, 651–666
Harms from Increasingly Agentic Algorithmic Sys- tems. In 2023 ACM Conference on Fairness, Accountability, and Transparency, 651–666. Chen, H.; and Magramo, K. 2024. Finance worker pays out $25 million after video call with deepfake ‘chief financial officer’. CNN. https://perma.cc/KF4N-KDRE. Accessed: 2024-02-06. Cheng, L.; Sun, N.; Li, Y .; Zhang, Z.; Wang...
-
[5]
Regulation (EU) 2024/1689 of the European Par- liament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence and amending Regulations (EC) No 300/2008, (EU) No 167/2013, (EU) No 168/2013, (EU) 2018/858, (EU) 2018/1139 and (EU) 2019/2144 and Directives 2014/90/EU, (EU) 2016/797 and (EU) 2020/1828 (Artificial Intelligence ...
work page 2024
-
[6]
United States Federal Aviation Administration (F AA) . https://perma.cc/PJT3-UEDM. Accessed: 2024-05-11. FAA. 2020a. AC 120-66C - Aviation Safety Action Pro- gram. United States Federal Aviation Administration (F AA). https://perma.cc/2PQS-8SMA. Accessed: 2024-07-12. FAA. 2020b. Aviation Safety Action Program - MOU. https://perma.cc/9CWS-AABK. Accessed: 2...
work page 2024
-
[7]
United States Federal Aviation Administration (F AA). https://perma.cc/W8Y5-7TLG. Accessed: 2024-07-12. FAA. 2024a. Aeronautical Information Manual: Offi- cial Guide to Basic Flight Information and A TC Proce- dures. United States Federal Aviation Administration (F AA). https://perma.cc/UXT8-FAN7. Accessed: 2024-04-28. FAA. 2024b. Aviation Safety Action Pr...
work page 2024
-
[8]
https://www.fda.gov/safety/medwatch-forms-fda- safety-reporting/instructions-completing-form-fda-3 500. Accessed: 2024-07-13. FDA. 2022d. MedWatch Consumer V oluntary Reporting (Form FDA 3500B). United States F ood and Drug Ad- ministration (FDA). https://www.fda.gov/safety/medwatch- fda-safety-information-and-adverse-event-reporting- program/medical-prod...
-
[9]
BMJ Quality & Safety , 19(2): 122–127
Impact of system-level activities and reporting desi gn on the number of incident reports for patient safety. BMJ Quality & Safety , 19(2): 122–127. G7. 2023a. Hiroshima Process International Code of Con- duct for Advanced AI Systems. G7. https://perma.cc/ V6WK-S6HB. Accessed: 2024-05-12. G7. 2023b. Hiroshima Process International Guiding Princi - ples fo...
-
[10]
A Right to Warn about Advanced Artificial Intelli- gence (Open Letter). https://perma.cc/6M2V -PPTD. Hoffman, M.; and Frase, H. 2023. Adding Structure to AI Harm. Technical report, Center for Security and Emerging Technology. Hopkins, A.; Cen, S. H.; Ilyas, A.; Struckman, I.; Vide- garay, L.; and Madry, A. 2025. AI Supply Chains: An Emerging Ecosystem of A...
-
[11]
A8195 Advanced Artificial Intelligence Licensing Act. https://perma.cc/6FHE-9D56. IAEA. 2006. IAEA/NEA Fuel Incident Notification and Analysis System (FINAS) Guidelines. Technical Report 14, International Atomic Energy Agency (IAEA), Vienna. IAEA. 2011. Guide on Incident Reporting System for Research Reactors. International Atomic Energy Agency (IAEA). http...
-
[12]
Medical Journal of Australia, 181(1): 36–39
Attitudes of doctors and nurses towards incident re- porting: a qualitative analysis. Medical Journal of Australia, 181(1): 36–39. Klijn, E.-H.; and Koppenjan, J. F. M. 2006. Institutional design: Changing institutional features of networks. Public Management Review, 8(1): 141–160. Knight, J. C. 2002. Safety critical systems: challenges and directions. In...
-
[13]
Stocktaking for the development of an AI incident definition. Technical Report 4, Organisation for Economic Co-operation and Development (OECD), Paris. Petkovic, D. 2023. It is Not “Accuracy vs. Explainabil- ity”—We Need Both for Trustworthy AI Systems. IEEE Transactions on T echnology and Society, 4(1): 46–53. Petschnig, W .; and Haslinger-Baumann, E. 201...
-
[14]
Technical report, National Security Commission on Artificial Intelligence
Final Report. Technical report, National Security Commission on Artificial Intelligence. Schuett, J. 2022. Defining the scope of AI regulations. arXiv:1909.01095. Schuett, J. 2023. Risk management in the Artificial Intelli- gence Act. European Journal of Risk Regulation , 1–19. Schuett, J.; Anderljung, M.; Carlier, A.; Koessler, L.; and Garfinkel, B. 2024. Fr...
-
[15]
Model evaluation for extreme risks
Driving U.S. Innovation in Artificial Intelligence: A Roadmap for Artificial Intelligence Policy in the United States Senate. Technical report, The Bipartisan Senate AI Working Group. Schwarcz, D.; Wolff, J.; and Woods, D. W . 2023. How Priv- ilege Undermines Cybersecurity. Harvard Journal of Law & T echnology, 36(2). Schwartz, R.; V assilev, A.; Greene, K....
-
[16]
Open Journal of Nursing, 5(11): 1042–1052
The Culture of Incident Reporting and Feedback: A Cross-Sectional Study in a Hospital Setting. Open Journal of Nursing, 5(11): 1042–1052. Vredenburgh, A. G. 2002. Organizational safety: Which management practices are most effective in reducing em- ployee injury rates? Journal of Safety Research, 33(2): 259– 276. Walker, C. P .; Schiff, D. S.; and Schiff, ...
work page 2002
-
[17]
https://perma.cc/7F7W-FU8K. Weatherbed, J. 2024. Trolls have flooded X with graphic Taylor Swift AI fakes. The V erge. https://perma.cc/HK8D- N4CH. Accessed: 2024-02-06. Webster, C. S. 2016. Safety in unpredictable complex sys- tems – a framework for the analysis of safety derived from the nuclear power industry. Prometheus, 34(2): 115–132. Wei, K.; Ezell,...
-
[18]
Computer Law & Security Review, 41:105567
Sociotechnical Safety Evaluation of Generative AI Systems. arXiv:2310.11986. Weidinger, L.; Uesato, J.; Rauh, M.; Griffin, C.; Huang, P .- S.; Mellor, J.; Glaese, A.; Cheng, M.; Balle, B.; Kasirzadeh , A.; Biles, C.; Brown, S.; Kenton, Z.; Hawkins, W .; Steple- ton, T.; Birhane, A.; Hendricks, L. A.; Rimell, L.; Isaac, W .; Haas, J.; Legassick, S.; Irving,...
-
[19]
Identification of case studies (Appendix B.1)
-
[20]
Development of a framework for the institutional design of incident reporting systems Appendix B.2); and
-
[21]
Application of that framework to our case studies to gen- erate insights for AI incident reporting Appendix B.3). B.1 Identification of Case Studies First, we identify case studies of safety-critical industr ies with existing and robust incident reporting frameworks. A case study approach is appropriate because incident re- porting systems of various forms...
work page 2023
-
[22]
as well as allow for cross-case comparisons (Sadovnik 2007). In particular, content analysis is particularly sui ted to investigating “[i]nstitutional realities” (Krippendorf f 2019) and for complementing other methods (here, typologies) to iterate on the scope of data collection (Bowen 2009). The first step in building our typologies was to examine the do...
work page 2007
-
[23]
Hazards vs. Situations: this distinction has been recog- nized in traditional safety science (ISO and IEC 2014) but has been under-theorized in AI governance. In the 15Incident response consists of the actions taken by a system operator to mitigate the effects or harms of an incident (NIST n.d.a; UK HPA 2009). 16Note that incident response occurs after an...
work page 2014
-
[24]
Issues vs. Incidents: We follow Hoffman and Frase (2023) in referring to hazards and situations as issues,20 and to near misses and harm events as incidents. The dis- tinction is that issues are conditions in systems or envi- ronments that are prerequisites to harm occurring, while incidents are events in which harm could have or did oc- cur. Note that re...
work page 2023
-
[25]
that harms people, animals and/or the environment, regar dless of accidental or deliberate cause, and/or
-
[26]
for which response needs have the potential to overwhelm s tate and local resources (both governmental and private sector), an d/or
-
[27]
for which the Environmental Protection Agency (EPA) and/ or the United States Coast Guard (USCG), co-Chairs of the National Re- sponse Team (NRT) for Oil and Chemical Spills under the Natio nal Oil and Hazardous Substances Pollution Contingency Plan (N CP), deems that support is or will be required.” (FEMA 2022, p. 2) Organisation for Economic Co-operatio...
work page 2022
-
[28]
The communication will be made within a reasonable time- frame, as defined by the competent authority
-
[29]
The competent authority will assess the incident’s sever ity and may order the agent, whenever necessary, to take steps and measures to reverse or mitigate the effects of the inci- dent.” (Pacheco 2023, Art. 31) Table 22: Policy Initiatives for General-Purpose AI Incide nt Reporting, Proposed Jurisdiction Initiative Type T ext Canada Artificial In- tellige...
work page 2023
-
[30]
Based on existing best practices and adequate review of th e benefits and risks of disclosing certain information, define the category of misuse events to report
-
[31]
Collate verified reports of misuse in a commonly used and machine-readable format
-
[32]
The following documentation can help provide transparency about how Practice 7.3 is implemented: a
Share verified reports of misuse with relevant third parti es, such as AI incident databases. The following documentation can help provide transparency about how Practice 7.3 is implemented: a. A description of the incident reporting process, includi ng an example of a type of incident that might be reported and to whom it would be reported.” (U.S. AI Safe...
work page 2024
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.