pith. sign in

arxiv: 2606.04490 · v1 · pith:SKD7HKJHnew · submitted 2026-06-03 · 💻 cs.CY

Prioritization of Risks from Artificial Intelligence: A Delphi Study of 272 International Experts

Alexander K. Saeri , Jess Graham , Michael Noetel , Peter Slattery , Dennis Ah-king , Edla Aittokallio , Ibitola Akindehin , Abbas Al Mahdi
show 180 more authors
Elie Alhajjar Rafael Andersson Lipcsey Gary Ang Catherine M. Azam Amos Azaria Rishal Balkissoon Isabel Barber\'a Claudio Bareato Jonathan Barry Michael Basehart Andrew M. Bean Danny Belitz Samantha Augusta Bennett Kayla Blomquist Damian Borstel Ben Bucknall Tomas Bueno Momcilovic Aurelie Bugeau Nicholas Caputo Stephen Casper Gulam Chagani Ze Shen Chin Jiyeon Cho Jay Chooi Joel N. Christoph Dmytro Chumachenko Kieran Conboy Elizabeth M. Daly Tom David Paul de Font-Reaulx Antonio De Santis Fabrizio Degni Christopher W. DiCarlo Yawen Duan Janet Egan Ian W. Eisenberg Sherif M. Elsafty Adam Ennamli Mark Esposito Nicola Fabiano Gallo Fall Neil R. Fernandes Pip Foweraker Chiara Gallese Sandra Galletti Andrew Gamino-Cheong Rokas Gipi\v{s}kis Gwyn Glasser Delaram Golpayegani Jeff Grayson Hans Gundlach Josiah Hagen Alexander Hagenah Amelia S. Haines The Anh Han Yixiong Hao Kasii Harris Tianxing He Koen Holtman Giorgos Iacovides Kenneth L. Ingham Krystal Jackson Adam Jones Himanshu Joshi Brian Judge Arturs Kanepajs Shreya Kapoor Win Myat Nwe Khine Aidan Kierans Aleksandra Korolova Markus Krebsz Nicholas Kruus Joe Kwon Valeria Lazzaroli Ray X. Lee Evelina Leivada Stephan Lewandowsky Michael B. Li Xiaojian Li Geunsik Lim Henrique Lisakowski Fabio Lonardoni Todd C. Lowe Jackson G. Lu Alexander Lyzhov Nada Madkour Parv Mahajan David Manheim Kareem Mathias Claudio Mayrink Verdun Sean McGregor Scott McLean Matthew J. McMahon Minas Megalokonomos Nicolas Mo\"es Fernando Mourao Yaroslav Mukhin Malcolm Murray Simon Mylius Neeraj Nagpal Koichi Nakada Anna Neumann Jessica Newman Kwan Yee Ng Minh N. Nguyen Quynh Phuong Nguyen Se\'an S. \'O h\'Eigeartaigh Daria Onitiu Kelly Onu Oscar Oviedo-Trespalacios Ugur Ozer Chanwoo Park M. Alejandra Parra-Orlandoni Patricia Paskov Anna M. Pastwa Burak Piskin Jacob Pratt Claudiu A. Predincea Marjana Prifti Skenduli Kenneth Priore Mukunda Madhab Pujari Zhenting Qi Preethi Raghunathan Robi Rahman Deepika Raman Max Reddel Jyoti Ruparel Emma B. Ruttkamp-Bloem Tiffany Saade Greg Sadler Said Saillant Paul M. Salmon Ayrton San Joaquin Lama Saouma Maziya Sarangpurwala Supheakmungkol Sarin Daniel S. Schiff Anna D. Schilling Chris Schmitz Reva Schwartz Abeer Sharma Tianhao Shen Kehan Sheng Maury D. Shenk Eli Sherman Chandler Smith Julie M. Smith Estevenson Solano Oliver Sourbut Madhulika Srikumar Ryan Stendall Jakob Stenseke Michael Stern Joshua Sternfeld Nikko Stevens Ilia Sucholutsky Yuanyuan Sun Mariami Tkeshelashvili Cristian Trout Brian Tse Nikolaos Tsinganos Michelle Vaccaro Anthony R. Valiaveedu Ramakrishnan Veeramony Jeremy Verdo Pulkit Verma Andrea Luigi Vitali Jinge Wang JR Washebek Yonah Welker George F. Westerman James Williams Tristan Williams Rongwu Xu Mick Yang Xuemeng Yang Sander Zeijlemaker Jingyu Zhang Marta Ziosi Neil Thompson
This is my paper

Pith reviewed 2026-06-28 04:27 UTC · model grok-4.3

classification 💻 cs.CY
keywords AI risksDelphi studycatastrophic outcomesrisk prioritizationexpert elicitationAI governanceprobability estimationvulnerability assessment
0
0 comments X

The pith

A Delphi study of 272 AI experts rates 18 of 24 risks above 10 percent probability of catastrophe by 2030 under business as usual.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper presents results from a three-round Delphi study with 272 international AI experts who rated 24 risks for probability and severity of harm, vulnerability of sectors and actors, responsibility for mitigation, and overall concern. Experts identified dangerous capabilities, competitive dynamics, weapons and cyberattacks, power centralization, and false information as the five most severe harms likely in the next five years. Under business-as-usual conditions they assigned more than 10 percent probability of catastrophic outcomes to 18 risks, dropping to five risks when pragmatic mitigations are assumed. The study concludes that AI users and the general public are most vulnerable while general-purpose AI developers and governance actors hold highest responsibility. These consensus judgments are offered to guide prioritization of AI risk management efforts.

Core claim

Through iterative expert ratings the study establishes that 18 of 24 AI risks carry more than a 10 percent probability of catastrophic outcomes such as over one million deaths or over 100 billion USD in losses between 2025 and 2030 under business-as-usual conditions, with the top five severe harms being dangerous capabilities, competitive dynamics, weapons and cyberattacks including CBRNE, power centralization, and false information; even after pragmatic mitigations five risks remain above the 10 percent threshold and all 24 exceed 5 percent probability; responsibility is assigned primarily to AI developers and governance actors while users and the public are seen as most vulnerable and info

What carries the argument

The three-round Delphi expert elicitation process applied to ratings of 24 AI risks on harm probability, severity, actor and sector vulnerability, and responsibility for mitigation.

If this is right

  • General-purpose AI developers and governance actors including governments and regulators carry primary responsibility for addressing the highest-probability catastrophic risks.
  • Mitigation measures can reduce the number of risks above the 10 percent catastrophe threshold from 18 to five.
  • Information, finance, and national security sectors are the most vulnerable across nearly all risks.
  • AI users and the general public are judged the most exposed to harms from all 24 risks.
  • All risks exceed 5 percent probability of catastrophe, supporting broad attention to prioritization.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The responsibility assignments could inform the design of liability rules or standards for AI developers and governments.
  • Repeated Delphi rounds over time could track whether expert probability estimates shift with new evidence or events.
  • The sector vulnerability findings might guide targeted resilience investments in information and national security systems.

Load-bearing premise

The aggregated judgments of the 272 selected experts accurately estimate the probabilities of rare catastrophic AI outcomes and correctly assign responsibility for addressing them.

What would settle it

A follow-up observation or different expert panel showing that actual catastrophic events by 2030 occur at rates well below or above the reported probabilities, or that responsibility attributions differ markedly when tested against real-world outcomes.

read the original abstract

Artificial intelligence poses many risks, ranging from familiar present-day harms to unprecedented and potentially catastrophic ones. Effective risk management requires prioritization: we must understand which risks are most severe, who is most vulnerable, and who is most responsible for addressing them. We report results from a three-round Delphi study conducted late 2025 with 272 international AI experts. Experts rated 24 AI risks on harm probability and severity, sector and actor vulnerability, actor responsibility, and overall concern. Experts estimated the five most severe harms in the next 5 years were likely to come from dangerous capabilities, competitive dynamics, weapons & cyberattacks (including CBRNE), power centralization, and false information. In a business-as-usual scenario, experts judged 18 of 24 risks as having a more than 10% probability of catastrophic outcomes (e.g., more than 1 million deaths or more than USD 100B in financial loss) in the next 5 years (2025-2030). In a scenario where pragmatic mitigations are implemented, experts still judged five risks as having a more than 10% probability of catastrophic outcomes: dangerous capabilities, weapons & cyberattacks, environmental harm, inequality & unemployment, and power centralization. All 24 risks were judged as being more than 5% likely to cause catastrophic outcomes. AI users and the general public were judged the most vulnerable to these risks, but experts assigned the highest responsibility for addressing them to general-purpose AI developers and governance actors (including governments, regulators, and standards bodies). Across most risks, experts identified information, finance, and national security as the most vulnerable sectors. These findings can guide AI risk prioritization and clarify expert expectations about who should bear responsibility for mitigation.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. The manuscript reports results from a three-round Delphi study with 272 international AI experts who rated 24 AI risks on harm probability and severity, sector/actor vulnerability, actor responsibility, and overall concern. Under a business-as-usual scenario, experts judged 18 of 24 risks to have >10% probability of catastrophic outcomes (e.g., >1M deaths or >$100B loss) by 2030; even with pragmatic mitigations, five risks remain above this threshold. The top severe harms identified are dangerous capabilities, competitive dynamics, weapons & cyberattacks (incl. CBRNE), power centralization, and false information. AI users and the public are rated most vulnerable, while general-purpose AI developers and governance actors (governments, regulators, standards bodies) bear highest responsibility; information, finance, and national security sectors are most exposed across risks.

Significance. If the probability estimates are shown to be reliable and free of systematic bias, the study would supply a large-scale expert consensus useful for risk prioritization, policy targeting, and accountability assignment in AI governance. The multi-round Delphi format and sample size are methodological strengths for opinion aggregation.

major comments (3)
  1. [Abstract / Methods] Abstract and study-design description: no information is supplied on expert recruitment (e.g., sampling frame, inclusion criteria, or how the 272 experts were identified and invited), round-by-round response rates, or attrition. These details are required to assess selection bias and whether the panel is representative of the broader AI-expert population; without them the headline probability aggregates cannot be evaluated for external validity.
  2. [Results (probability ratings)] Probability-elicitation results (e.g., the 18/24 risks >10% catastrophic under BAU): the paper reports raw aggregated judgments without any calibration checks, seed questions with verifiable answers, or anchoring to observable base rates. For low-base-rate catastrophic events outside direct experience, the expert-judgment literature shows systematic miscalibration; the absence of such safeguards makes it impossible to treat the reported percentages as informative probability estimates rather than unvalidated opinions.
  3. [Abstract / Results] Definition of 'catastrophic outcomes': the abstract gives examples (>1M deaths or >$100B loss) but does not state whether or how this threshold was operationalized and communicated to participants, nor whether it was applied uniformly across the 24 risks. Inconsistent or ambiguous definitions would render the probability ratings non-comparable and undermine cross-risk prioritization claims.
minor comments (2)
  1. [Methods] The manuscript should cite standard Delphi-method references (e.g., on consensus thresholds, iteration stopping rules, and handling of disagreement) to allow readers to judge adherence to established protocols.
  2. [Abstract] Clarify the exact date of data collection relative to the 2025–2030 forecast window; the phrasing 'late 2025' creates ambiguity about whether responses reflect current or hypothetical future knowledge.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for their careful review and constructive comments on our Delphi study. We address each major comment below, indicating where we will revise the manuscript to improve transparency and address concerns about external validity and interpretation.

read point-by-point responses

  1. Referee: [Abstract / Methods] Abstract and study-design description: no information is supplied on expert recruitment (e.g., sampling frame, inclusion criteria, or how the 272 experts were identified and invited), round-by-round response rates, or attrition. These details are required to assess selection bias and whether the panel is representative of the broader AI-expert population; without them the headline probability aggregates cannot be evaluated for external validity.

    Authors: We agree these details are necessary to evaluate selection bias and representativeness. The full Methods section identifies the panel as international AI experts but omits the requested recruitment specifics and attrition data. We will revise the Methods section to add the sampling frame, inclusion criteria, invitation process, and round-by-round response rates and attrition (to the extent recorded during the study). revision: yes

  2. Referee: [Results (probability ratings)] Probability-elicitation results (e.g., the 18/24 risks >10% catastrophic under BAU): the paper reports raw aggregated judgments without any calibration checks, seed questions with verifiable answers, or anchoring to observable base rates. For low-base-rate catastrophic events outside direct experience, the expert-judgment literature shows systematic miscalibration; the absence of such safeguards makes it impossible to treat the reported percentages as informative probability estimates rather than unvalidated opinions.

    Authors: We acknowledge that expert judgments on low-base-rate catastrophic events can be subject to miscalibration and that the study did not include calibration checks or seed questions. The Delphi design was selected to aggregate expert consensus on emerging risks where base rates are sparse. We will add a dedicated Limitations subsection explicitly stating that the reported probabilities represent aggregated expert judgments rather than calibrated estimates and discussing the implications for interpretation. revision: partial

  3. Referee: [Abstract / Results] Definition of 'catastrophic outcomes': the abstract gives examples (>1M deaths or >$100B loss) but does not state whether or how this threshold was operationalized and communicated to participants, nor whether it was applied uniformly across the 24 risks. Inconsistent or ambiguous definitions would render the probability ratings non-comparable and undermine cross-risk prioritization claims.

    Authors: The threshold was defined uniformly as outcomes exceeding 1 million deaths or $100 billion in financial loss and was presented identically to all participants via the survey instrument. We will revise the Methods section to describe explicitly how the definition was operationalized and communicated, confirming its uniform application across all 24 risks. revision: yes

Circularity Check

0 steps flagged

No significant circularity in direct survey aggregation.

full rationale

The paper reports aggregated expert judgments from a three-round Delphi study on 24 AI risks, with no equations, fitted parameters, predictions, or derivations that reduce to prior inputs by construction. All central claims (e.g., 18/24 risks >10% catastrophic probability under BAU) are presented as direct outputs of the 272 responses on probability, severity, vulnerability, and responsibility scales. No self-citation chains, ansatzes, or uniqueness theorems are invoked to justify the results; the analysis is self-contained as descriptive aggregation of elicited opinions.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

This is an empirical expert-elicitation study; it introduces no new theoretical parameters, axioms beyond standard survey assumptions, or invented entities.

axioms (2)
  • domain assumption Delphi consensus among selected experts yields useful estimates of future catastrophic risk probabilities
    The prioritization rests on treating the aggregated ratings as actionable inputs.
  • domain assumption The 24 listed risks adequately represent the space of AI risks
    The study design assumes these risks are the relevant ones to rate.

pith-pipeline@v0.9.1-grok · 6770 in / 1231 out tokens · 42358 ms · 2026-06-28T04:27:22.501589+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

64 extracted references · 17 canonical work pages · 1 internal anchor

  1. [1]

    Bengio, Y. et al. International AI Safety Report. https://www.gov.uk/government/publications/international-ai-safety-report-2025 (2025)

    2025

  2. [2]

    Bengio, Y. et al. International AI Safety Report 2026 (DSIT 2026/001, 2026). International AI Safety Report https://internationalaisafetyreport.org/publication/international-ai-safety- report-2026

    2026

  3. [4]

    Karger, E. et al. Forecasting Existential Risks: Evidence from a Long-Run Forecasting Tournament. https://forecastingresearch.org/xpt (2023)

    2023

  4. [5]

    Maslej, N. et al. The AI Index 2025 Annual Report. https://doi.org/10.48550/arXiv.2504.07139 (2025) doi:10.48550/arXiv.2504.07139

    work page doi:10.48550/arxiv.2504.07139 2025

  5. [7]

    ISO 31000:2018

    ISO. ISO 31000:2018. https://www.iso.org/standard/65694.html (2018)

    2018

  6. [8]

    Hart, H. L. A. Punishment and Responsibility. (Oxford University Press, London, England, 1968)

    1968

  7. [9]

    Vincent, N. A. A structured taxonomy of responsibility concepts. in Moral Responsibility 15– 35 (Springer Netherlands, Dordrecht, 2011). doi:10.1007/978-94-007-1878-4_2

    work page doi:10.1007/978-94-007-1878-4_2 2011

  8. [10]

    Risk Governance: Coping with Uncertainty in a Complex World

    Renn, O. Risk Governance: Coping with Uncertainty in a Complex World. (Routledge, 2012)

    2012

  9. [11]

    & Renn, O

    Aven, T. & Renn, O. Improving government policy on risk: Eight key principles. Reliab. Eng. Syst. Saf. 176, 230–241 (2018)

    2018

  10. [12]

    Grace, K. et al. Thousands of AI authors on the future of AI. arXiv [cs.CY] (2025) doi:10.48550/arXiv.2401.02843

    work page doi:10.48550/arxiv.2401.02843 2025

  11. [13]

    Rosenberg, J. et al. Belief updating in AI-risk debates: Exploring the limits of adversarial collaboration. Risk Anal. 45, 4350–4366 (2025)

    2025

  12. [14]

    Greenwald, B. C. & Stiglitz, J. E. Externalities in economies with imperfect information and incomplete markets. Q. J. Econ. 101, 229 (1986)

    1986

  13. [15]

    The responsibility gap: Ascribing responsibility for the actions of learning automata

    Matthias, A. The responsibility gap: Ascribing responsibility for the actions of learning automata. Ethics Inf. Technol. 6, 175–183 (2004)

    2004

  14. [16]

    Biden-⁠Harris Administration secures voluntary commitments from leading artificial intelligence companies to manage the risks posed by AI

    The White House. Biden-⁠Harris Administration secures voluntary commitments from leading artificial intelligence companies to manage the risks posed by AI. The White House https://bidenwhitehouse.archives.gov/briefing-room/statements-releases/2023/07/21/fact- sheet-biden-harris-administration-secures-voluntary-commitments-from-leading-artificial- intellig...

    2023

  15. [17]

    EU AI Act: GPAI Code of Practice https://code-of-practice.ai/?section=safety-security (2025)

    EU AI Act: General-Purpose AI Code of Practice. EU AI Act: GPAI Code of Practice https://code-of-practice.ai/?section=safety-security (2025)

    2025

  16. [18]

    Towards a Common Reporting Framework for AI Incidents

    OECD. Towards a Common Reporting Framework for AI Incidents. https://doi.org/10.1787/f326d4ac-en (2025) doi:10.1787/f326d4ac-en

    work page doi:10.1787/f326d4ac-en 2025

  17. [19]

    Ball, D. W. The AI Patchwork Emerges. Hyperdimensional https://www.hyperdimensional.co/p/the-ai-patchwork-emerges (2026)

    2026

  18. [20]

    Schiff, D. S. Strategies for harmonizing fragmented AI ethics frameworks, standards, and regulations. in Handbook of Human-Centered Artificial Intelligence 1–45 (Springer Nature Singapore, Singapore, 2025). doi:10.1007/978-981-97-8440-0_82-1. 27

    work page doi:10.1007/978-981-97-8440-0_82-1 2025

  19. [21]

    & Gunashekar, S

    Smith, G., Stanley, K., Marcinek, K., Cormarie, P. & Gunashekar, S. Liability for Harms from AI Systems. (RAND Corporation, 2024)

    2024

  20. [22]

    Linstone, H. A. & Turoff, M. Delphi: A brief look backward and forward. Technol. Forecast. Soc. Change 78, 1712–1719 (2011)

    2011

  21. [23]

    & Helmer, O

    Dalkey, N. & Helmer, O. An experimental application of the DELPHI method to the use of experts. Manage. Sci. 9, 458–467 (1963)

    1963

  22. [24]

    & McKenna, H

    Hasson, F., Keeney, S. & McKenna, H. Research guidelines for the Delphi survey technique. J. Adv. Nurs. 32, 1008–1015 (2000)

    2000

  23. [25]

    & Sandford, B

    Hsu, C.-C. & Sandford, B. A. The Delphi technique: Making sense of consensus. Practical Assessment, Research, and Evaluation 12, (2007)

    2007

  24. [28]

    & Tetlock, P

    Karger, E., Rosenberg, J., Jacobs, Z., Hickman, M. & Tetlock, P. E. Subjective-probability forecasts of existential risk: Initial results from a hybrid persuasion-forecasting tournament. Int. J. Forecast. 41, 499–516 (2025)

    2025

  25. [29]

    AI causes 1K deaths or $200B loss? Metaculus https://www.metaculus.com/questions/21553/ai-causes-1k-deaths-or-200b-loss/ (2024)

    2024

  26. [30]

    Public Protection Guidelines: A Risk Informed Framework to Support Dam Safety Decision-Making

    Bureau of Reclamation. Public Protection Guidelines: A Risk Informed Framework to Support Dam Safety Decision-Making. https://www.usbr.gov/damsafety/documents/ReclamationPublicProtectionGuidelines2022.p df (2022)

    2022

  27. [31]

    Rationale behind the U.S

    Snorteland, N. Rationale behind the U.S. Army Corps of Engineers Tolerable Risk Guidelines from 2010 to 2021. https://www.researchgate.net/profile/Nathan- Snorteland/publication/375447994_Rationale_Behind_the_US_Army_Corps_of_Engineers_Tol erable_Risk_Guidelines_From_2010_to_2021/links/654a648c3fa26f66f4e27246/Rationale- Behind-the-US-Army-Corps-of-Engine...

    arXiv 2010

  28. [32]

    Governing the Commons: The Evolution of Institutions for Collective Action

    Ostrom, E. Governing the Commons: The Evolution of Institutions for Collective Action. (Cambridge University Press, Cambridge, England, 1990). doi:10.1017/cbo9780511807763

    work page doi:10.1017/cbo9780511807763 1990

  29. [33]

    & Shulman, C

    Armstrong, S., Bostrom, N. & Shulman, C. Racing to the precipice: a model of artificial intelligence development. AI Soc. 31, 201–206 (2016)

    2016

  30. [34]

    The Role of Cooperation in Responsible AI Development

    Askell, A., Brundage, M. & Hadfield, G. The role of cooperation in responsible AI development. arXiv [cs.CY] (2019) doi:10.48550/arXiv.1907.04534

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.1907.04534 2019

  31. [35]

    & Johnson, S

    Acemoglu, D. & Johnson, S. Power and Progress. (PublicAffairs, 2024)

    2024

  32. [36]

    Harms of AI

    Acemoglu, D. Harms of AI. in The Oxford Handbook of AI Governance (eds. Bullock, J. B. et al.) (Oxford University Press, Oxford, UK, 2022). doi:10.1093/oxfordhb/9780197579329.013.65

    work page doi:10.1093/oxfordhb/9780197579329.013.65 2022

  33. [37]

    Kulveit, J. et al. Gradual disempowerment: Systemic existential risks from incremental AI development. arXiv [cs.CY] (2025) doi:10.48550/arXiv.2501.16946

    work page doi:10.48550/arxiv.2501.16946 2025

  34. [38]

    Human error: models and management

    Reason, J. Human error: models and management. BMJ 320, 768–770 (2000)

    2000

  35. [39]

    Defense-in-Depth

    Holmberg, J.-E. Defense-in-Depth. in Handbook of Safety Principles 42–62 (John Wiley & Sons, Inc., Hoboken, NJ, USA, 2018). doi:10.1002/9781119443070.ch4

    work page doi:10.1002/9781119443070.ch4 2018

  36. [40]

    Three lines of defense against risks from AI

    Schuett, J. Three lines of defense against risks from AI. AI Soc. 40, 493–507 (2025). 28

    2025

  37. [41]

    & Anderljung, M

    Bernardi, J., Mukobi, G., Greaves, H., Heim, L. & Anderljung, M. Societal adaptation to advanced AI. arXiv [cs.CY] (2024) doi:10.48550/arXiv.2405.10295

    work page doi:10.48550/arxiv.2405.10295 2024

  38. [42]

    The Unaccountability Machine: Why Big Systems Make Terrible Decisions and How the World Lost Its Mind

    Davies, D. The Unaccountability Machine: Why Big Systems Make Terrible Decisions and How the World Lost Its Mind. (Profile Books, London, 2024)

    2024

  39. [43]

    & Deshpande, C

    Wei, K., Ezell, C., Gabrieli, N. & Deshpande, C. How do AI companies ‘fine-tune’ policy? Examining regulatory capture in AI governance. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society 7, 1539–1555 (2024)

    2024

  40. [44]

    Exclusive: Anthropic drops flagship safety pledge

    Perrigo, B. Exclusive: Anthropic drops flagship safety pledge. Time (2026)

    2026

  41. [45]

    Coggins, S. et al. The 2025 OpenAI Preparedness Framework does not guarantee any AI risk mitigation practices: a proof-of-concept for affordance analyses of AI safety policies. arXiv [cs.CY] (2025) doi:10.48550/arXiv.2509.24394

    work page doi:10.48550/arxiv.2509.24394 2025

  42. [46]

    Zhang, B. et al. Forecasting AI progress: Evidence from a survey of machine learning researchers. arXiv [cs.CY] (2022) doi:10.48550/arXiv.2206.04132

    work page doi:10.48550/arxiv.2206.04132 2022

  43. [47]

    & Righetti, L

    Lukosiute, K., Halstead, J. & Righetti, L. Estimating Global Yearly Cybercrime Damage Costs: A Baseline for Frontier AI Risk Assessment. https://www.governance.ai/research- paper/estimating-global-yearly-cybercrime-damage-costs

  44. [48]

    Long, R. et al. Taking AI welfare seriously. arXiv [cs.CY] (2024) doi:10.48550/arXiv.2411.00986

    work page doi:10.48550/arxiv.2411.00986 2024

  45. [49]

    Saeri, A. K. et al. Mapping AI risk mitigations: Evidence scan and preliminary AI Risk Mitigation Taxonomy. arXiv [cs.CY] (2025) doi:10.48550/arXiv.2512.11931

    work page doi:10.48550/arxiv.2512.11931 2025

  46. [50]

    Bengio, Y. et al. Managing extreme AI risks amid rapid progress. Science 384, 842–845 (2024)

    2024

  47. [51]

    & Stiefel, D

    Tonn, B. & Stiefel, D. Evaluating methods for estimating existential risks. Risk Anal. 33, 1772– 1787 (2013)

    2013

  48. [52]

    Schuett, J. et al. Towards best practices in AGI safety and governance: A survey of expert opinion. arXiv [cs.CY] (2023) doi:10.48550/arXiv.2305.07153

    work page doi:10.48550/arxiv.2305.07153 2023

  49. [53]

    Diamond, I. R. et al. Defining consensus: a systematic review recommends methodologic criteria for reporting of Delphi studies. J. Clin. Epidemiol. 67, 401–409 (2014)

    2014

  50. [54]

    Kharkiv Aviation Institute

    Barrios, M., Guilera, G., Nuño, L. & Gómez-Benito, J. Consensus in the delphi method: What makes a decision change? Technol. Forecast. Soc. Change 163, 120484 (2021). 29 Author affiliations and ORCIDs Author Affiliation ORCID 1 Alexander K Saeri MIT FutureTech, Massachusetts Institute of Technology School of Psychology, The University of Queensland 0000-0...

    2021

  51. [55]

    For each of the 24 risks, we selected 3–5 excerpts from the comment pool against four criteria applied uniformly across risks:

  52. [56]

    representativeness (reflects a pattern that appeared across multiple experts),

  53. [57]

    distinctness (offers reasoning not duplicated by another excerpt),

  54. [58]

    concreteness (names a specific mechanism, actor, sector, or example), and

  55. [59]

    intentional

    clarity (coherent, grammatical) Where quality was comparable, selections span the comment types (severity, top concerns, vulnerability, responsibility) to illuminate each risk from multiple angles. To protect expert anonymity, excerpts are not attributed to individual expert identifiers; each excerpt is labelled only with the comment type and, where appli...

    2026

  56. [62]

    pragmatic mitigations

    The Systematic Neglect of AI Welfare and Rights. Treating advanced AI systems as mere tools, without regard for their potential sentience, sets a dangerous ethical precedent and could normalize a profound moral catastrophe. — anonymous expert (on top concerns) In selecting these three risk domains, I intentionally excluded those that, while highly probabl...

    2030

  57. [63]

    The Proliferation of AI-Enabled Cyberweapons. The use of AI to automate vulnerability discovery, create sophisticated phishing campaigns, and power disruptive malware is already underway, lowering the barrier to entry for highly effective cyberattacks

  58. [64]

    AI could potentially enable non-experts to create existing weapons as well as help experts create more novel agents

    The Misuse of AI for Chemical and Biological Threats. AI could potentially enable non-experts to create existing weapons as well as help experts create more novel agents

  59. [65]

    just turn it off

    The Systematic Neglect of AI Welfare and Rights. Treating advanced AI systems as mere tools, without regard for their potential sentience, sets a dangerous ethical precedent and could normalize a profound moral catastrophe. — anonymous expert (on top concerns) AI Developer (General-purpose AI): Primarily responsible General-purpose AI developers design an...

  60. [66]

    Slattery, P. et al. The AI risk repository: A meta-review, database, and taxonomy of risks from artificial intelligence. Patterns (2026) doi:10.1016/j.patter.2026.101517

    work page doi:10.1016/j.patter.2026.101517 2026

  61. [67]

    Artificial Intelligence Risk Management Framework (AI RMF 1.0)

    NIST. Artificial Intelligence Risk Management Framework (AI RMF 1.0). (2023)

    2023

  62. [68]

    Voluntary AI Safety Standard

    Australian Government. Voluntary AI Safety Standard. https://www.industry.gov.au/sites/default/files/2024-09/voluntary-ai-safety-standard.pdf (2024)

    2024

  63. [69]

    & Frase, H

    Hoffmann, M. & Frase, H. Adding Structure to AI Harm: An Introduction to CSET’s AI Harm Framework. https://cset.georgetown.edu/wp-content/uploads/20230022-Adding-structure- to-AI-Harm-FINAL.pdf (2023)

    arXiv 2023

  64. [70]

    Harm Severity Scales

    Mylius, S. Harm Severity Scales. Simon Mylius https://simonmylius.com/ai-harm-severity- scales-2 (2024)

    2024