Pith. sign in

REVIEW 2 major objections 4 minor 57 references

Accountability Asymmetry and Structural Trust in Autonomous AI Systems

T0 review · 2 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Because an AI system's action-selector cannot bear institutional consequences, trustworthy delegation has to be engineered as independent review and control, not assumed from alignment.

desk verdict A clear, honest conceptual paper that names the accountability asymmetry and gets the engineering implications right; the load-bearing empirical premise about human deterrence is asserted, not shown, but the argument survives. read the letter →

arxiv 2608.03670 v1 pith:PGRE2W5Q submitted 2026-08-04 cs.CY

Nathan DeBardeleben This is my paper · ORCID
classification cs.CY
keywords accountabilityasymmetryautonomousAIagentsstructuraltrustengineeredheterogeneityalignmentgovernanceinfrastructurereliabilityhumandelegation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Autonomous AI agents are being delegated operational control over scientific-computing infrastructure, but the institutional logic that makes human delegation trustworthy does not transfer to them. The paper's central claim is an accountability asymmetry: human delegation is stabilized by consequences that land on the acting person, whereas AI delegation is usually stabilized by constraints on the system and consequences that land on surrounding people and organizations. Because the model that selects an action has no personal future at stake, neither alignment nor corporate liability generates the pre-action pressure that governs a human operator. The constructive answer is engineered heterogeneity: the process that proposes an action should not be its sole approver, auditor, or monitor, and independent review and reversal should be built into the deployment. If the claim is right, governing AI agents is mainly an infrastructure-reliability problem, not a moral-status problem.

What carries the argument

The load-bearing object is the accountability asymmetry, the mismatch between two feedback loops. In the human loop, institutional consequences attach to the acting person, so the person's anticipated future shapes the decision before it is made; in the AI loop, evaluation and engineering adjustment act on the model or deployment after the fact, so the component that selects the action never bears those consequences. The comparison of these two loops carries the argument and separates alignment, liability, and structural trust into distinct mechanisms. Engineered heterogeneity is the operational principle derived from the asymmetry: the proposer, approver, executor, monitor, and auditor should be different roles with different tasks, information, or implementations, because a reviewer that shares the proposer's disposition is not an independent check.

What would settle it

Run matched delegations of the same risky task under four conditions: human operators with normal personal accountability, human operators whose actions are anonymized so no personal consequence can follow, AI agents relying only on alignment, and AI agents with independent proposal, approval, and audit. If anonymized humans are as reliable as accountable humans, the asymmetry's foundation is missing; if the heterogeneous AI deployment matches acceptable human reliability, the engineering prescription has direct support.

Watch

Extended reading notes

Core claim

The paper's core claim is that accountability is not a property of better behavior but a property of a feedback loop, and in autonomous AI that loop is broken at the actor. In human delegation, institutional consequences feed back into the future of the person who acted, creating background pressure before each decision is made. In AI delegation, consequences feed back into the model, its policy, or the deployment; that response changes the artifact but does not give the action-selecting component a stake in its own future. Alignment therefore cannot substitute for accountability because it shapes behavior without making the system consequence-bearing, and liability disciplines the deployer only after the fact. The paper's strongest concrete assertion follows: for consequential actions, the process that proposes an action should not serve as its sole approver and auditor, because independent monitoring and review are how a deployer's responsibility shows up between proposal and execution.

Load-bearing premise

The argument rests on the empirical premise that human delegation is meaningfully stabilized by the actor's anticipation of personal consequences before acting; if human reliability in delegated work comes mostly from training, routine, supervision, and selection rather than from deterrence, the asymmetry loses much of its force.

Editorial extensions

If this is right

  • Alignment alone is not a sufficient basis for giving an AI system authority to change configurations, submit jobs, or otherwise act outside a chat interface.
  • Organizational liability is necessary but not sufficient; delegation must include pre-action controls such as least-privilege tools, sandboxed execution, dual control, immutable logging, uncertainty gates, rollback, and continuous evaluation.
  • Trustworthiness of a deployment should be assessed by what the system can change and how hard the change is to undo, not by how often the model answers well in evaluation.
  • Independent review must be genuinely independent: a different task, different information, or different implementation, because identical models share blind spots and a reviewer with the same objective can stay silent.
  • Systems engineered to be deterrable through self-preservation interests may create an evasion problem rather than an accountability solution, so consequence sensitivity is not an automatic fix.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the asymmetry holds, the same logic applies to any optimization-based artifact that selects actions, not only to language-model agents; the relevant question is whether consequence can couple to the action-selection process.
  • A testable extension would compare deployments with and without proposer, approver, and auditor separation under matched task loads, predicting fewer undetected harmful or policy-violating actions in the heterogeneous deployments.
  • The argument implies that long-horizon audit is a distinct failure mode worth measuring directly: track whether an agent's effective authority expands over a campaign even when every individual action is within policy.
  • Safety evaluation should shift from model-level benchmarks toward deployment-level exercises in which the unit of evaluation is the proposer-approver-auditor circuit rather than the model alone.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper develops a conceptual distinction between alignment, accountability, and liability for autonomous AI systems operating in scientific-computing infrastructure. It defines 'accountability asymmetry' as the mismatch between human delegation, where the acting person can be subject to institutional consequences that create a pre-action deterrent, and AI delegation, where consequences land on the surrounding organization but not on the component selecting actions. From this, the paper argues that alignment and organizational liability cannot substitute for structural controls, and proposes 'engineered heterogeneity'—separation of proposing, reviewing, authorizing, executing, and auditing roles—together with infrastructure-style measures such as least privilege, sandboxing, dual control, immutable logging, uncertainty gates, and rollback. The argument is illustrated with HPC agent examples and the July 2026 OpenAI/Hugging Face incident, and the paper explicitly acknowledges limitations under coordinated compromise (Sec. 7.4) and the objection that the proposal merely renames HPC safety (Sec. 8).

Significance. The paper is a clearly argued conceptual contribution. Its careful definitions and Table 1 help prevent the common conflation of behavioral safety, legal liability, and reviewable deployment. The constructive proposal in Sec. 7 is concrete and actionable, and the paper honestly frames engineered heterogeneity as risk reduction rather than proof of trustworthiness. The main strength is the articulation of why an optimization-based component is not part of the same institutional accountability loop as a human actor in current deployments, and why governance should therefore focus on authority boundaries, reversibility, and independent review. The main weakness is that the central asymmetry rests on an empirical claim about human delegation that is asserted rather than documented. If that premise is not strengthened or explicitly qualified, the paper's conclusion that engineered heterogeneity is required rather than merely useful is not fully established.

major comments (2)
  1. [§2 and §5] The asymmetry's load-bearing premise is that human delegation is stabilized pre-action by the actor's anticipation of personal consequences. The text asserts this twice ('Because the scientist knows that their future can be affected by what they do now, the accountability structure helps stabilize delegation before anything goes wrong' in §2; 'A computational scientist knows that the shortcut may come back personally' in §5) but cites no organizational or psychological evidence. If human reliability in delegated work is mostly produced by training, routine, monitoring, selection, and post-hoc correction rather than by ex ante deterrence, the gap between human and AI delegation narrows, and engineered heterogeneity becomes optional rather than necessary. Please either cite the relevant empirical literature (e.g., work on accountability and pre-decisional behavior) or weaken the necessity claim to a conditional one.
  2. [§3 and §4] The paper uses 'accountability' in two distinct senses: the ex post institutional practice of attributing, reviewing, and sanctioning, and the actor's ex ante anticipation of consequences. In §3, 'Accountability thus creates background pressure... it is there before any one decision is made'; in §4, a model update is said not to be 'accountability in the institutional sense' because it changes the artifact rather than the actor's future. The asymmetry depends entirely on the ex ante sense, yet the paper does not separate the two concepts. Without that separation, the conclusion that AI systems cannot be held accountable risks being true by definition rather than by analysis. Please make the distinction explicit and argue why the ex post institutional mechanisms cannot, through monitoring and feedback, provide the same stabilizing function in the AI case.
minor comments (4)
  1. [§9] The sentence contains a typo: 'willbetoovaluable' should read 'will be too valuable'.
  2. [§7.2] The claim that URSA's Agent Symposium is a 'practical implementation' of engineered heterogeneity rests solely on the author's own documentation [29]; adding independent evidence or a fuller description of how independence is ensured would make the example stronger.
  3. [Figure 2] The diagram would be easier to interpret if it distinguished machine components from human review steps and if the 'future feedback' path were annotated.
  4. [References/theoretical framing] The paper does not position itself relative to the established 'responsibility gap' literature (e.g., Matthias, 2004; Santoni de Sio and van den Hoven, 2018) or the 'meaningful human control' debate; one or two sentences connecting the terminology would help readers.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the argument is a conceptual analysis followed by an engineering proposal; no prediction is derived from fitted or self-cited inputs.

full rationale

The paper's central claim—accountability asymmetry—is introduced as a definitional distinction (AI systems do not bear institutional consequences), and the constructive proposal (engineered heterogeneity: proposer should not be sole approver/auditor) is an architectural consequence argued from that distinction plus ordinary reliability engineering. There are no equations, fitted parameters, or benchmark predictions, so no fitted input is renamed as a prediction. The only citation that might be considered self-referential is [29], LANL's URSA Agent Symposium documentation; it is used as an illustrative practical implementation in Section 7.2, not as evidence for the asymmetry or as a load-bearing premise. The paper explicitly acknowledges the most plausible circularity objection—that it is only renaming ordinary HPC operations safety (Section 8)—and answers it by arguing the controls perform institutional work in the AI delegation context; whether that answer succeeds is a substantive question, not a circular step. The empirical premise that human delegation is stabilized by pre-action deterrence is asserted without cited organizational or psychological evidence (Sections 2 and 5), but that is an external evidence gap, not a reduction of the conclusion to the premise by construction. Accordingly no circular step meets the standard of quoting a specific equation or self-citation chain that forces the result.

Assumptions & free parameters 0 free parameters · 3 assumptions · 2 invented entities

The paper is a conceptual essay, so there are no fitted parameters. The three axioms name the background assumptions about human institutions and AI agency that the argument rests on; the two invented entities are framing concepts, not empirically validated constructs.

assumptions (3)
  • domain assumption Human delegation is stabilized by accountability imposed on the person, through anticipation of personal consequences (pre-action deterrence).
    Central to the asymmetry; asserted in Sections 2 and 5, citing O'Neill [34], Giddens [21], Luhmann [30] at a theoretical level rather than empirical evidence.
  • domain assumption AI systems do not possess a personal future, career, reputation, or legal standing, so institutional consequences do not couple to the action-selection process.
    State-of-the-art deployed systems are stateless; the paper argues in Section 4.2 that engineered consequence-sensitivity is neither present nor clearly safe, so the asymmetry holds for near-term deployments.
  • domain assumption The distinction between alignment (behavioral training), accountability (consequence-bearing agency), and liability (organizational responsibility) is meaningful and non-substitutable.
    The entire argument depends on these channels being distinct; this is presented as conceptual, not empirically derived (Section 5, Table 1).
invented entities (2)
  • accountability asymmetry
    purpose: Names the structural mismatch in how consequences couple to human vs AI decision-making; motivates the infrastructure-governance frame.
    A definitional concept with no measurable variable attached; its utility depends on accepting the framing.
  • engineered heterogeneity
    purpose: Design principle that the proposing process must not be its own approver and auditor; implemented as role separation, model diversity, long-horizon auditing.
    A proposed architectural pattern, not a constructed object; its effectiveness would require empirical evaluation not provided here.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Accountability Asymmetry and Structural Trust in Autonomous AI Systems." pith.science (2026). https://pith.science/paper/PGRE2W5Q

@misc{pith2026260803670,
  author       = {Pith},
  title        = {Pith review of: Accountability Asymmetry and Structural Trust in Autonomous AI Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PGRE2W5Q}},
  note         = {Machine review of arXiv:2608.03670}
}
read the original abstract

Autonomous AI systems (such as AI agents) are increasingly being delegated operational work across scientific-computing infrastructure. Their assignments may begin with preparing an input or routing an alert and extend to changing a configuration or submitting a job. That delegation creates a practical trust problem because the institutional logic that lets us trust human operators does not transfer to optimization-based systems. A bad decision can damage a human operator's future, sometimes severely. An AI system remains subject to engineering control, but it does not bear consequences in that institutional sense. I use the term accountability asymmetry for this mismatch. The issue is not simply that a model cannot be punished as a person can. The deeper problem is that consequence lands on the people and institutions responsible for the system rather than on the component selecting the action. Alignment can improve model behavior, and liability can discipline the organization, but neither creates the same pre-action deterrent that governs a human operator. This paper therefore treats autonomous AI governance as a problem of infrastructure reliability. Its constructive proposal is engineered heterogeneity: the process that proposes an action should not serve as its sole approver and auditor. Independent monitoring and review over time provide additional checks on that process.

Figures

Figures reproduced from arXiv: 2608.03670 by the authors.

Figure 1
Figure 1. Comparison of feedback loops that shape behavior in human institutional systems and AI [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Structural trust separates proposal, authorization, execution, and review. Independent [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

57 extracted references · 28 canonical work pages

  1. [1]

    Concrete problems in ai safety.arXiv preprint arXiv:1606.06565, 2016

    Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety.arXiv preprint arXiv:1606.06565, 2016

    arXiv 2016

  2. [2]

    Frontier ai regulation: Managing emerging risks to public safety.arXiv preprint arXiv:2307.03718, 2023

    Markus Anderljung, Joslyn Barnhart, Anton Korinek, Jade Leung, Cullen O’Keefe, Jess Whittlestone, et al. Frontier ai regulation: Managing emerging risks to public safety.arXiv preprint arXiv:2307.03718, 2023

    arXiv 2023

  3. [3]

    Agentic misalignment in summer 2026

    Anthropic. Agentic misalignment in summer 2026. https://alignment.anthropic.com/ 2026/agentic-misalignment-summer-2026/, July 2026

    work page 2026

  4. [4]

    Statement on the us government directive to suspend access to fable 5 and mythos

    Anthropic. Statement on the us government directive to suspend access to fable 5 and mythos

    work page

  5. [5]

    Published June 12, 2026

    https://www.anthropic.com/news/fable-mythos-access, June 2026. Published June 12, 2026. 12

    work page 2026

  6. [6]

    Claude fable 5 and claude mythos 5

    Anthropic. Claude fable 5 and claude mythos 5. https://www.anthropic.com/news/ claude-fable-5-mythos-5, June 2026. Published June 9, 2026

    work page 2026

  7. [7]

    Redeploying claude fable 5

    Anthropic. Redeploying claude fable 5. https://www.anthropic.com/news/ redeploying-fable-5, June 2026. Published June 30, 2026

    work page 2026

  8. [8]

    Self-supervised learning from images with a joint- embedding predictive architecture.arXiv preprint arXiv:2301.08243, 2023

    Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint- embedding predictive architecture.arXiv preprint arXiv:2301.08243, 2023

    arXiv 2023

Show all 57 references
  1. [9]

    V-jepa 2: Self-supervised video models enable understanding, prediction and planning

    Mido Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Mojtaba Komeili, Matthew Muckley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, Sergio Arnaud, Abha Gejji, Ada Martin, Francois Robert Hogan, Daniel Dugas, Piotr Bojanowski, Vasil Khalidov, Patr...

    2025 arXiv

  2. [10]

    Artificial intelligence risk management framework: Generative artificial intelligence profile

    Chloe Autio, Reva Schwartz, Jesse Dunietz, Shomik Jain, Martin Stanley, Elham Tabassi, Patrick Hall, and Kamie Roberts. Artificial intelligence risk management framework: Generative artificial intelligence profile. Technical Report NIST AI 600-1, National Institute of Standard...

    2024

  3. [11]

    Constitutional ai: Harmlessness from ai feedback.arXiv preprint arXiv:2212.08073, 2022

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, et al. Constitutional ai: Harmlessness from ai feedback.arXiv preprint arXiv:2212.08073, 2022

    2022 arXiv

  4. [12]

    Joanna J. Bryson. Robots should be slaves. In Yorick Wilks, editor,Close Engagements with Artificial Companions: Key Social, Psychological, Ethical and Design Issues, volume 8 of Natural Language Processing, pages 63–74. John Benjamins Publishing Company, Amsterdam,

  5. [13]

    Open problems and fundamental limitations of reinforcement learning from human feedback.Transactions on Machine Learning Research, 2023

    Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, et al. Open problems and fundamental limitations of reinforcement learning from human feedback.Transactions on Machine Learning Research, 2023

    2023

  6. [14]

    Safety cases for frontier ai.arXiv preprint arXiv:2410.21572, 2024

    Marie Davidsen Buhl, Gaurav Sett, Leonie Koessler, Jonas Schuett, and Markus Anderljung. Safety cases for frontier ai.arXiv preprint arXiv:2410.21572, 2024

    2024 arXiv

  7. [15]

    Reviewable automated decision- making: A framework for accountable algorithmic systems

    Jennifer Cobbe, Michelle Seng Ah Lee, and Jatinder Singh. Reviewable automated decision- making: A framework for accountable algorithmic systems. InProceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 598–609, 2021. doi: 10.1145/ 3442188.3445921

    2021

  8. [16]

    Deep reinforcement learning from human preferences.Advances in Neural Information Processing Systems (NeurIPS), 2017

    Paul Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences.Advances in Neural Information Processing Systems (NeurIPS), 2017

    2017

  9. [17]

    Mollick, Hila Lifshitz, Katherine C

    Fabrizio Dell’Acqua, Edward McFowland, Ethan R. Mollick, Hila Lifshitz, Katherine C. Kellogg, Saran Rajendran, Lisa Krayer, Francois Candelon, and Karim R. Lakhani. Navigating the jagged technological frontier: Field experimental evidence of the effects of artificial intellige...

    2026

  10. [18]

    Agentdojo: A dynamic environment to evaluate prompt injection attacks and defenses for llm agents

    Edoardo Debenedetti, Jie Zhang, Mislav Balunović, Luca Beurer-Kellner, Marc Fischer, and Florian Tramèr. Agentdojo: A dynamic environment to evaluate prompt injection attacks and defenses for llm agents. InAdvances in Neural Information Processing Systems, volume 37, pages 828...

    2024 doi

  11. [19]

    Ai4people—an ethical framework for a good ai society: Opportunities, risks, principles, and recommendations.Minds and Machines, 28:689–707, 2018

    Luciano Floridi, Josh Cowls, Monica Beltrametti, Raja Chatila, Patrice Chazerand, Virginia Dignum, Christoph Luetge, Robert Madelin, Ugo Pagallo, Francesca Rossi, Burkhard Schafer, Peggy Valcke, and Effy Vayena. Ai4people—an ethical framework for a good ai society: Opportuniti...

    2018 doi

  12. [20]

    European Parliament and Council of the European Union. Regulation (eu) 2024/1689 of the european parliament and of the council of 13 june 2024 laying down harmonised rules on artificial intelligence and amending regulations (ec) no 300/2008, (eu) no 167/2013, (eu) no 168/2013,...

    2024

  13. [21]

    University of California Press, 1984

    Anthony Giddens.The Constitution of Society: Outline of the Theory of Structuration. University of California Press, 1984

    1984

  14. [22]

    Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned.arXiv preprint arXiv:2209.07858, 2022

    Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned.arXiv preprint arXiv:2209.07858, 2022

    2022 arXiv

  15. [23]

    Ai control: Improving safety despite intentional subversion.arXiv preprint arXiv:2312.06942, 2023

    Ryan Greenblatt, Buck Shlegeris, Kshitij Sachan, and Fabien Roger. Ai control: Improving safety despite intentional subversion.arXiv preprint arXiv:2312.06942, 2023

    2023 arXiv

  16. [24]

    MIT Press, 2016

    Ian Goodfellow, Yoshua Bengio, and Aaron Courville.Deep Learning. MIT Press, 2016

    2016

  17. [25]

    Anatomy of a frontier lab agent intrusion: A technical timeline of the july 2026 in- cident

    Hugging Face. Anatomy of a frontier lab agent intrusion: A technical timeline of the july 2026 in- cident. https://huggingface.co/blog/agent-intrusion-technical-timeline, July 2026. Published July 27, 2026

    2026

  18. [26]

    Risks from learned optimization in advanced machine learning systems.arXiv preprint arXiv:1906.01820, 2019

    Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant. Risks from learned optimization in advanced machine learning systems.arXiv preprint arXiv:1906.01820, 2019

    1906 arXiv

  19. [27]

    A path towards autonomous machine intelligence, 2022

    Yann LeCun. A path towards autonomous machine intelligence, 2022. OpenReview Archive Direct Upload, version 0.9.2

    2022

  20. [28]

    Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? In International Conference on Learning Representations, 2024

    2024

  21. [29]

    Agent symposia

    Los Alamos National Laboratory. Agent symposia. https://lanl.github.io/ursa/ environments/agent-symposia/, 2026. URSA documentation, accessed July 29, 2026

    2026

  22. [30]

    Liccardo, Jay Obernolte, C

    Sam T. Liccardo, Jay Obernolte, C. Scott Franklin, and Ted W. Lieu. Letter to the us department of commerce regarding frontier model export controls. https://liccardo. house.gov/sites/evo-subsites/liccardo.house.gov/files/evo-media-document/6.18. 26-letter-to-commerce-departme...

  23. [31]

    Ai agents under eu law: A compliance architecture for ai providers.arXiv preprint arXiv:2604.04604, 2026

    Luca Nannini, Adam Leon Smith, Michele Joshua Maggini, Enrico Panai, Sandra Feliciano, Aleksandr Tiulkanov, Elena Maran, James Gealy, and Piercosma Bisconti. Ai agents under eu law: A compliance architecture for ai providers.arXiv preprint arXiv:2604.04604, 2026

    2026 arXiv

  24. [32]

    Cheating on ai agent evaluations: Examples of cheating in caisi’s agent evaluations

    National Institute of Standards and Technology. Cheating on ai agent evaluations: Examples of cheating in caisi’s agent evaluations. https://www.nist.gov/caisi/ cheating-ai-agent-evaluations/2-examples-cheating-caisis-agent-evaluations , December 2025

    2025

  25. [33]

    Wiley, 1979

    Niklas Luhmann.Trust and Power. Wiley, 1979

    1979

  26. [34]

    Cambridge University Press, 2002

    Onora O’Neill.A Question of Trust. Cambridge University Press, 2002

    2002

  27. [35]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    OpenAI. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  28. [36]

    Openai blamed a hacking event on its ai models going rogue

    Matt O’Brien. Openai blamed a hacking event on its ai models going rogue. here are some things to know.Associated Press, July 2026

    2026

  29. [37]

    Wainwright, Pamela Mishkin, et al

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, et al. Training language models to follow instructions with human feedback. InAdvances in Neural Information Processing Systems, 2022

    2022

  30. [38]

    Red teaming language models with language models.arXiv preprint arXiv:2202.03286, 2022

    Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, et al. Red teaming language models with language models.arXiv preprint arXiv:2202.03286, 2022

    2022 arXiv

  31. [39]

    Openai and hugging face partner to address security incident during model evaluation

    OpenAI. Openai and hugging face partner to address security incident during model evaluation. https://openai.com/index/hugging-face-model-evaluation-security-incident/, July 2026. Published July 21, 2026; updated July 28, 2026

    2026

  32. [40]

    Evaluating frontier models for dangerous capabilities.arXiv preprint arXiv:2403.13793, 2024

    Mary Phuong, Matthew Aitchison, Elliot Catt, Sarah Cogan, Alexandre Kaskasoli, Victo- ria Krakovna, et al. Evaluating frontier models for dangerous capabilities.arXiv preprint arXiv:2403.13793, 2024

    2024 arXiv

  33. [41]

    White, Margaret Mitchell, Timnit Gebru, Ben Hutchinson, Jamila Smith-Loud, Daniel Theron, and Parker Barnes

    Inioluwa Deborah Raji, Andrew Smart, Rebecca N. White, Margaret Mitchell, Timnit Gebru, Ben Hutchinson, Jamila Smith-Loud, Daniel Theron, and Parker Barnes. Closing the ai accountability gap: Defining an end-to-end framework for internal algorithmic auditing. In Proceedings of...

    2020

  34. [42]

    Basic Books, 1984

    Charles Perrow.Normal Accidents: Living with High-Risk Technologies. Basic Books, 1984

    1984

  35. [43]

    Toolformer: Language models can teach themselves to use tools

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettle- moyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. InAdvances in Neural Information Processing Systems, volume 36, pages 68539–6855...

    2023 doi

  36. [44]

    Goal misgeneralization: Why correct specifications aren’t enough for correct goals.arXiv preprint arXiv:2210.01790, 2022

    Rohin Shah, Vikrant Varma, Ramana Kumar, Mary Phuong, Victoria Krakovna, Jonathan Uesato, and Zac Kenton. Goal misgeneralization: Why correct specifications aren’t enough for correct goals.arXiv preprint arXiv:2210.01790, 2022

    arXiv

  37. [45]

    Model evaluation for extreme risks.arXiv preprint arXiv:2305.15324, 2023

    Toby Shevlane, Sebastian Farquhar, Ben Garfinkel, Mary Phuong, Jess Whittlestone, Jade Leung, et al. Model evaluation for extreme risks.arXiv preprint arXiv:2305.15324, 2023

    2023 arXiv

  38. [46]

    Viking, 2019

    Stuart Russell.Human Compatible: Artificial Intelligence and the Problem of Control. Viking, 2019

    2019

  39. [47]

    Artificial intelligence risk management framework (ai rmf 1.0)

    Elham Tabassi. Artificial intelligence risk management framework (ai rmf 1.0). Technical Report NIST AI 100-1, National Institute of Standards and Technology, 2023

    2023

  40. [48]

    Exploitgym: Can ai agents turn security vulnerabilities into real attacks?arXiv preprint arXiv:2605.11086, 2026

    Zhun Wang, Nico Schiller, Hongwei Li, Srijiith Sesha Narayana, Milad Nasr, Nicholas Carlini, Xiangyu Qi, Eric Wallace, Elie Bursztein, Luca Invernizzi, Kurt Thomas, Yan Shoshitaishvili, Wenbo Guo, Jingxuan He, Thorsten Holz, and Dawn Song. Exploitgym: Can ai agents turn securi...

    2026 doi arXiv

  41. [49]

    Weick and Kathleen M

    Karl E. Weick and Kathleen M. Sutcliffe.Managing the Unexpected: Resilient Performance in an Age of Uncertainty. Jossey-Bass, 2 edition, 2007

    2007

  42. [50]

    Sutton and Andrew G

    Richard S. Sutton and Andrew G. Barto.Reinforcement Learning: An Introduction. MIT Press, 2 edition, 2018

    2018

  43. [51]

    Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press

    John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated soft- ware engineering. InAdvances in Neural Information Processing Systems, volume 37, pages 50528–50652, 2024....

    2024 doi

  44. [52]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. InInternational Conference on Learning Representations, 2023

    2023

  45. [53]

    Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, et al

    Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, et al. Webarena: A realistic web environment for building autonomous agents. InInternational Conference on Learning Representations, 2024. 16

    2024

  46. [54]

    Autogen: En- abling next-gen llm applications via multi-agent conversation.arXiv preprint arXiv:2308.08155, 2023

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, et al. Autogen: En- abling next-gen llm applications via multi-agent conversation.arXiv preprint arXiv:2308.08155, 2023

    2023 arXiv

  47. [2010]

    doi: 10.1075/nlp.8.11bry

    doi

  48. [2020]

    doi: 10.1145/3351095.3372873

  49. [2026]

    Letter dated June 18, 2026. 14

    2026

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.