Pith. sign in

REVIEW 4 major objections 5 minor 103 references

Agent Safety Should Be a Runtime Contract

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper argues that agent safety is not a model property but a runtime contract enforced by the system's harness, combining preventive gates and evidence-gated submission, and supports the claim with four audits.

desk verdict The core claim isn't new, but the trajectory/evidence-chain framing and the four audits make this a useful synthesis worth engaging; the 52-incident counterfactual coding is the load-bearing empirics and needs independent re-coding. read the letter →

arxiv 2608.11274 v1 pith:3XIDOM4N submitted 2026-08-11 cs.CR cs.AI

Albus W. Ng , Yi Han , Jusheng Zhang , Wenhao Wang This is my paper · ORCID
classification cs.CRcs.AI
keywords agentsafetyruntimecontractevidence-gatedsubmissiontrajectoryschemahardevidenceAIalignmentharnessincidentsurvey
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that safety for autonomous AI agents is not something a model can be trained to possess; it must be enforced at runtime by the system's harness. The claimed contract has two faces: a preventive face that blocks dangerous actions before they happen, and an evidential face that refuses to mark a task complete unless the trajectory contains verifiable artifacts. The authors support this with a survey of 52 documented incidents, a 32-case false-completion audit, an audit of 12 agent systems, and a title-level audit of 28,560 papers at three major machine-learning venues. If the argument is right, the field's current emphasis on training-time alignment is structurally misplaced for consequential agents.

What carries the argument

The load-bearing formalism is the Agent Trajectory Schema: a finite sequence of events, each carrying a type, timestamp, payload, and hash, where the hash chains each event to its predecessor so any modification invalidates all later hashes. The paper distinguishes hard evidence, verifiable by a deterministic polynomial-time verifier against an external reference state, from soft evidence that depends on trusting the model's self-report. An Evidence Chain is a subsequence that satisfies every task-specific requirement with hard evidence; the harness accepts the submission only when it can construct one. The compositional gating proposition shows that preventive finite-state monitors with disjoint alphabets and independent evidence gates compose into a single contract, with verification polynomial in the disjoint and sequential cases.

What would settle it

Using the paper's released row-level JSON, re-code the 40 'fully preventable' incidents under the published protocol but require every proposed counterfactual layer to be an actually existing, publicly documented mechanism at the incident date; if five or more cases no longer qualify as fully preventable, the headline proportion is unsupported.

Watch

Extended reading notes

Core claim

The central claim is that agent safety is a runtime contract, not a model property. The contract comprises preventive mechanisms—sandboxes, permission gates, output filters, and trajectory monitors—that bound what the agent can do, and evidential mechanisms—evidence-gated submission, hard-evidence chains, and replayable trajectories—that determine when a task counts as done. The unit of safety, the paper concludes, is the trajectory-with-checkable-evidence rather than the model itself. Four lines of evidence are offered: 40 of 52 incidents were coded fully preventable by a layered harness, the false-completion cases would have been caught by a minimal evidence requirement, only 2 of 12 audited systems document submission-like evidence gates, and publication counts show a pooled 8–12× imbalance toward training-time alignment.

Load-bearing premise

The load-bearing premise is that the counterfactual coding of the 52 incidents correctly infers from public reports alone whether a layered harness would have prevented or mitigated each failure.

Editorial extensions

If this is right

  • Deployed agents should be required to emit tamper-evident trajectory logs before their outputs are trusted.
  • Task completion should be gated on verifiable artifacts such as passing test runs, citation lookups, file diffs, and log captures, rather than on the model's declaration of being done.
  • Safety research and safety regulation should shift attention from model-level alignment benchmarks toward runtime harness design and deployment-time verification.
  • Architectural responsibility moves: the agent produces the artifact and the harness verifies it, and neither responsibility belongs to the user's trust in the model's reasoning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the hash-chained trajectory could become the basis of a vendor-neutral 'lab notebook' for agents, making agent outputs audit-ready in scientific and legal workflows; the paper gestures toward this but does not develop the interoperability format.
  • Editorial inference: a concrete near-term prediction follows from the 8–12× publication imbalance—if the runtime-contract view is correct, deployment-time harness work should visibly increase at major venues over the next few years, and this can be checked empirically.
  • Editorial inference: the evidential face suggests a benchmark design the paper lists only as agenda: disable a model's alignment (e.g., by fine-tuning on a small set of examples) and measure whether a two-faced harness still blocks dangerous actions and rejects incomplete evidence chains.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper argues that model-level alignment (RLHF, DPO, Constitutional AI) is structurally insufficient for autonomous agents that execute consequential actions, and that agent safety should instead be a runtime contract enforced by a harness with two faces: a preventive face (sandboxes, permission gates, output filters, trajectory monitors) and an evidential face (evidence-gated submission requiring verifiable artifacts such as test runs, log captures, file diffs, and citation grounding). The claim is supported by four lines of evidence: a survey of 52 documented incidents coded for preventability by layered harnesses, a false-completion audit of 31 core cases plus one disputed case, a trajectory-schema audit of 12 public agent systems, and a title-level audit of 28,560 NeurIPS/ICML/ICLR papers. The paper also formalizes an Agent Trajectory Schema and Evidence Chain, states a compositional gating proposition, and outlines a research agenda. The central conclusion is that the right unit of safety in agentic AI is the trajectory-with-checkable-evidence, not the model.

Significance. If the central claim holds, the paper offers a useful architectural reframing: safety shifts from a property instilled during training to a runtime contract with both preventive and evidential guarantees, aligning agentic AI with practices in computer security and experimental science. The paper is commendable for releasing row-level protocols and data in supplementary JSON files, for explicitly flagging disputed rows and limitations, and for making concrete falsifiable claims (e.g., that only 2 of 12 surveyed systems document submission-like evidence gates). The formal trajectory schema and the discussion of evidence-gated submission are valuable contributions to the agent-safety discussion. However, the empirical weight rests on counterfactual coding and selectively assembled audits; unless those can be independently reproduced, the 'structural' conclusion outruns the evidence.

major comments (4)
  1. [Section 5, 'Preventive Face: The 52-incident survey'] The 40/11/1 preventability split is the paper's strongest empirical support for the preventive face, but it depends entirely on counterfactual coding from public incident reports. The reviewed text does not include the coding protocol, the inclusion/exclusion criteria, or the row-level decisions; the supplementary JSON files are announced but not present in the reviewed material. Because public reports often lack details about the deployment's permission model, sandboxing, and network policy, an independent re-coding may not reproduce the split. The authors should present the protocol and row-level data in the paper or in a reviewer-accessible supplement, and report inter-coder reliability or a comparable audit trail.
  2. [Section 5 and Table 1] There is a partial circularity in the audits. Incidents are coded using a taxonomy of harness layers, and the false-completion audit is assembled from cases whose ground truth contradicts the agent's output, so the conclusion that harness layers are load-bearing is partly entailed by the coding scheme and case selection. The paper should address this by including negative cases (incidents where a layered harness would not have helped), pre-specifying coding rules before seeing outcomes, or explicitly reframing the audits as illustrative rather than as tests of the structural claim.
  3. [Section 4.3, 'Compositional Gating'] The compositional gating proposition is stated as a formal result, but the text says only that 'the proof is the standard parallel composition of finite-state monitors under disjoint observation alphabets' and then asserts complexity bounds. No proof sketch is given, the notion of verifier independence is not formalized, and the parameters governing the polynomial versus exponential cases are not specified. If the proposition is meant to be a formal contribution, it needs a proof or a precise statement with hypotheses; otherwise it should be explicitly labeled as a conjecture or research agenda item.
  4. [Section 5, 'The 28,560-paper proceedings audit'] The pooled 8–12x training/deployment imbalance is derived from title-level keyword counts with truncation-corrected ranges and five classification rules that are not fully specified in the text. The claim that publication attention remains concentrated on training-time alignment is directionally plausible, but the quantitative ratio is not independently verifiable from the reviewed text. The authors should provide the keyword sets, classification rules, and per-cell counts in the paper, or verify that the supplementary JSON contains all of them in a form reviewers can check.
minor comments (5)
  1. [Definitions 1–3] The distinction between hard and soft evidence depends on the existence of a verifier in the set V; as written, any event paired with some verifier becomes hard evidence, so the definition is tautological. Please clarify that hardness is a property of the verifier and the event's format, not of the event alone.
  2. [Table 2] The 'Yes count' row mixes binary and 'partial' entries; specify how partial scores are treated in the counts (for example, whether partial is counted as no).
  3. [References] References [88], [92], and [98] appear to concern image copy detection and video object segmentation rather than agent trajectories or database snapshots; please verify that these citations support the claims in Section 4.2.
  4. [Section 5] The disputed AWS Kiro incident appears to be used in both the incident survey and the false-completion audit; please clarify whether it is double-counted and how its disputed status affects both rows.
  5. [Appendix A, Limitations] The limitation that the audits oversample English-language coverage should also explicitly apply to the proceedings audit, which is restricted to three English-language venues; please make that scope explicit.

Circularity Check

1 steps flagged · score 4.0 of 10

Partial circularity in the false-completion audit; the runtime-contract claim otherwise has independent support.

  1. self definitional [Section 5, 'Evidential Face: The 32-case false-completion audit' (Table 1)]
    "Core cases met four criteria: (a) real, dated, publicly documented incident; (b) the agent or model produced an output claiming correctness or completion; (c) known ground truth contradicted the output; and (d) two independent sources were available. ... Each case includes a minimal evidence requirement. The evidence check would have prevented acceptance of the false completion."

    The audit selects cases by criterion (b) (a completion claim) and (c) (ground-truth contradiction). Under Definition 3, an evidence-gated harness accepts a submission only when it can construct an evidence chain of verified hard evidence for the task. So any false completion with a tailored 'minimal evidence requirement' fails the gate by construction. The audit therefore cannot empirically establish that evidence gates are load-bearing: the preventability claim is a restatement of the selection criteria plus the definition, while only the existence of the 32 incidents is an empirical finding.

full rationale

The paper's central architectural claim — agent safety should be a runtime contract with preventive and evidential faces — is not itself derived from a fitted parameter or a self-citation chain. The preventive-face incident survey is a counterfactual annotation and is reliability-critical (the paper flags one disputed row and relies on public reports), but the 40/11 count is not entailed by a definition, so I do not count it as circular under the hard rules. The false-completion audit, by contrast, defines its cases as outputs claiming completion that ground truth contradicts and then assigns a minimal evidence requirement; the statement that the evidence check would have prevented acceptance follows from Definition 3's evidence-gated submission contract, making that supporting claim circular. The remaining evidence lines (trajectory-schema audit, proceedings title audit, and the external computer-security/science histories) are independent, and the self-citations (e.g., Wang et al. for image/vision verification) are peripheral, not load-bearing. Overall, one supporting audit reduces by construction while the central position retains independent grounding, giving partial circularity rather than full collapse.

Assumptions & free parameters 0 free parameters · 5 assumptions · 1 invented entities

The central claim rests on several domain assumptions: the validity of counterfactual coding for incidents, the proxy value of title-level publication counts, the applicability of standard monitor composition, the historical analogy, and the reliability of external scheming and reward-hacking studies. No numeric free parameters are fitted. One conceptual entity, the trajectory-with-checkable-evidence as the unit of safety, is introduced without independent falsifiable evidence.

assumptions (5)
  • domain assumption The counterfactual coding of the incident survey, which assigns 40 of 52 incidents as fully preventable by a layered harness, is correct despite relying only on public reports.
    Section 5 applies the authors' own harness taxonomy to historical incidents without a controlled counterfactual, making the headline counts dependent on subjective judgment.
  • domain assumption Title-level keyword classification of 28,560 accepted papers approximates the true research focus on training-time versus deployment-time safety.
    The proceedings audit in Section 5 uses four keyword sets and five classification rules, and the paper itself notes that counts are lower-bound estimates, so the 8-12x imbalance is only an indirect proxy.
  • standard math Standard parallel composition of finite-state monitors applies to the proposed harness monitors and evidence gates.
    Section 4.3 relies on the composition theorem of Kupferman and Vardi, citing rather than proving it, and assumes pairwise disjoint observation alphabets.
  • domain assumption The historical analogy between computer security and experimental science is a valid model for agentic AI.
    Sections 2 and 3 generalize from those fields to agentic AI, but the analogy is argued from examples rather than established by a formal mapping.
  • domain assumption The cited evidence on in-context scheming and reward hacking (Apollo Research, METR) correctly describes frontier model behavior at deployment time.
    Section 3.2 uses these external results to assert that model self-reports are unfaithful, treating the cited evaluations as established ground truth.
invented entities (1)
  • Trajectory-with-checkable-evidence as the unit of agent safety
    purpose: Reframes the object to which safety guarantees attach: not the model, but the audited trajectory of events with verifiable artifacts.
    This is a proposed conceptual unit with no falsifiable prediction attached; it is a design choice and framing device rather than an empirically discovered entity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Agent Safety Should Be a Runtime Contract." pith.science (2026). https://pith.science/paper/3XIDOM4N

@misc{pith2026260811274,
  author       = {Pith},
  title        = {Pith review of: Agent Safety Should Be a Runtime Contract},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3XIDOM4N}},
  note         = {Machine review of arXiv:2608.11274}
}
read the original abstract

The dominant paradigm treats AI safety as a property to be instilled during model training via RLHF, DPO, or Constitutional AI. We argue this is structurally insufficient for autonomous agents that execute code, mutate files, send messages, and modify databases. Agent safety should be a runtime contract enforced by the harness, and the contract has two complementary faces. The preventive face blocks dangerous actions before they happen via sandboxes, permission gates, output filters, and trajectory monitors. The evidential face requires verifiable proof that good actions actually happened, gating task submission on hard evidence such as test runs, log captures, file diffs, and citation grounding. We ground the position in four lines of public evidence, with row-level protocols and data released in the supplementary JSON files: a survey of 52 documented AI-agent and LLM safety incidents, a false-completion audit with 31 non-contested core cases plus one disputed illustrative case, a trajectory-schema audit of 12 public agent systems and harnesses, and a title-level audit of all 28,560 papers accepted at NeurIPS, ICML, and ICLR 2023-2025 showing a pooled 8-12x imbalance between training-time and deployment-time publication. Two prior communities that needed to enforce safety, computer security and the experimental sciences, converged on runtime contracts with both preventive and evidential elements; agentic AI is now under the same pressure. We formalize an Agent Trajectory Schema and Evidence Chain, state a compositional gating proposition based on standard monitor composition, and outline a research agenda. The right unit of safety in agentic AI is the trajectory-with-checkable-evidence, not the model.

Figures

Figures reproduced from arXiv: 2608.11274 by the authors.

Figure 1
Figure 1. Two-faced harness for AI agents. Preventive and structural layers control execution, while [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

103 extracted references · 55 canonical work pages

  1. [1]

    Incident 622: Chevrolet dealer chatbot agrees to sell tahoe for $1, 2023

    AI Incident Database. Incident 622: Chevrolet dealer chatbot agrees to sell tahoe for $1, 2023. https: //incidentdatabase.ai/cite/622/

    2023

  2. [2]

    Incident 631: DPD chatbot malfunctioned and swore at customers, 2024

    AI Incident Database. Incident 631: DPD chatbot malfunctioned and swore at customers, 2024. https: //incidentdatabase.ai/cite/631/

    2024

  3. [3]

    Incident 1152: LLM-driven Replit agent reportedly executed unauthorized destructive commands during code freeze, 2025.https://incidentdatabase.ai/cite/1152/

    AI Incident Database. Incident 1152: LLM-driven Replit agent reportedly executed unauthorized destructive commands during code freeze, 2025.https://incidentdatabase.ai/cite/1152/

    2025

  4. [4]

    Alur.Principles of Cyber-Physical Systems

    R. Alur.Principles of Cyber-Physical Systems. MIT Press, 2015

    2015

  5. [5]

    Anderson.Security Engineering: A Guide to Building Dependable Distributed Systems

    R. Anderson.Security Engineering: A Guide to Building Dependable Distributed Systems. John Wiley & Sons, 3rd edition, 2020

    2020

  6. [6]

    C. Anil, E. Durmus, N. Panickssery, M. Sharma, J. Benton, S. Kundu, J. Batson, M. Tong, J. Mu, D. Ford, et al. Many-shot jailbreaking.Advances in Neural Information Processing Systems, 37:129696–129742, 2024

    2024

  7. [7]

    Claude 3.5 sonnet model card addendum.Anthropic Technical Report, 2024

    Anthropic. Claude 3.5 sonnet model card addendum.Anthropic Technical Report, 2024

    2024

  8. [8]

    Cursor CLI output format reference, 2025

    Anysphere / Cursor. Cursor CLI output format reference, 2025. https://cursor.com/docs/cli/re ference/output-format

    2025

Show all 103 references
  1. [9]

    Y . Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from hu- man feedback.arXiv preprint arXiv:2204.05862, 2022

    2022 arXiv

  2. [10]

    Y . Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon, et al. Constitutional AI: Harmlessness from AI feedback.arXiv preprint arXiv:2212.08073, 2022

    2022 arXiv

  3. [11]

    Baker, J

    B. Baker, J. Huizinga, L. Gao, Z. Dou, M. Y . Guan, A. Madry, W. Zaremba, J. Pachocki, and D. Farhi. Monitoring reasoning models for misbehavior and the risks of promoting obfuscation.arXiv preprint arXiv:2503.11926, 2025

    2025 arXiv

  4. [12]

    K. Beck. Test-driven development: By example. 2002

    2002

  5. [13]

    Bishop.Computer Security: Art and Science

    M. Bishop.Computer Security: Art and Science. Addison-Wesley, 2003

    2003

  6. [14]

    Bricken, A

    T. Bricken, A. Templeton, J. Batson, B. Chen, A. Jermyn, T. Conerly, N. Turner, C. Anil, C. Denison, A. Askell, et al. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023

    2023

  7. [15]

    Burke, H

    G. Burke, H. Schellmann, and The Associated Press. OpenAI’s transcription tool Whisper hallucinates more than any other, 2024. https://fortune.com/2024/10/26/openai-transcription-tool-w hisper-hallucination-rate-ai-tools-hospitals-patients-doctors/

    2024

  8. [16]

    Carlini, M

    N. Carlini, M. Nasr, C. A. Choquette-Choo, M. Jagielski, I. Gao, A. Awadalla, P. W. Koh, D. Ippolito, K. Lee, F. Tramer, et al. Are aligned neural networks adversarially aligned?Advances in Neural Information Processing Systems, 36, 2024

    2024

  9. [17]

    M. Cerullo. Air canada chatbot costs airline discount it wrongly offered customer, 2024. https: //www.cbsnews.com/news/aircanada-chatbot-discount-customer/

    2024

  10. [18]

    P. Chao, A. Robey, E. Dobriban, H. Hassani, G. J. Pappas, and E. Wong. Jailbreaking black box large language models in twenty queries.arXiv preprint arXiv:2310.08419, 2024

    2024 arXiv

  11. [19]

    P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences.Advances in neural information processing systems, 30, 2017

    2017

  12. [20]

    Moffatt v

    Civil Resolution Tribunal of British Columbia. Moffatt v. air canada, 2024 bccrt 149. CanLII, February

    2024

  13. [21]

    T. Claburn. Cursor AI support bot hallucinated its own company policy, 2025. https://www.theregis ter.com/2025/04/18/cursor_ai_support_bot_lies/

    2025

  14. [22]

    E. M. Clarke, Jr., O. Grumberg, and D. A. Peled.Model Checking. Cyber-Physical Systems Series. The MIT Press, 1999. ISBN 9780262032704. URL https://mitpress.mit.edu/9780262032704/mod el-checking/. 10

    1999

  15. [23]

    claude-code-log: Convert Claude Code JSONL transcripts to HTML, 2024

    Daaain. claude-code-log: Convert Claude Code JSONL transcripts to HTML, 2024. https://github .com/daaain/claude-code-log

    2024

  16. [24]

    Dalrymple, J

    D. Dalrymple, J. Skalse, Y . Bengio, S. Russell, M. Tegmark, S. Seshia, et al. Towards guaranteed safe AI: A framework for ensuring robust and reliable AI systems.arXiv preprint arXiv:2405.06624, 2024

    2024 arXiv

  17. [25]

    Elhage, T

    N. Elhage, T. Hume, C. Olsson, N. Schiefer, T. Henighan, S. Kravec, Z. Hatfield-Dodds, R. Lasenby, D. Drain, C. Chen, et al. Toy models of superposition.Transformer Circuits Thread, 2022

    2022

  18. [26]

    Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (AI act).Official Journal of the European Union, 2024

    European Parliament and Council. Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (AI act).Official Journal of the European Union, 2024

    2024

  19. [27]

    Fowler and M

    M. Fowler and M. Foemmel. Continuous integration. InThoughtWorks Technical Article, 2006

    2006

  20. [28]

    Scaling laws: Eugene V olokh on navigating libel and liability in the AI age, 2024

    Frazier, Kevin and V olokh, Eugene. Scaling laws: Eugene V olokh on navigating libel and liability in the AI age, 2024. https://www.lawfaremedia.org/article/scaling-laws--eugene-volokh--n avigating-libel-and-liability-in-the-ai-age

    2024

  21. [29]

    Ganguli, L

    D. Ganguli, L. Lovitt, J. Kernion, A. Askell, Y . Bai, S. Kadavath, B. Mann, E. Perez, N. Schiefer, K. Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned.arXiv preprint arXiv:2209.07858, 2022

    2022 arXiv

  22. [30]

    L. Gao, J. Schulman, and J. Hilton. Scaling laws for reward model overoptimization. InInternational Conference on Machine Learning. PMLR, 2023

    2023

  23. [31]

    Gauthier

    P. Gauthier. Aider git integration documentation, 2024.https://aider.chat/docs/git.html

    2024

  24. [32]

    Amazon experiences AWS outage believed to be caused by AI tools, 2026

    Gigazine. Amazon experiences AWS outage believed to be caused by AI tools, 2026. https://gigazi ne.net/gsc_news/en/20260223-aws-ai-outage/

    2026

  25. [33]

    Tracking GitHub Copilot’s sessions, 2024

    GitHub. Tracking GitHub Copilot’s sessions, 2024. https://docs.github.com/en/copilot/how -tos/use-copilot-agents/coding-agent/track-copilot-sessions

    2024

  26. [34]

    Greenblatt, C

    R. Greenblatt, C. Denison, B. Wright, F. Roger, M. MacDiarmid, S. Marks, J. Treutlein, T. Belonax, J. Chen, D. Duvenaud, et al. Alignment faking in large language models.arXiv preprint arXiv:2412.14093, 2024

    2024 arXiv

  27. [35]

    M. Gurman. Samsung bans staff’s ai use after spotting chatgpt data leak. Bloomberg, May 2023. URL https://www.bloomberg.com/news/articles/2023-05-02/samsung-bans-chatgpt-and-o ther-generative-ai-use-by-staff-after-leak

    2023

  28. [36]

    Cursor IDE support hallucinates lockout policy, causes user cancellations, 2025

    Hacker News. Cursor IDE support hallucinates lockout policy, causes user cancellations, 2025. https: //news.ycombinator.com/item?id=43683012

    2025

  29. [37]

    A. E. hattami, M. Thakkar, N. Chapados, and C. Pal. Webarena verified: Reliable evaluation for web agents. InWorkshop on Scaling Environments for Agents, 2025. URL https://openreview.net/for um?id=94tlGxmqkN

    2025

  30. [38]

    C. A. R. Hoare.Communicating Sequential Processes. Prentice Hall, 1985

    1985

  31. [39]

    Hubinger, C

    E. Hubinger, C. Denison, J. Mu, M. Lambert, M. Tong, M. MacDiarmid, T. Lanham, D. M. Ziegler, T. Maxwell, N. Cheng, et al. Sleeper agents: Training deceptive LLMs that persist through safety training. arXiv preprint arXiv:2401.05566, 2024

    2024 arXiv

  32. [40]

    Humble and D

    J. Humble and D. Farley. Continuous delivery: Reliable software releases through build, test, and deployment automation. 2010

    2010

  33. [41]

    H. Inan, K. Upasani, J. Chi, R. Rungta, K. Iyer, Y . Mao, M. Tontchev, Q. Hu, B. Fuller, D. Testuggine, et al. Llama guard: LLM-based input-output safeguard for human-AI conversations.arXiv preprint arXiv:2312.06674, 2023

    2023 arXiv

  34. [42]

    D. Kang. Swe-bench verified is flawed despite expert review: Utboost exposes gaps in test coverage. Medium, 2025. https://medium.com/@danieldkang/swe-bench-verified-is-flawed-despi te-expert-review-utboost-exposes-gaps-in-test-coverage-4b75c6b940c6

    2025

  35. [43]

    Kent and M

    K. Kent and M. Souppaya. Guide to computer security log management, NIST special publication 800-92. Technical report, NIST, 2006. 11

    2006

  36. [44]

    R. Kirk, I. Mediratta, C. Nalmpantis, J. Luketina, E. Hambro, E. Grefenstette, and R. Raileanu. Under- standing the effects of rlhf on llm generalisation and diversity. InThe Twelfth International Conference on Learning Representations

  37. [45]

    Metr’s observations of reward hacking in recent frontier models, 2025

    Kokotajlo, Daniel. Metr’s observations of reward hacking in recent frontier models, 2025. https: //www.lesswrong.com/posts/Zu4ai9GFpwezyfB2K/metr-recent-frontier-models-are-r eward-hacking

    2025

  38. [46]

    Krakovna, J

    V . Krakovna, J. Uesato, V . Mikulik, M. Rahtz, T. Everitt, R. Kumar, Z. Kenton, J. Leike, and S. Legg. Specification gaming: The flip side of AI ingenuity.DeepMind Blog, 2020

    2020

  39. [47]

    Kupferman and M

    O. Kupferman and M. Y . Vardi. Model checking of safety properties.Formal Methods in System Design, 19(3):291–314, 2001

    2001

  40. [48]

    C. Lecher. NYC’s AI chatbot tells businesses to break the law, 2024. https://themarkup.org/arti ficial-intelligence/2024/03/29/nycs-ai-chatbot-tells-businesses-to-break-the-l aw

    2024

  41. [49]

    C. Lecher. NYC AI chatbot touted by Adams tells businesses to break the law, 2024. https://www.th ecity.nyc/2024/03/29/ai-chat-false-information-small-business/

    2024

  42. [50]

    H. Lee, S. Phatale, H. Mansoor, T. Mesnard, J. Ferret, K. Lu, C. Bishop, E. Hall, V . Carbune, A. Rastogi, et al. Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback.arXiv preprint arXiv:2309.00267, 2023

    2023 arXiv

  43. [51]

    P. Lee. Learning from Tay’s introduction.Official Microsoft Blog, March 2016

    2016

  44. [52]

    Leucker and C

    M. Leucker and C. Schallhart. A brief account of runtime verification.The Journal of Logic and Algebraic Programming, 78(5):293–303, 2009

    2009

  45. [53]

    N. G. Leveson.Engineering a Safer World: Systems Thinking Applied to Safety. MIT Press, 2011

    2011

  46. [54]

    D. Mangan. Judge sanctions lawyers for brief written by AI with fake citations, 2023. https://www.cn bc.com/2023/06/22/judge-sanctions-lawyers-whose-ai-written-filing-contained-f ake-citations.html

    2023

  47. [55]

    Mazeika, L

    M. Mazeika, L. Phan, X. Yin, A. Zou, Z. Wang, N. Mu, E. Sakhaee, N. Li, S. Basart, B. Li, et al. HarmBench: A standardized evaluation framework for automated red teaming and robust refusal.arXiv preprint arXiv:2402.04249, 2024

    2024 arXiv

  48. [56]

    McGregor

    S. McGregor. Preventing repeated real world AI failures by cataloging incidents: The AI incident database. Proceedings of the AAAI Conference on Artificial Intelligence, 35(17):15458–15463, 2021

    2021

  49. [57]

    Meinke, B

    A. Meinke, B. Schoen, J. Scheurer, M. Balesni, R. Shah, and M. Hobbhahn. Frontier models are capable of in-context scheming. Technical report, Apollo Research, 2024

    2024

  50. [58]

    D. Merkel. Docker: Lightweight linux containers for consistent development and deployment.Linux Journal, 2014(239):2, 2014

    2014

  51. [59]

    Llama guard 3.Meta AI Technical Report, 2024.https://www.llama.com/docs/model-car ds-and-prompt-formats/llama-guard-3/

    Meta. Llama guard 3.Meta AI Technical Report, 2024.https://www.llama.com/docs/model-car ds-and-prompt-formats/llama-guard-3/

    2024

  52. [60]

    Nanda, L

    N. Nanda, L. Chan, T. Lieberum, J. Smith, and J. Steinhardt. Progress measures for grokking via mechanistic interpretability.International Conference on Learning Representations, 2023

    2023

  53. [61]

    Framework for improving critical infrastructure cyberse- curity, version 1.1

    National Institute of Standards and Technology. Framework for improving critical infrastructure cyberse- curity, version 1.1. Technical report, NIST, 2018

    2018

  54. [62]

    Artificial intelligence risk management framework (AI RMF 1.0)

    National Institute of Standards and Technology. Artificial intelligence risk management framework (AI RMF 1.0). Technical Report AI 100-1, NIST, 2024

    2024

  55. [63]

    ChatGPT wrongly accuses law professor of sexual assault, 2023

    Nelson, Jason. ChatGPT wrongly accuses law professor of sexual assault, 2023. https://decrypt.co /125712/chatgpt-wrongly-accuses-law-professor-sexual-assault

    2023

  56. [64]

    C. Olah, N. Cammarata, L. Schubert, G. Goh, M. Petrov, and S. Carter. Zoom in: An introduction to circuits.Distill, 2020

    2020

  57. [65]

    GPT-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    OpenAI. GPT-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  58. [66]

    Codex CLI features, 2025.https://developers.openai.com/codex/cli/features

    OpenAI. Codex CLI features, 2025.https://developers.openai.com/codex/cli/features. 12

    2025

  59. [67]

    SWE-Bench Benchmark Evaluation, 2024

    OpenHands. SWE-Bench Benchmark Evaluation, 2024. https://github.com/OpenHands/benchma rks/blob/main/benchmarks/swebench/README.md

    2024

  60. [68]

    Ouyang, J

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. InAdvances in Neural Information Processing Systems, volume 35, 2022

    2022

  61. [69]

    Perez, S

    E. Perez, S. Ringer, K. Lukoši ¯ut˙e, K. Nguyen, E. Chen, S. Heiner, C. Pettit, C. Olsson, S. Kundu, S. Kadavath, et al. Discovering language model behaviors with model-written evaluations.arXiv preprint arXiv:2212.09251, 2022

    2022 arXiv

  62. [70]

    T. Pillay. New tests reveal AI’s capacity for deception, 2024. https://time.com/7202312/new-tes ts-reveal-ai-capacity-for-deception/

    2024

  63. [71]

    Pnueli and R

    A. Pnueli and R. Rosner. On the synthesis of a reactive module. InProceedings of the 16th ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, pages 179–190, 1989

    1989

  64. [72]

    Provos, M

    N. Provos, M. Friedl, and P. Honeyman. Preventing privilege escalation. InProceedings of the 12th USENIX Security Symposium, 2003

    2003

  65. [73]

    X. Qi, Y . Zeng, T. Xie, P.-Y . Chen, R. Jia, P. Mittal, and P. Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to.arXiv preprint arXiv:2310.03693, 2024

    2024 arXiv

  66. [74]

    Rafailov, A

    R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn. Direct preference opti- mization: Your language model is secretly a reward model.Advances in neural information processing systems, 36:53728–53741, 2023

    2023

  67. [75]

    Rebedea, R

    T. Rebedea, R. Dinu, M. N. Sreedhar, C. Parisien, and J. Cohen. NeMo guardrails: A toolkit for controllable and safe LLM applications with programmable rails.arXiv preprint arXiv:2310.10501, 2023

    2023 arXiv

  68. [76]

    Reddy and A

    P. Reddy and A. S. Gujral. Echoleak: The first real-world zero-click prompt injection exploit in a production llm system. InProceedings of the AAAI Symposium Series, volume 7, pages 303–311, 2025

    2025

  69. [77]

    C. Reis, A. Moshchuk, and N. Oskov. Site isolation: Process separation for web sites within the browser. InUSENIX Security Symposium, pages 1661–1678, 2019

    2019

  70. [78]

    Robinson

    D. Robinson. Amazon’s vibe-coding tool kiro reportedly vibed too hard and brought down aws, 2026. https://www.theregister.com/2026/02/20/amazon_denies_kiro_agentic_ai_behind_ou tage/

    2026

  71. [79]

    Y . Ruan, H. Dong, A. Wang, S. Pitis, Y . Zhou, J. Ba, Y . Dubois, C. J. Maddison, and T. Hashimoto. Identifying the risks of LM agents with an LM-emulated sandbox.arXiv preprint arXiv:2309.15817, 2024

    2024 arXiv

  72. [80]

    J. H. Saltzer and M. D. Schroeder. The protection of information in computer systems.Proceedings of the IEEE, 63(9):1278–1308, 1975

    1975

  73. [81]

    K. F. Schulz, D. G. Altman, and D. Moher. CONSORT 2010 statement: Updated guidelines for reporting parallel group randomised trials.BMJ, 340:c332, 2010

    2010

  74. [82]

    Sharma, M

    M. Sharma, M. Tong, T. Korbak, D. Duvenaud, A. Askell, S. R. Bowman, N. Cheng, E. Durmus, Z. Hatfield-Dodds, S. R. Johnston, et al. Towards understanding sycophancy in language models.arXiv preprint arXiv:2310.13548, 2023

    2023 arXiv

  75. [83]

    Sharwood

    S. Sharwood. Vibe coding service replit deleted user’s production database, faked data, told fibs galore,

  76. [84]

    Shavit et al

    Y . Shavit et al. Practices for governing agentic AI systems.OpenAI Research, 2023

    2023

  77. [85]

    do anything now

    X. Shen, Z. Chen, M. Backes, Y . Shen, and Y . Zhang. “do anything now”: Characterizing and evaluating in-the-wild jailbreak prompts on large language models. InProceedings of the 2024 ACM SIGSAC Conference on Computer and Communications Security, 2024

    2024

  78. [86]

    Skalse, N

    J. Skalse, N. H. R. Howe, D. Krasheninnikov, and D. Krueger. Defining and characterizing reward hacking.Advances in Neural Information Processing Systems, 35, 2022

    2022

  79. [87]

    Stiennon, L

    N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. V oss, A. Radford, D. Amodei, and P. F. Christiano. Learning to summarize with human feedback. InAdvances in Neural Information Processing Systems, volume 33, 2020. 13

    2020

  80. [88]

    Z. Tan, W. Wang, and C. Shan. Vision transformers are active learners for image copy detection. Neurocomputing, 587:127687, 2024. doi: 10.1016/j.neucom.2024.127687

    2024

  81. [89]

    Templeton, T

    A. Templeton, T. Conerly, J. Marcus, J. Lindsey, T. Bricken, B. Chen, A. Pearce, C. Citro, E. Ameisen, A. Jones, et al. Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet, 2024. URL https://transformer-circuits. pub/2024/scaling-monosemanticity/ind...

    2024

  82. [90]

    United States District Court for the Southern District of New York. Mata v. Avianca, Inc., 22-cv-1461 (S.D.N.Y . june 22, 2023): Sanctions for submission of ChatGPT-fabricated citations, 2023. https: //en.wikipedia.org/wiki/Mata_v._Avianca,_Inc

    2023

  83. [91]

    Recent frontier models are reward hacking

    V on Arx, Sydney and Chan, Lawrence , and Barnes, Elizabeth. Recent frontier models are reward hacking. 2025.https://metr.org/blog/2025-06-05-recent-reward-hacking/

    2025

  84. [92]

    W. Wang, Y . Sun, W. Zhang, and Y . Yang. D2LV: A data-driven and local-verification approach for image copy detection.arXiv preprint arXiv:2111.07090, 2021. doi: 10.48550/arXiv.2111.07090

    2021 doi arXiv

  85. [93]

    solved issues

    Y . Wang, M. Pradel, and Z. Liu. Are “solved issues” in SWE-bench really solved correctly? an empirical study.arXiv preprint arXiv:2503.15223, 2025

    2025 arXiv

  86. [94]

    A. Wei, N. Haghtalab, and J. Steinhardt. Jailbroken: How does LLM safety training fail? InAdvances in Neural Information Processing Systems, volume 37, 2024

    2024

  87. [95]

    J. Wei, Y . Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, et al. Emergent abilities of large language models.Transactions on Machine Learning Research, 2022

    2022

  88. [96]

    Y . Wolf, N. Wies, O. Avnery, Y . Levine, and A. Shashua. Fundamental limitations of alignment in large language models.arXiv preprint arXiv:2304.11082, 2023

    2023 arXiv

  89. [97]

    X. Yang, X. Wang, Q. Zhang, L. Petzold, W. Y . Wang, X. Zhao, and D. Lin. Shadow alignment: The ease of subverting safely-aligned language models.arXiv preprint arXiv:2310.02949, 2023

    2023 arXiv

  90. [98]

    Z. Yang, J. Zhang, W. Wang, W. Han, Y . Yu, Y . Li, J. Wang, Y . Wei, Y . Sun, and Y . Yang. Towards multi-object association from foreground-background integration. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2021. URLhttps:/...

    2021

  91. [99]

    hallucinates

    Yang, John and Young, Kaisha. What to know about an AI transcription tool that “hallucinates” medical interactions, 2024. https://www.pbs.org/newshour/show/what-to-know-about-an-ai-tra nscription-tool-that-hallucinates-medical-interactions

    2024

  92. [100]

    Y . Zhu, T. Jin, Y . Pruksachatkun, et al. Establishing best practices for building rigorous agentic benchmarks. arXiv preprint arXiv:2507.02825, 2025

    2025 arXiv

  93. [101]

    A. Zou, Z. Wang, N. Carlini, M. Nasr, J. Z. Kolter, and M. Fredrikson. Universal and transferable adversarial attacks on aligned language models.arXiv preprint arXiv:2307.15043, 2023. 14 A Limitations and Research Agenda Limitations.The contract constrainsactions and submissio...

    2023 arXiv

  94. [2024]

    URLhttps://canlii.ca/t/k2spq

  95. [2025]

    https://www.theregister.com/2025/07/21/replit_saastr_vibe_coding_inciden t/

    2025

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.