Pith. sign in

REVIEW 3 major objections 4 minor 71 references

SAGE makes catastrophic risk a hard pre-utility release gate

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 04:07 UTC pith:TSJYGM76

load-bearing objection A careful, honest design proposal for safety-first lifecycle control; the formal results are spec-level and the empirics are too weak for vendor rankings, but the architecture and the common-cause bound are worth engaging with. the 3 major comments →

arxiv 2607.22926 v1 pith:TSJYGM76 submitted 2026-07-24 cs.AI cs.CR

SAGE: Safety-First Defense-in-Depth Guardrails for Verified Lifecycle Control of High-Impact Generative AI

Mahdi Eslamimehr This is my paper
classification cs.AI cs.CR
keywords generative AI safetydefense in depthsafety-first controlrelease manifestsruntime verificationauthorization separationrobust risk boundsguardrails
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

SAGE argues that catastrophic misuse of high-impact generative AI cannot be handled by prompt filtering alone; it must be a lifecycle-control problem. The paper's central proposal is a safety-first, authorization-separated architecture in which a signed release manifest fixes a maximum admissible catastrophic-enablement risk, and a lexicographic policy chooses the least costly safe action among actions that stay under that ceiling. Formal results establish that safety dominates secondary objectives, that tightening a ceiling or acknowledging distribution shift only shrinks admissible actions, that defense-in-depth miss probabilities cannot be certified below a common-cause mass without layer diversity, and that an unprivileged classifier or monitor cannot mint authorization to disclose. The accompanying frozen study of ten current assistant snapshots finds low observed harmful compliance, with differences concentrated in benign utility and redirection; the paper stresses this is a conservative, protocol-bound view, not an upper bound on operational risk. A sympathetic reader would care because, if the architecture and its verified abstractions hold, providers have a concrete, auditable way to make catastrophic-risk ceilings non-negotiable across release, operation, and incident response.

Core claim

The paper's central claim is that catastrophic-enablement risk should be modelled as a robust worst-case risk under an ambiguity set of calibrated posteriors, and that admissible actions are exactly those legal actions whose worst-case risk is at or below a signed ceiling. The lexicographic policy then minimizes secondary losses only inside that feasible set, and if no action is feasible it falls back to the least catastrophic legal action rather than relaxing the ceiling. The paper proves this ordering is unique and safe, shows that acknowledged distribution shift tightens the gate, derives conservative miss bounds for layered detectors, and proves an authorization cut: in an abstract model

What carries the argument

The load-bearing mechanism is the lexicographic safety-first policy: choose the action, from a legal ladder G0 through G4, with the smallest secondary loss among actions whose robust catastrophic-enablement risk is below the manifest's signed ceiling; if the feasible set is empty, choose the least catastrophic legal fallback. The robust risk is the supremum of expected catastrophic loss over an ambiguity set of posteriors, with a closed-form contamination-class instantiation, making the gate directly auditable. Around this core sit four supporting mechanisms: a common-cause dependence bound that makes layer diversity a formal requirement, a signed release manifest whose eligibility predicate

Load-bearing premise

The whole formal case rests on the assumption that the actual control plane, not just the abstract model, has no path from an unprivileged component into the authorized region — the paper verifies this only in a small model, and states that the formal results are specifications and abstractions, not deployed endpoints.

What would settle it

One concrete test: instrument the real control-plane implementation and search for any reachable state in which a classifier, policy engine, or monitor can set the authorization-valid flag or trigger the authorized incident action without a separately authorized token and human approval; a single such path refutes the authorization-cut guarantee. Separately, estimate the common-cause miss mass from red-team and drift data and compare it with the target joint-miss ceiling; if the measured mass exceeds the target, the defense-in-depth bound cannot be certified no matter how many detector layers

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the architecture is right, a model cannot go live simply because its prompt-level average looks safe; admission requires passing a signed manifest's domain-risk thresholds, readiness bits, rollback rehearsal, and independent approval, and any material change revokes admission.
  • Acknowledging distribution shift or tightening the risk ceiling can only shrink the set of admissible actions; it can never create admissibility, so cautious updates do not silently open new capability paths.
  • Defense in depth is formally bounded by common-cause failure: without measured diversity, no number of detector layers can certify a joint miss probability below the shared-failure mass, which is a concrete argument for varying providers, training data, and policy dependencies.
  • The authorization-cut property, if realized in an implementation, means a compromised classifier or policy engine cannot escalate to external disclosure; disclosure requires a valid token and separately authorized human approval.
  • The empirical results imply that snapshot-level differences among current assistants, under this single-generation protocol, are driven mainly by benign utility and safe redirection rather than harmful compliance; the paper explicitly cautions this is not an upper bound on operationally elicitable assistance.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A reader could take the common-cause bound as a design rule for any high-stakes layered safety system: report and reduce the shared-failure mass before counting layers, since layer count alone cannot certify below that mass.
  • If the authorization cut is implemented as a true structural separation rather than a nominal flag, it suggests a general control-plane pattern: components that can assert authorization are privileged and must be placed in a separate trust domain, not merely constrained by policy.
  • The preregistered best–worst gap protocol offers a direct test that future evaluations could reuse; if repeated sampling, multi-turn interaction, and sandboxed tool use reveal a wider gap than the observed 0.0333, the conservative single-generation protocol will have been shown to understate operational risk.
  • The architecture implies a likely shift from model-level safety scores to artifact-level release manifests, where each deployed exact configuration carries its own thresholds, expiry, and rollback contract.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes SAGE, a safety-first defense-in-depth guardrail architecture for high-impact generative AI. It formalizes a lexicographic safety-first policy, a robust ambiguity-set risk, detector miss bounds, signed release manifests, runtime monitoring, and an authorization cut, and reports two PRISM model checks plus a frozen vendor-symmetric empirical comparison of ten model snapshots. The empirical study reports low observed harmful compliance (0.0000–0.0333) and seven multiplicity-adjusted contrasts, mainly driven by benign utility and redirection; the paper explicitly frames the empirical results as a conservative, protocol-bound snapshot and prespecifies a stronger confirmatory extension.

Significance. The paper is a serious attempt to turn lifecycle-safety governance into explicit predicates and machine-checkable models. Its strengths are the candor of the limitations section, the closed-form contamination-class risk bound (Proposition 2), the common-cause dependence bound (Proposition 4) as a formal argument for detector diversity, and the transparent separation of illustrative PRISM constants from structural invariants. If the architecture were implemented with a faithful code-to-model mapping, the safety-first lexicographic gate and authorization separation could serve as a useful template. However, the formal results are mostly definitional, the central authorization-cut and lifecycle guarantees are verified only at the level of small abstract models, and the empirical evidence is too sparse to carry independent weight. The contribution is therefore best described as a specification-level design with a pilot evaluation, not a verified deployment.

major comments (3)
  1. [§4 (Theorem 4), §5.5, §9] The load-bearing 'authorization cut' is an assumption about the model, not a derived property of a concrete system. Theorem 4 assumes P(s'|s,a)=0 for all unprivileged states s, actions a, and authorized states s'; the induction then restates that assumption. Section 5.5 confirms the Boolean rows depend only on the transition support, and Section 9 states that production control-plane components are 'specifications and abstractions, not endpoint deployments.' As written, the PRISM result verifies the 9-state model, not SAGE. To support the abstract's 'verified lifecycle control,' the paper must either provide a concrete mapping from the actual policy engine/classifier/monitor code paths to the MDP states and transitions (ideally generated and model-checked), or explicitly downgrade the claim to a specification-level design invariant.
  2. [§4 (Theorem 1, Proposition 1, Theorem 3), Abstract] Several 'formal results' are definitional and should not be presented as substantive discoveries. Theorem 1 follows immediately from the lexicographic minimization and the branch order in the definition of gSF; Proposition 1 is the subset relation of feasibility sets; Theorem 3 says that a monotone conjunction cannot become true when a conjunct is falsified. These are useful consistency checks but they do not independently 'establish safety priority' in the sense implied by the abstract. The paper should label them as framework consistency/tautology checks and place the weight on the model-checking results plus an implementation mapping.
  3. [§7, Table 3, §9] The empirical claims rest on very sparse data: 449 successful judgments out of 1,588 judge attempts, 375 unique responses, minimum domain-specific shared counts 'frequently equal to one,' no human adjudication, and two Claude Opus snapshots non-estimable. The Holm-adjusted 'supported' contrasts are therefore computed on tiny, uneven cells and are sensitive to a handful of judgments. The paper is explicit about many of these limits, but the abstract's statement that seven contrasts 'were supported' conveys more robustness than the data justify. Please add a missingness/sensitivity analysis (e.g., bounds under worst-case imputation, or leave-one-judgment-out) and describe the contrasts as pilot-level evidence, or restrict the abstract to the architecture contribution.
minor comments (4)
  1. [§5.4 / §10] The preregistered confirmatory null Delta <= 0.0333 is motivated by the observed range, but 0.0333 is a point estimate from the current sample; a boundary based on the upper end of a confidence interval would be more defensible as a pre-registered threshold.
  2. [§4 (Proposition 6, 7)] The bounded-evidence and decay propositions are elementary; consider compressing them or moving to an appendix so the main text focuses on the model-checking contribution.
  3. [§8 / Table 4] The 'safety-gated view' re-presents point estimates as if they were known admissible values. Add a note that zero-tolerance admissibility is based on observed HCS and does not account for sampling uncertainty.
  4. [Reproducibility Statement] The statement says archival locations, content digests, and access procedure 'must be inserted before submission.' In the current manuscript these are placeholders; a journal submission should include them, along with the aggregate CSV/PRISM inputs that are currently omitted from the LaTeX archive.

Circularity Check

2 steps flagged

The formal 'authorization cut' and 'safety priority' results are built into the model's definitions and transition assumptions; the empirical study itself is independent.

specific steps
  1. self definitional [Section 4, Theorem 1 (existence and safety-priority separation)]
    "Theorem 1 (existence and safety-priority separation). If Alegal(e) ≠ ∅ and K, J are finite, gSF(e) exists uniquely. Secondary loss cannot select outside non-empty Fκ(e), or displace a strictly lower-K fallback when that set is empty. Proof. Finite non-empty argmin sets exist and ≺ selects one element. The branch order fixes feasibility, or minimumK, before any secondary tie break."

    gSF is defined earlier as min_≺ J over Fκ when Fκ is non-empty, and otherwise min_≺ K over legal actions. The theorem's conclusion and proof merely restate this definitional branch order. 'Safety priority' is therefore an axiom written into the lexicographic policy, not a result derived from more basic principles. The formal claim is true, but its content is supplied by the definition of gSF.

  2. self definitional [Section 4, Theorem 4 (authorization-cut reachability); Section 5.5]
    "If P (s′ |s, a) = 0 for all s∈N, all actions a available to the classifier or policy engine, and all s′ ∈A, then the maximum probability that an unprivileged component reaches disclosure is zero. ... unauthorized disclosure is excluded by the absence of any transition from unprivileged states into the authorized region (Theorem 4)."

    The theorem's antecedent is exactly the authorization cut it claims to establish: A contains all states with a valid authorization token and all disclosure states, so the assumption that no transition leads from N to A already says that unprivileged components cannot reach the authorized/disclosure region. The conclusion (zero reachability) is the same statement in path language. The PRISM check verifies a model whose transition support was constructed to satisfy this assumption, as Section 5.5 acknowledges ('enforced by construction'). Thus the central 'verified' authorization separation is an input assumption rather than an emergent, independently derived property.

full rationale

There is no fitted-input-called-prediction circularity: the empirical study is a frozen, vendor-symmetric benchmark with complete-case analysis, and the PRISM constants are explicitly illustrative rather than tuned. There is also no load-bearing self-citation chain; the cited conformal and cryptographic results are external and stated as conditional. The circularity is at the level of the formal framework: Theorem 1 makes safety priority part of the definition of gSF, and Theorem 4 makes the authorization cut part of the transition-graph assumption and then 'verifies' the same cut in an abstraction. The paper is transparent about this, repeatedly noting that the formal properties are specifications and abstractions, not endpoint deployments (§5.5, §9), which lowers the severity but does not remove the construction-equivalence of the central formal invariant. The architecture and empirical measurements retain independent content, so the overall circularity is partial rather than total.

Axiom & Free-Parameter Ledger

6 free parameters · 8 axioms · 0 invented entities

The paper introduces no new physical or ontological entities. Its components — signed release manifests, audit capsules, three-valued monitors — are process artifacts, not entities with independent falsifiable handles. The main free parameters are hand-chosen metric weights, illustrative PRISM constants, and manifest thresholds whose values are left unspecified.

free parameters (6)
  • BGS component weights = 0.45, 0.25, 0.15, 0.15
    Hand-chosen in Section 3; express a safety-first but utility- and rights-sensitive policy. The primary ranking uses these fixed weights, though sensitivity analysis over Dirichlet draws is reported.
  • Harmful-compliance score normalizations = denominator 12 for HCS; BU/4; RD/2
    Arbitrary ordinal scaling in Section 3 that defines all empirical metrics; rankings depend on this choice.
  • PRISM illustrative constants = 0.50, 0.04, 0.06, 0.20
    Table 5 values such as 0.02 and 0.03 are explicitly illustrative parameters, not measurements. They produce the reported quantitative diagnostics but are not fitted to data.
  • Manifest risk and review thresholds
    κ(e), εe, τu, τΔ, and secondary-loss weights are manifest-defined in Section 3 and 4; no calibration procedure is given, so any concrete safety guarantee depends on unstated values.
  • Confirmatory-test boundary 0.0333 = 0.0333
    Section 5.4 sets H0: Δconfirm ≤ 0.0333 using the observed HCS range from this study, not an independent standard.
  • Detector miss bounds εj and common-cause mass γ
    Proposition 4's certifiable miss probability depends on these; the paper says they are measured inputs but provides no estimation protocol or values.
axioms (8)
  • domain assumption Risk-domain set D and action ladder A = {G0..G4} are fixed as given.
    Section 3: the categorical structure of domains and actions is adopted, not derived from external requirements.
  • domain assumption Catastrophic loss C is bounded in [0,1] and posteriors lie in an ambiguity set Π(e).
    Section 3: all robust risk computations depend on this parameterization of uncertainty and loss.
  • ad hoc to paper Contamination class Π(e) = {(1−εe)π̂e + εe ν : ν ∈ Δ(Z)}.
    Section 4, Proposition 2: a modeling choice that yields a closed-form robust risk; a KL-ball alternative is mentioned but not used.
  • domain assumption Detector failure model: system failure requires ∩j Fj, with independence or conditional independence given a common-cause event B.
    Section 4, Propositions 3–4: the bounds are conditional on this dependence structure, which the paper acknowledges can be violated by shared training data or infrastructure.
  • domain assumption Cryptographic assumptions: collision-resistant hashing and existentially unforgeable signatures.
    Theorem 2: tamper evidence relies on standard cryptographic assumptions that may not hold in a given deployment.
  • domain assumption Conformal risk control's exchangeability and monotonicity conditions.
    Corollary 3: the cited finite-sample guarantee holds only under these conditions; the paper explicitly suspends it under unmeasured shift.
  • ad hoc to paper MDP transition support contains no edges from unprivileged states N to authorized states A.
    Theorem 4: the authorization cut is enforced by construction in the model; the PRISM check verifies this modeled graph, not any implementation.
  • standard math Standard mathematical facts (affine supremum attained at a vertex, geometric series, projection idempotence).
    Used in Propositions 2, 6–8; these are routine and correct.

pith-pipeline@v1.3.0-alltime-deepseek · 18823 in / 15372 out tokens · 156025 ms · 2026-08-01T04:07:58.714046+00:00 · methodology

0 comments
read the original abstract

High-impact generative AI makes catastrophic misuse a lifecycle-control problem, not merely a prompt-filtering problem. SAGE is a safety-first, authorization-separated architecture in which credible catastrophic-enablement risk constrains admissibility before utility, latency, or commercial objectives are considered. It combines signed release manifests, diverse detectors, robust risk envelopes, least-risk defaults, output checking, three-valued monitoring, protected audit chains, containment, and rollback. Formal results establish safety priority, conservative detector bounds, monotone release gating, tamper-evident records, and an authorization cut; two PRISM abstractions verify authorization separation and lifecycle invariants under explicit assumptions. A frozen, vendor-symmetric study sent 84 cases to each of four GPT, four Claude, and two Gemini snapshots: 840 calls yielded 794 target responses, 46 provider errors, and 449 successful judgments covering 375 responses. Eight snapshots had complete judged domain coverage. Harmful-compliance estimates were low; variation arose mainly from benign utility and safe redirection. Seven multiplicity-adjusted contrasts involving Claude, Gemini, or GPT-5 snapshots and the GPT-5 mini and GPT-5 nano snapshots were supported, while no tested contrast between the Claude or Gemini snapshots and GPT-5 or GPT-5.5 survived correction. The observed harmful-compliance range is a conservative, protocol-bound view from one generation per prompt with no tools, retrieval, history, or human adjudication; it is not an upper bound on operational assistance. A preregistered extension specifies how to test a wider best-worst gap using a locked split, repeated sampling, multi-turn and sandboxed-tool conditions, and domain-expert scoring.

Figures

Figures reproduced from arXiv: 2607.22926 by Mahdi Eslamimehr.

Figure 1
Figure 1. Figure 1: Safety-first SAGE architecture. Signed release gating precedes authenticated runtime [PITH_FULL_IMAGE:figures/full_fig_p011_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Complete-case balanced guardrail score intervals for estimable snapshots. [PITH_FULL_IMAGE:figures/full_fig_p018_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Component-level differences show that utility and redirection drove most observed BGS [PITH_FULL_IMAGE:figures/full_fig_p019_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Largest Holm-supported pairwise contrast, shown per BGS component on the full [0 [PITH_FULL_IMAGE:figures/full_fig_p020_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Judgment coverage was incomplete and uneven across domains, motivating complete-case [PITH_FULL_IMAGE:figures/full_fig_p021_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

71 extracted references · 6 linked inside Pith

  1. [1]

    and Bates, Stephen and Fisch, Adam and Lei, Lihua and Schuster, Tal , title =

    Angelopoulos, Anastasios N. and Bates, Stephen and Fisch, Adam and Lei, Lihua and Schuster, Tal , title =. International Conference on Learning Representations , year =

  2. [2]

    Claude System Cards , year =

  3. [3]

    225 , year =

    Framework Convention on Artificial Intelligence and Human Rights, Democracy and the Rule of Law, CETS No. 225 , year =

  4. [4]

    Proceedings of the 42nd International Conference on Machine Learning , year =

    Cui, Jiaxing and Chiang, Wei-Lin and Stoica, Ion and Hsieh, Cho-Jui , title =. Proceedings of the 42nd International Conference on Machine Learning , year =

  5. [5]

    Gemini 3 Pro Model Card , year =

  6. [6]

    Official Journal of the European Union , url =

    Regulation (EU) 2024/1689 Laying Down Harmonised Rules on Artificial Intelligence , year =. Official Journal of the European Union , url =

    2024

  7. [7]

    Advances in Neural Information Processing Systems , volume =

    Geifman, Yonatan and El-Yaniv, Ran , title =. Advances in Neural Information Processing Systems , volume =. 2017 , url =

    2017

  8. [8]

    Computer Aided Verification , series =

    Kwiatkowska, Marta and Norman, Gethin and Parker, David , title =. Computer Aided Verification , series =. 2011 , doi =

    2011

  9. [9]

    Journal of Logic and Algebraic Programming , volume =

    Leucker, Martin and Schallhart, Christian , title =. Journal of Logic and Algebraic Programming , volume =. 2009 , doi =

    2009

  10. [10]

    Proceedings of the 41st International Conference on Machine Learning , series =

    Mazeika, Mantas and Phan, Long and Yin, Xuwang and Zou, Andy and Wang, Zifan and Mu, Norman and Sakhaee, Elham and Li, Nathaniel and Basart, Steven and Li, Bo and Forsyth, David and Hendrycks, Dan , title =. Proceedings of the 41st International Conference on Machine Learning , series =. 2024 , url =

    2024

  11. [11]

    2023 , doi =

    Artificial Intelligence Risk Management Framework (. 2023 , doi =

    2023

  12. [12]

    2024 , doi =

    Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile , institution =. 2024 , doi =

    2024

  13. [13]

    2025 , howpublished =

    2025

  14. [14]

    Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages =

    R. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages =. 2024 , doi =

    2024

  15. [15]

    and Schroeder, Michael D

    Saltzer, Jerome H. and Schroeder, Michael D. , title =. Proceedings of the IEEE , volume =. 1975 , doi =

    1975

  16. [16]

    Shi, Lin and Yang, Wei and Cao, Ling and Lin, Yong and Liu, Yang and Liu, Yue and Li, Zhiqiang , title =. Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics , pages =. 2025 , doi =

    2025

  17. [17]

    Advances in Neural Information Processing Systems , volume =

    Souly, Alexandra and Lu, Qingyuan and Bowen, Dillon and Trinh, Tu and Hsieh, Elvis and Pandey, Sana and Abbeel, Pieter and Svegliato, Justin and Emmons, Scott and Watkins, Olivia and Toyer, Sam , title =. Advances in Neural Information Processing Systems , volume =. 2024 , doi =

    2024

  18. [18]

    International Conference on Learning Representations , year =

    Xie, Tinghao and Qi, Xiangyu and Zeng, Yi and Huang, Yangsibo and Sehwag, Udari and Huang, Kaixuan and others , title =. International Conference on Learning Representations , year =

  19. [19]

    Grok 4.1 Model Card , year =

  20. [20]

    Advances in Neural Information Processing Systems , volume =

    Zheng, Lianmin and Chiang, Wei-Lin and Sheng, Ying and Zhuang, Siyuan and Wu, Zhanghao and Zhuang, Yonghao and others , title =. Advances in Neural Information Processing Systems , volume =

  21. [21]

    2024 , url =

    Frontier. 2024 , url =

    2024

  22. [22]

    Preparedness Framework, Version 2.0 , year =

  23. [23]

    Responsible Scaling Policy, Version 3.0 , year =

  24. [24]

    Frontier Safety Framework, Version 3.0 , year =

  25. [25]

    2025 , url =

    General-Purpose. 2025 , url =

    2025

  26. [26]

    Secure by Design , year =

  27. [27]

    2023 , howpublished =

    Inan, Hakan and Upasani, Kartikeya and Chi, Jianfeng and Rungta, Rashi and Iyer, Krithika and Mao, Yuning and Tontchev, Michael and Hu, Qing and Fuller, Brian and Testuggine, Davide and Khabsa, Madian , title =. 2023 , howpublished =

    2023

  28. [28]

    Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages =

    Rebedea, Traian and Dinu, Razvan and Sreedhar, Makesh Narsimhan and Parisien, Christopher and Cohen, Jonathan , title =. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages =. 2023 , doi =

    2023

  29. [29]

    2024 , howpublished =

    Zeng, Wenjun and Liu, Yuchi and Mullins, Ryan and Peran, Ludovic and Fernandez, Joe and Harkous, Hamza and Narasimhan, Karthik and Proud, Drew and Kumar, Piyush and Radharapu, Bhaktipriya and Sturman, Olivia and Wahltinez, Oscar , title =. 2024 , howpublished =

    2024

  30. [30]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume =

    Markov, Todor and Zhang, Chong and Agarwal, Sandhini and Eloundou, Tyna and Lee, Teddy and Adler, Steven and Jiang, Angela and Weng, Lilian , title =. Proceedings of the AAAI Conference on Artificial Intelligence , volume =. 2023 , doi =

    2023

  31. [31]

    2025 , howpublished =

    Sharma, Mrinank and Tong, Meg and Mu, Jesse and Wei, Jerry and Kruthoff, Jorrit and Goodfriend, Scott and Ong, Euan and Peng, Alwin and Agarwal, Raj and Anil, Cem and others , title =. 2025 , howpublished =

    2025

  32. [33]

    and Deshpande, Kaustubh and Sirdeshmukh, Ved and Mankikar, Meher and Scale Red Team and SEAL Research Team and Michael, Julian , journal =

    Knight, Christina Q. and Deshpande, Kaustubh and Sirdeshmukh, Ved and Mankikar, Meher and Scale Red Team and SEAL Research Team and Michael, Julian , journal =. 2025 , url =

    2025

  33. [35]

    Safer or Luckier?

    Chen, Hongyu and Goldfarb-Tarrant, Seraphina , booktitle =. Safer or Luckier?. 2025 , doi =

    2025

  34. [36]

    Investigating the Potential Use of Frontier

    Aguirre, Jair and van Soest, Henri and Sperisen, Benjamin and Lopez, Zylex and Kong, Nicholas and Seri-Levi, Adam and Caridi-Doyle, James and Moisan, Elizabeth and Graham, Evie and Reid, William Mitchell , institution =. Investigating the Potential Use of Frontier. 2026 , doi =

    2026

  35. [37]

    2025 , howpublished =

    Why Do We Take. 2025 , howpublished =

    2025

  36. [38]

    Investigating the potential use of frontier AI models for offensive cyberattacks: A human uplift study

    Jair Aguirre, Henri van Soest, Benjamin Sperisen, Zylex Lopez, Nicholas Kong, Adam Seri-Levi, James Caridi-Doyle, Elizabeth Moisan, Evie Graham, and William Mitchell Reid. Investigating the potential use of frontier AI models for offensive cyberattacks: A human uplift study. Technical Report RR-A3892-1, RAND Corporation, 2026. URL https://www.rand.org/pub...

    2026

  37. [39]

    Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, and Tal Schuster

    Anastasios N. Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, and Tal Schuster. Conformal risk control. In International Conference on Learning Representations, 2024. URL https://proceedings.iclr.cc/paper_files/paper/2024/hash/f3549ef9b5ff520a7e41ff3cc306ab2b-Abstract-Conference.html

    2024

  38. [40]

    Why do we take LLM s seriously as a potential source of biorisk? https://www.anthropic.com/research/biorisk, 2025

    Anthropic . Why do we take LLM s seriously as a potential source of biorisk? https://www.anthropic.com/research/biorisk, 2025. Accessed 24 July 2026

    2025

  39. [41]

    Claude system cards, 2026

    Anthropic . Claude system cards, 2026. URL https://www.anthropic.com/system-cards. Accessed 21 July 2026

    2026

  40. [42]

    Safer or luckier? LLM s as safety evaluators are not robust to artifacts

    Hongyu Chen and Seraphina Goldfarb-Tarrant. Safer or luckier? LLM s as safety evaluators are not robust to artifacts. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 19750--19766, 2025. doi:10.18653/v1/2025.acl-long.970. URL https://aclanthology.org/2025.acl-long.970/

    doi:10.18653/v1/2025.acl-long.970 2025

  41. [43]

    Framework convention on artificial intelligence and human rights, democracy and the rule of law, cets no

    Council of Europe . Framework convention on artificial intelligence and human rights, democracy and the rule of law, cets no. 225, 2024. URL https://www.coe.int/en/web/artificial-intelligence/the-framework-convention-on-artificial-intelligence

    2024

  42. [44]

    OR-Bench : An over-refusal benchmark for large language models

    Jiaxing Cui, Wei-Lin Chiang, Ion Stoica, and Cho-Jui Hsieh. OR-Bench : An over-refusal benchmark for large language models. In Proceedings of the 42nd International Conference on Machine Learning, 2025. URL https://icml.cc/virtual/2025/poster/46052

    2025

  43. [45]

    Secure by design, 2023

    Cybersecurity and Infrastructure Security Agency . Secure by design, 2023. URL https://www.cisa.gov/resources-tools/resources/secure-by-design

    2023

  44. [46]

    General-purpose AI code of practice, 2025

    European Commission . General-purpose AI code of practice, 2025. URL https://digital-strategy.ec.europa.eu/en/policies/contents-code-gpai. Published 1 August 2025

    2025

  45. [47]

    Regulation (eu) 2024/1689 laying down harmonised rules on artificial intelligence, 2024

    European Union . Regulation (eu) 2024/1689 laying down harmonised rules on artificial intelligence, 2024. URL https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng

    2024

  46. [48]

    Selective classification for deep neural networks

    Yonatan Geifman and Ran El-Yaniv. Selective classification for deep neural networks. In Advances in Neural Information Processing Systems, volume 30, 2017. URL https://proceedings.neurips.cc/paper/2017/hash/4a8423d5e91fda00bb7e46540e2b0cf1-Abstract.html

    2017

  47. [49]

    Gemini 3 pro model card, 2026

    Google DeepMind . Gemini 3 pro model card, 2026. URL https://storage.googleapis.com/deepmind-media/Model-Cards/Gemini-3-Pro-Model-Card.pdf. Updated May 2026

    2026

  48. [50]

    Gowda and Hanrui Luo

    Shreyank N. Gowda and Hanrui Luo. An empirical study of multi-generation sampling for jailbreak detection in large language models. arXiv preprint arXiv:2604.18775, 2026. URL https://arxiv.org/abs/2604.18775

    Pith/arXiv arXiv 2026

  49. [51]

    Llama guard: LLM -based input-output safeguard for human- AI conversations

    Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa. Llama guard: LLM -based input-output safeguard for human- AI conversations. arXiv:2312.06674, 2023. URL https://arxiv.org/abs/2312.06674

    Pith/arXiv arXiv 2023

  50. [52]

    Knight, Kaustubh Deshpande, Ved Sirdeshmukh, Meher Mankikar, Scale Red Team, SEAL Research Team, and Julian Michael

    Christina Q. Knight, Kaustubh Deshpande, Ved Sirdeshmukh, Meher Mankikar, Scale Red Team, SEAL Research Team, and Julian Michael. FORTRESS : Frontier risk evaluation for national security and public safety. arXiv preprint arXiv:2506.14922, 2025. URL https://arxiv.org/abs/2506.14922

    Pith/arXiv arXiv 2025

  51. [53]

    PRISM 4.0: Verification of probabilistic real-time systems

    Marta Kwiatkowska, Gethin Norman, and David Parker. PRISM 4.0: Verification of probabilistic real-time systems. In Computer Aided Verification, volume 6806 of Lecture Notes in Computer Science, pages 585--591. Springer, 2011. doi:10.1007/978-3-642-22110-1_47

    doi:10.1007/978-3-642-22110-1_47 2011

  52. [54]

    A brief account of runtime verification

    Martin Leucker and Christian Schallhart. A brief account of runtime verification. Journal of Logic and Algebraic Programming, 78 0 (5): 0 293--303, 2009. doi:10.1016/j.jlap.2008.08.004

    doi:10.1016/j.jlap.2008.08.004 2009

  53. [55]

    A holistic approach to undesired content detection in the real world

    Todor Markov, Chong Zhang, Sandhini Agarwal, Tyna Eloundou, Teddy Lee, Steven Adler, Angela Jiang, and Lilian Weng. A holistic approach to undesired content detection in the real world. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 15009--15018, 2023. doi:10.1609/aaai.v37i12.26752

    doi:10.1609/aaai.v37i12.26752 2023

  54. [56]

    HarmBench : A standardized evaluation framework for automated red teaming and robust refusal

    Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. HarmBench : A standardized evaluation framework for automated red teaming and robust refusal. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Mach...

    2024

  55. [57]

    Artificial intelligence risk management framework ( AI RMF 1.0)

    National Institute of Standards and Technology . Artificial intelligence risk management framework ( AI RMF 1.0). Technical Report NIST AI 100-1, National Institute of Standards and Technology, 2023

    2023

  56. [58]

    Artificial intelligence risk management framework: Generative artificial intelligence profile

    National Institute of Standards and Technology . Artificial intelligence risk management framework: Generative artificial intelligence profile. Technical Report NIST AI 600-1, National Institute of Standards and Technology, 2024

    2024

  57. [59]

    CAISI evaluation of DeepSeek ai models finds shortcomings and risks

    National Institute of Standards and Technology . CAISI evaluation of DeepSeek ai models finds shortcomings and risks. NIST News, 2025. 30 September 2025

    2025

  58. [60]

    GPT-5 system card, 2025

    OpenAI . GPT-5 system card, 2025. URL https://openai.com/index/gpt-5-system-card/. 7 August 2025

    2025

  59. [61]

    GPT-5.5 system card, 2026

    OpenAI . GPT-5.5 system card, 2026. URL https://openai.com/index/gpt-5-5-system-card/. 23 April 2026

    2026

  60. [62]

    Evaluating frontier models for dangerous capabilities

    Mary Phuong, Matthew Aitchison, Elliot Catt, Sarah Cogan, Alexandre Kaskasoli, Victoria Krakovna, David Lindner, Matthew Rahtz, Yannis Assael, Sarah Hodkinson, Heidi Howard, Tom Lieberum, Ramana Kumar, Maria Abi Raad, Albert Webson, Lewis Ho, Sharon Lin, Sebastian Farquhar, Marcus Hutter, Gr \'e goire Del \'e tang, Anian Ruoss, Seliem El-Sayed, Sasha Brow...

    Pith/arXiv arXiv 2024

  61. [63]

    NeMo guardrails: A toolkit for controllable and safe LLM applications with programmable rails

    Traian Rebedea, Razvan Dinu, Makesh Narsimhan Sreedhar, Christopher Parisien, and Jonathan Cohen. NeMo guardrails: A toolkit for controllable and safe LLM applications with programmable rails. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 431--445. Association for Computational Ling...

    doi:10.18653/v1/2023.emnlp-demo.40 2023

  62. [64]

    XSTest : A test suite for identifying exaggerated safety behaviours in large language models

    Paul R \"o ttger, Hannah Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. XSTest : A test suite for identifying exaggerated safety behaviours in large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5377--5400....

    doi:10.18653/v1/2024.naacl-long.301 2024

  63. [65]

    Saltzer and Michael D

    Jerome H. Saltzer and Michael D. Schroeder. The protection of information in computer systems. Proceedings of the IEEE, 63 0 (9): 0 1278--1308, 1975. doi:10.1109/PROC.1975.9939

    arXiv 1975

  64. [66]

    Constitutional classifiers: Defending against universal jailbreaks across thousands of hours of red teaming

    Mrinank Sharma, Meg Tong, Jesse Mu, Jerry Wei, Jorrit Kruthoff, Scott Goodfriend, Euan Ong, Alwin Peng, Raj Agarwal, Cem Anil, et al. Constitutional classifiers: Defending against universal jailbreaks across thousands of hours of red teaming. arXiv:2501.18837, 2025. URL https://arxiv.org/abs/2501.18837

    Pith/arXiv arXiv 2025

  65. [67]

    Judging the judges: A systematic study of position bias in LLM -as-a-judge

    Lin Shi, Wei Yang, Ling Cao, Yong Lin, Yang Liu, Yue Liu, and Zhiqiang Li. Judging the judges: A systematic study of position bias in LLM -as-a-judge. In Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, pages 292--314. ...

    doi:10.18653/v1/2025.ijcnlp-long.18 2025

  66. [68]

    A StrongREJECT for empty jailbreaks

    Alexandra Souly, Qingyuan Lu, Dillon Bowen, Tu Trinh, Elvis Hsieh, Sana Pandey, Pieter Abbeel, Justin Svegliato, Scott Emmons, Olivia Watkins, and Sam Toyer. A StrongREJECT for empty jailbreaks. In Advances in Neural Information Processing Systems, volume 37, 2024. doi:10.52202/079017-3984

    doi:10.52202/079017-3984 2024

  67. [69]

    Frontier AI safety commitments, AI seoul summit 2024, 2024

    UK Department for Science, Innovation and Technology . Frontier AI safety commitments, AI seoul summit 2024, 2024. URL https://www.gov.uk/government/publications/frontier-ai-safety-commitments-ai-seoul-summit-2024

    2024

  68. [70]

    Grok 4.1 model card, 2025

    xAI . Grok 4.1 model card, 2025. URL https://data.x.ai/2025-11-17-grok-4-1-model-card.pdf. 17 November 2025

    2025

  69. [71]

    SORRY-Bench : Systematically evaluating large language model safety refusal

    Tinghao Xie, Xiangyu Qi, Yi Zeng, Yangsibo Huang, Udari Sehwag, Kaixuan Huang, et al. SORRY-Bench : Systematically evaluating large language model safety refusal. In International Conference on Learning Representations, 2025

    2025

  70. [72]

    ShieldGemma : Generative AI content moderation based on Gemma

    Wenjun Zeng, Yuchi Liu, Ryan Mullins, Ludovic Peran, Joe Fernandez, Hamza Harkous, Karthik Narasimhan, Drew Proud, Piyush Kumar, Bhaktipriya Radharapu, Olivia Sturman, and Oscar Wahltinez. ShieldGemma : Generative AI content moderation based on Gemma . arXiv:2407.21772, 2024. URL https://arxiv.org/abs/2407.21772

    Pith/arXiv arXiv 2024

  71. [73]

    Judging LLM -as-a-judge with MT-Bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, et al. Judging LLM -as-a-judge with MT-Bench and chatbot arena. In Advances in Neural Information Processing Systems, volume 36, 2023

    2023