Pith. sign in

REVIEW 4 major objections 5 minor 26 references

Safety-evaluation scores are not load-bearing until you can reconstruct the decision they claim to measure from the retained evidence.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-15 05:50 UTC pith:XZNLYB5Q

load-bearing objection Honest engineering methods paper: ships a usable reconstructability scorer and release gate with full reproducibility, but the scores are schema-conditioned by author-declared required sets that remain uncalibrated. the 4 major comments →

arxiv 2607.12469 v1 pith:XZNLYB5Q submitted 2026-07-14 cs.SE cs.AI

Agent-Safety Evaluations as Load-Bearing Evidence: A Vendor-Neutral, Cross-Harness Reconstructability Metric

Oleg Solozobov This is my paper
classification cs.SE cs.AI
keywords evaluation validityevidence sufficiencyagent-safety evaluationproperty-level reconstructabilitycounterfactual replaymonitorabilityEvidence Sufficiency Cardmonitor-coverage
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Agent-safety evaluations often report a clean outcome — task success, attack blocked, monitor AUROC — while the harness may have kept far too little of the run to reconstruct the decision that claim depends on. Two runs can share the same headline number and rest on materially different evidence regimes: one with tool calls, permissions, monitor view, and policy state; the other with a success flag and little else. This paper treats that gap as an evaluation-validity problem and supplies a vendor-neutral instrument for it: a property-level reconstructability metric over eight decision-property classes, a cross-harness adapter that emits per-decision Evidence Sufficiency Cards, a claim-evidence overclaim gap, a replayability-precondition probe, and a per-run monitor-coverage release gate. On existing public and bundled traces, with no new model runs, twelve-field sufficiency spans 0.458 to 0.833 under a shared surface reading, and replay preconditions fail on every scored real trace. The practical claim is narrow: safety claims should travel with their reconstructability vector so a reader can see what the number actually rests on.

Core claim

Identical nominal agent-safety outcomes can sit atop materially different evidence regimes, and reconstructability — whether captured evidence can reconstruct the decision a claim depends on — can be scored as a deterministic, vendor-neutral evaluation-validity metric. The paper operationalises that idea over eight property classes, Evidence Sufficiency Cards, an overclaim gap, a replay-precondition probe, and a monitor-coverage gate, and shows on existing traces that surface-similar inputs already differ substantially in evidence sufficiency.

What carries the argument

Property-level reconstructability over eight decision-property classes, materialised as per-decision Evidence Sufficiency Cards that score present-and-bound evidence against fixed required sets and feed a bottleneck vector, overclaim gap, replay-precondition probe, and monitor-coverage release gate.

Load-bearing premise

The lists of evidence each decision property is required to retain, and which properties a given claim depends on, are author-declared design choices rather than multi-annotator ground truth; if those lists are wrong, every score and gate inherits the error.

What would settle it

A controlled matched-outcome study across at least two real harnesses with multi-annotator-calibrated required-evidence sets in which reconstructability is uniformly high and the overclaim gap is near zero for shared nominal outcomes, or in which executing the specified counterfactual-replay protocol never changes a reconstructability verdict.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Safety-evaluation reports should ship the per-decision reconstructability vector and bottleneck score next to any task-success, attack-success, or monitor-AUROC number.
  • Harness maintainers can treat the scorer’s exit code as a release gate that blocks when required evidence or monitor coverage is missing.
  • Instrumentation effort can be prioritised by measured property-level gaps rather than by guesswork about which fields matter.
  • Structured safety cases that rest on evaluation results can treat the bottleneck reconstructability score as an explicit bound on how much weight those results can carry.
  • A zero overclaim gap between matched regimes means only equal evidence sufficiency, not absolute adequacy of either regime.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If reconstructability vectors became standard metadata on leaderboards, headline agent-safety rankings would start separating high-score low-evidence runs from high-score high-evidence ones.
  • The same Card schema could later gate continuous-integration pipelines for agent deployments, not only offline evaluation papers.
  • Because the scorer is deterministic and model-free, low scores on public traces are a packaging signal rather than a donor deficiency — a distinction that could reduce unfair blame when open datasets are re-used as release evidence.
  • Once required-evidence sets are multi-annotator calibrated, the overclaim gap becomes a natural audit unit for comparing harness configurations of the same evaluation.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper argues that many agent-safety evaluation outcomes are not load-bearing evidence because identical nominal results can rest on materially different evidence regimes, and that no vendor-neutral instrument yet scores reconstructability of the decision a claim depends on. It defines a property-level reconstructability metric over eight decision-property classes, a cross-harness adapter that emits per-decision Evidence Sufficiency Cards, a claim-evidence (overclaim) gap, a counterfactual-replay intervention protocol with an implemented replayability-precondition probe, and a per-run monitor-coverage release gate. On existing public and bundled traces (no new model runs), the released v0 scorer reports twelve-field sufficiency spanning 0.458–0.833 under a shared surface reading, unmet replay preconditions on every scored real trace, and a synthetic release-gate pair that blocks the raw variant (0.542) and passes the instrumented variant (0.667). A reproducibility package regenerates every reported number. The demonstration is explicitly descriptive and not pre-registered; Definition 4’s G and executed replay are deferred to a v2 protocol.

Significance. If the construct holds after calibration, the paper would give the agent-safety community a practical evaluation-validity instrument: a deterministic, schema-based reconstructability vector that can travel with task-success, attack-success, and monitor-score claims, plus a machine-checkable monitor-coverage release gate. Strengths that should count in the assessment include: (i) an unusually explicit design-to-release correspondence (Table 1) and threat map (T1–T5, FW1–FW2); (ii) a runnable, deterministic scorer with byte-identical cards and a package that regenerates every §6 number; (iii) a clear wedge against failure-attribution systems (AgenTracer) and agent-claim gates (EviBound) by admitting insufficient evidence as a first-class verdict. Even as a documentation-completeness checker relative to a declared contract, the reporting practice—publish the reconstructability vector with the headline number—is a useful contribution to evaluation validity in cs.SE / agent-safety evaluation.

major comments (4)
  1. [Abstract, §1, Definitions 2–4, Table 1, §6.1–§6.2] The abstract and §1 frame a property-level metric over eight unit-interval classes, bottleneck R_min (Definition 3), and overclaim gap G (Definition 4), but §6 reports only the released v0 twelve-field mean-like sufficiency_score and a field-level analogue of G. Table 1 states that equivalence is not established and that G itself is unmeasured. This is load-bearing for the central empirical claim: the 0.458–0.833 span and the 0.542/0.667 gate pair are not measurements of Definitions 2–4. Either retitle/reframe results strictly as v0 field-level documentation completeness, or add the bottleneck and property-resolved quantities the design promises.
  2. [Definition 6, §4.2, §7 T1, §6 Tables 2–3] Definition 2’s r_p = |present(p)|/|R_p| and the bottleneck R_min inherit author-declared required-evidence sets R_p and claim-required subset R (Definition 6; §4.2). Threat T1 correctly names this the central calibration target and defers multi-annotator labeling to FW2, but without any sensitivity analysis (e.g., leave-one-element-out or alternative R_p variants) every reported sufficiency, blocking_gaps list, and gate exit is schema-relative rather than trace-relative. For the instrument to support the claim that it measures reconstructability of the decision a safety claim depends on, the manuscript needs at least a partial calibration or a systematic sensitivity study of R_p membership on the same traces.
  3. [§1 RQ2/H2, Definition 4, §6.2] RQ2 and the paper’s central empirical story rest on “identical nominal outcomes” / “shared surface reading,” yet §6.2 states that the four scored inputs share only surface usability, not a matched nominal outcome, so the comparison sits outside Definition 4’s domain. The only same-task matched pair is synthetic (raw vs instrumented billing exchange). The 0.375 cross-substrate spread therefore does not instantiate the overclaim gap the introduction advertises. Tighten H2/RQ2 claims to what was measured, or add at least one real matched-outcome pair before treating the spread as evidence of overclaim.
  4. [Abstract, §3.5, §4.3, §5.1, §6.3] Cross-harness and replay claims are overstated relative to the substrate. §5.1 demonstrates ingestion on Inspect and generic JSON/JSONL; ControlArena is implemented but fixture-only. RQ3 is answered only in precondition form on a sparse v0 mapping (F03/F09/F10; §4.3), with several §3.5 preconditions not operationalised. The abstract’s “cross-harness” and “counterfactual-replay intervention protocol” language should be scoped to what was actually exercised, with executed replay and a second real harness left explicitly to v2.
minor comments (5)
  1. [§4.4, Table 3, §6.1] §4.4 notes that F01 and F12 are scoring-time / card-self provenance, so sufficiency_score modestly overstates trace-earned completeness. Report a trace-earned-only score alongside the twelve-field score in Table 3 to avoid misreading.
  2. [§3.1, Table 1] The eight property classes are a designed cross-walk from Nian et al. (2026) plus two additions; a short table mapping each class to the auditability dimensions and to the v0 F-fields would help readers track the refactor.
  3. [Definition 6, §7 T5] Unconditional membership (Definition 6) scores inapplicable elements as absent. Flag this more prominently in the results when interpreting low r_p / partial fields on read-only or non-delegated runs.
  4. [§2] Several related-work citations are 2025–2026 arXiv preprints; where peer-reviewed versions exist or are forthcoming, note status so the neighbourhood survey ages better.
  5. [§6.2] Minor clarity: “surface reading” is used as a technical term in §6.2 without a one-sentence definition; define it when first used.

Circularity Check

2 steps flagged

Minor self-citation of the author's prior pilot plus synthetic fixtures that raise scores by construction; public-trace gaps remain independent observations relative to a fixed (if uncalibrated) schema.

specific steps
  1. self citation load bearing [§1 Introduction (extension of prior pilot); also §3.1, References Solozobov 2026b]
    "This construct does not begin from scratch. The author's prior pilot introduced property-level reconstructability for agent decisions, observing that "per-property reconstructability of an agent decision already varies between regimes at this anchor scale" across six vendor SDK trace regimes (Solozobov, 2026b)."

    The eight property classes and the reconstructability construct are framed as an extension of the same author's single-annotator descriptive pilot. That prior work is not an external, independently verified uniqueness or calibration result; it is the author's own unvalidated schema pilot. The present paper's design inherits that self-sourced foundation. The circularity is mild because the pilot is cited as background rather than as the sole warrant for the public-trace findings, and the current scorer's application to new inputs is still an independent observation relative to the schema.

  2. self definitional [§5.1 Table 2 roles; §6.2 release-gate pair; Abstract / §6 span 0.458–0.833]
    "the instrumented variant adding a resolved model digest, a scorer manifest, an explicit monitor, and recorded scores. The raw variant cards at sufficiency 0.542 ... the instrumented variant at 0.667: the pair's field-level analogue of G ... is |0.667 − 0.542| = 0.125 ... Gate consequence: under –gate with –require-present F07, the raw variant exits 1 and the instrumented variant exits 0"

    The synthetic release-gate pair and the best-instrumented synthetic fixture (sufficiency 0.833) were hand-built to add exactly the fields the scorer rewards. The gate block/pass and the upper end of the reported sufficiency span are therefore true by construction of the fixtures rather than independent empirical discoveries about real harnesses. The paper labels them anchors, so the circularity is limited to those demonstration numbers, not to the public-trace scores.

full rationale

This is a methods/engineering paper that defines a reconstructability metric, implements a deterministic scorer, and applies it to existing traces. It does not claim first-principles predictions of external quantities that reduce to fitted inputs. The required-evidence sets R_p are author-declared design choices (honestly flagged as threat T1 / FW2), which is a construct-validity limitation, not circular derivation: applying a fixed schema to public traces and observing different field statuses is an independent observation relative to that schema. The only mild circularities are (1) extension from the author's own prior pilot as background for the eight property classes, and (2) synthetic fixtures deliberately instrumented to add the fields that raise sufficiency_score, so the release-gate pair 0.542/0.667 and the upper end of the 0.458–0.833 span are partly by construction of the anchors. Neither forces the central empirical claim about public traces. No uniqueness theorem, fitted-parameter-as-prediction, or ansatz-smuggling chain is present. Score 2 reflects minor self-citation and synthetic demonstration-by-construction without load-bearing circular reduction of the main result.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 5 invented entities

The central claim rests on a designed schema (eight property classes, twelve v0 fields, three-valued statuses, bottleneck/min aggregation, six replay preconditions) whose membership is author-declared, plus domain assumptions imported from auditability and control literature. Free parameters are discrete design choices rather than continuous fits to outcome data. Invented entities are the Card, the overclaim gap, and the reconstructability vector as formal reportable objects. Independent multi-annotator validation of R_p is explicitly deferred.

free parameters (5)
  • Required-evidence set membership R_p (eight classes)
    Author-declared finite field lists that fix every r_p denominator (Definition 6); not fitted to external labels in this paper.
  • v0 twelve-field check set F01–F12 and three-valued statuses (1.0/0.5/0.0)
    Discrete operationalisation of the design contract; aggregation into sufficiency_score and verdict tiers is a hand-chosen mapping (Table 1, §4.4).
  • Claim-required subset R for bottleneck R_min
    Which properties enter the headline scalar is a declared per-claim parameter (Definition 3), not learned.
  • Risky-tool set for monitor-coverage gate
    Team-owned policy defaulting to all tool-bearing turns; changes coverage_ratio by construction (§4.5).
  • Verdict capping rules (missing/partial F03/F07/F09/F10 cap at conditional)
    Conservative tier rules chosen by the author; they determine whether –gate blocks.
axioms (5)
  • ad hoc to paper A safety-evaluation claim is reconstructable to the degree that a fixed set of decision properties can be recovered from captured evidence.
    Definitional core of §3; not derived from a prior theorem.
  • domain assumption Auditability dimensions (action recoverability, policy checkability, responsibility attribution, etc.) can be refactored to the evaluation-claim unit with two added classes (claim sufficiency, evaluation-awareness confounding).
    Cross-walk from Nian et al. (2026) and the author's prior pilot; asserted in §3.1.
  • domain assumption Deterministic schema-based presence counting without a model-in-the-loop is an appropriate sufficiency instrument (vs. LLM judges).
    Motivated by TRAIL's poor LLM trace-debugging scores; design choice in §2–§3.
  • ad hoc to paper Opaque but in-principle-observable elements (e.g., internal reasoning) count as absent for scoring.
    Scoring convention in Definition 2; collapses opaque and structural absence for the numeric score.
  • ad hoc to paper Unconditional membership of required elements (no applicability mask) is acceptable for v0, even if it depresses r_p for inapplicable fields.
    Stated in Definition 6; deferred fix to calibration stage.
invented entities (5)
  • Evidence Sufficiency Card (schema evidence-sufficiency-card.v0) independent evidence
    purpose: Per-decision (v0: per-trace) machine-checkable record of present/partial/missing evidence and reconstructability outputs.
    New schema contract operationalising the metric; independent handle is the released scorer and fixtures, not external physics-style prediction.
  • Property-level reconstructability vector r = (r1..r8) and bottleneck R_min independent evidence
    purpose: Primary reportable quantity for evaluation validity; non-compensatory aggregate.
    Formalises reconstructability as evaluation-validity metric; independent evidence is only via the shipped deterministic scorer.
  • Claim-evidence (overclaim) gap G no independent evidence
    purpose: Scalar/vector divergence of reconstructability between matched-outcome regimes.
    Defined in Definition 4; v0 only measures a field-level analogue, so the full entity is partly prospective.
  • Replayability-precondition probe (possible/degraded/impossible) independent evidence
    purpose: Score whether counterfactual replay is well-defined without executing it.
    Separates precondition form of RQ3 from executed-replay v2 protocol; implemented against a sparse F03/F09/F10 mapping.
  • Monitor-coverage release gate (monitor-coverage-card.v0) independent evidence
    purpose: Per-run all-or-block invariant that an identified monitor scored each risky tool span.
    Composes Card presence into a CI-usable exit code; demonstrated on synthetic covered/uncovered fixtures.

pith-pipeline@v1.1.0-grok45 · 28514 in / 3995 out tokens · 35090 ms · 2026-07-15T05:50:28.552241+00:00 · methodology

0 comments
read the original abstract

Many agent-safety evaluation results are not yet load-bearing evidence: identical nominal outcomes (task success, attack success, monitor scores) may sit atop materially different evidence regimes. No vendor-neutral, runnable instrument scores reconstructability as an evaluation-validity metric: whether captured evidence can reconstruct the decision a claim depends on. This paper introduces a property-level reconstructability metric over eight decision-property classes and a cross-harness adapter emitting per-decision Evidence Sufficiency Cards backing a per-run monitor-coverage release check. It specifies a counterfactual-replay intervention protocol, implements its replayability-precondition probe, and defines a claim-evidence overclaim gap. On public and bundled traces, without new model runs, twelve-field sufficiency spans 0.458-0.833 across four inputs sharing a surface reading; replay preconditions are unmet in every scored trace. In a synthetic release-gate pair, the sufficiency gate blocks the raw variant (0.542) and passes the instrumented (0.667). Safety-evaluation claims should travel with their reconstructability vector; a reproducibility package regenerates every reported number.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

26 extracted references · 4 canonical work pages · 1 internal anchor

  1. [1]

    How does information access affect LLM monitors’ ability to detect sabotage? arXiv

    https://doi.org/10.48550/arXiv.2602.10133 • Arike, R.; Moreno, R.M.; Subramani, R.; Biswas, S.; Ward, F.R. How does information access affect LLM monitors’ ability to detect sabotage? arXiv

    doi:10.48550/arxiv.2602.10133

  2. [2]

    Towards evaluations-based safety cases for AI scheming

    https://doi.org/10.4 8550/arXiv.2601.21112 • Balesni, M.; Hobbhahn, M.; Lindner, D.; Meinke, A.; Korbak, T.; Clymer, J.; et al. Towards evaluations-based safety cases for AI scheming. arXiv

    arXiv

  3. [3]

    org/10.48550/arXiv.2411.03336 • Benton, J.; Wagner, M.; Christiansen, E.; Anil, C.; Perez, E.; Srivastav, J.; et al

    https://doi. org/10.48550/arXiv.2411.03336 • Benton, J.; Wagner, M.; Christiansen, E.; Anil, C.; Perez, E.; Srivastav, J.; et al. Sabotage Evaluations for Frontier Models. arXiv

    Pith/arXiv arXiv doi:10.48550/arxiv.2411.03336

  4. [4]

    Ctrl-Z: Controlling AI Agents via Resampling

    https://doi.org/10.48550/arXiv.2410.21514 33 • Bhatt, A.; Rushing, C.; Kaufman, A.; Tracy, T.; Georgiev, V.; Matolcsi, D.; et al. Ctrl-Z: Controlling AI Agents via Resampling

    Pith/arXiv arXiv doi:10.48550/arxiv.2410.21514

  5. [5]

    In-Context Environments Induce Evaluation-Awareness in Language Models

    https://doi.org/10.48550/arXiv.2504.10374 • Chaudhary, M. In-Context Environments Induce Evaluation-Awareness in Language Models. arXiv

    Pith/arXiv arXiv doi:10.48550/arxiv.2504.10374

  6. [6]

    In-Context Environments Induce Evaluation-Awareness in Language Models

    https://doi.org/10.48550/arXiv.2603.03824 • Chen, R. Evidence-Bound Autonomous Research (EviBound): A Governance Framework for Eliminating False Claims. arXiv

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2603.03824

  7. [7]

    TRAIL: Trace Reasoning and Agentic Issue Localization

    https://doi.org/10.48550/arXiv.2511.05524 • Deshpande, D.; Gangal, V.; Mehta, H.; Krishnan, J.; Kannappan, A.; Qian, R. TRAIL: Trace Reasoning and Agentic Issue Localization

    doi:10.48550/arxiv.2511.05524

  8. [8]

    Seven simple steps for log analysis in AI systems

    https://doi.org/10.48550/arXiv.2 505.08638 • Dubois, M.; Zorer, E.; Hamin, M.; Skinner, J.; Souly, A.; Wynne, J.; et al. Seven simple steps for log analysis in AI systems. arXiv

    doi:10.48550/arxiv.2

  9. [9]

    Monitoring Monitorability

    https://doi.org/10.48550/arXiv.2604.09 563 • Guan, M.Y.; Wang, M.; Carroll, M.; Dou, Z.; Wei, A.Y.; Williams, M.; et al. Monitoring Monitorability. arXiv

    work page doi:10.48550/arxiv.2604.09

  10. [10]

    MonitoringBench: Semi-Automated Red-Teaming for Agent Monitoring

    https://doi.org/10.48550/arXiv.2512.18311 • Jotautaitė, M.; Martinez, M.A.; Matthews, O.; Tracy, T. MonitoringBench: Semi-Automated Red-Teaming for Agent Monitoring. arXiv

    doi:10.48550/arxiv.2512.18311

  11. [11]

    Evaluating whether AI models would sabotage AI safety research

    https://doi.org/10 .48550/arXiv.2605.09684 • Kirk, R.; Souly, A.; Fronsdal, K.; D’Cruz, A.; Davies, X. Evaluating whether AI models would sabotage AI safety research. arXiv

    Pith/arXiv arXiv

  12. [12]

    SHADE-Arena: Evaluating Sabotage and Monitoring in LLM Agents

    https://doi.org/10.48550/arXiv.2604.24 618 • Kutasov, J.; Sun, Y.; Colognese, P.; van der Weij, T.; Petrini, L.; Zhang, C.B.C.; et al. SHADE-Arena: Evaluating Sabotage and Monitoring in LLM Agents

    work page doi:10.48550/arxiv.2604.24

  13. [13]

    LLMs Can Covertly Sandbag on Capability Evaluations Against Chain-of-Thought Monitoring

    https: //doi.org/10.48550/arXiv.2506.15740 • Li, C.; Siegel, N.Y. LLMs Can Covertly Sandbag on Capability Evaluations Against Chain-of-Thought Monitoring. IJCNLP-AACL (short)

    Pith/arXiv arXiv doi:10.48550/arxiv.2506.15740

  14. [14]

    Auditing language models for hidden objectives

    https://doi.org/10.186 53/v1/2025.ijcnlp-short.33 • Marks, S.; Treutlein, J.; Bricken, T.; Lindsey, J.; Marcus, J.; Mishra-Sharma, S.; et al. Auditing language models for hidden objectives. arXiv

    2025

  15. [15]

    Frontier Models are Capable of In-context Scheming

    https://doi.org/10.48550 /arXiv.2503.10965 • Meinke, A.; Schoen, B.; Scheurer, J.; Balesni, M.; Shah, R.; Hobbhahn, M. Frontier Models are Capable of In-context Scheming. arXiv

    Pith/arXiv arXiv

  16. [16]

    MALT: Dataset of natural and prompted behaviors

    https://doi.org/10.48550/arXiv.2412.04 984 • METR. MALT: Dataset of natural and prompted behaviors. METR

    work page doi:10.48550/arxiv.2412.04

  17. [17]

    SHUSHCAST: Early work on monitorability evaluations

    A vailable online: https://metr.org/blog/2025-10-14-malt-dataset-of-natural-and-prompted-behaviors/ 34 • METR. SHUSHCAST: Early work on monitorability evaluations. METR

    2025

  18. [18]

    Auditable Agents

    A vailable online: https://metr.org/blog/2026-01-19-early-work-on-monitorability-evaluations/ • Nian, Y.; Yuan, A.; Zhang, H.; Li, J.; Zhao, Y. Auditable Agents. arXiv 2026, 1–23. https://doi.org/10.48550/arxiv.2604.05485 • OpenAI. Tracing - OpenAI Agents SDK Documentation. OpenAI Agents SDK Documentation (Tier A vendor primary doc)

    Pith/arXiv arXiv doi:10.48550/arxiv.2604.05485 2026

  19. [19]

    Towards a Science of AI Agent Reliability

    https: //doi.org/10.48550/arXiv.2505.01420 • Rabanser, S.; Kapoor, S.; Kirgis, P.; Liu, K.; Utpala, S.; Narayanan, A. Towards a Science of AI Agent Reliability. arXiv (Princeton University preprint) 2026, 1–66. https://doi.or g/10.48550/arxiv.2602.16666 • Saparning, D.D. AI Deployment Authorisation: A Global Standard for Machine-Readable Governance of Hig...

    Pith/arXiv arXiv doi:10.48550/arxiv.2505.01420 2026

  20. [20]

    inspect-evidence-sufficiency: Evidence-Sufficiency-Card scorer and monitor-coverage release check (v0.3.0 archived software deposit)

    https://doi.org/10.48550/a rXiv.2601.08869 • Solozobov, O. inspect-evidence-sufficiency: Evidence-Sufficiency-Card scorer and monitor-coverage release check (v0.3.0 archived software deposit). Zenodo (archived software release) 2026a. https://doi.org/10.5281/zenodo.21055696 • Solozobov, O. Property-Level Reconstructability of Agent Decisions: An Anchor-Leve...

    doi:10.48550/a

  21. [21]

    LinuxArena: A Control Setting for AI Agents in Live Production Software Environments

    https://doi.org/10.4 8550/arXiv.2606.00341 • Tracy, T.; Potham, R.; Kuhn, N.; Heller, M.; Khandelwal, A.; Rushing, C.; et al. LinuxArena: A Control Setting for AI Agents in Live Production Software Environments. arXiv

    Pith/arXiv arXiv

  22. [22]

    Inspect: An open-source framework for large language model evaluations

    https://doi.org/10.48550/arXiv.2604.15384 • UK AI Security Institute (AISI). Inspect: An open-source framework for large language model evaluations. UK AI Security Institute (official documentation)

    Pith/arXiv arXiv doi:10.48550/arxiv.2604.15384

  23. [23]

    CTRL-ALT-DECEIT: Sabotage Evaluations for Automated AI R&D

    https://doi.org/10.48550/arXiv.2606.04990 • Ward, F.R.; van der Weij, T.; Gábor, H.; Martin, S.; Moreno, R.M.; Lidar, H.; et al. CTRL-ALT-DECEIT: Sabotage Evaluations for Automated AI R&D

    Pith/arXiv arXiv doi:10.48550/arxiv.2606.04990

  24. [24]

    AI Sandbagging: Language Models can Strategically Underperform on Evaluations

    https: //doi.org/10.48550/arXiv.2511.09904 • Weij, T.V.D.; Hofstätter, F.; Jaffe, O.; Brown, S.F.; Ward, F.R. AI Sandbagging: Language Models can Strategically Underperform on Evaluations. arXiv

    doi:10.48550/arxiv.2511.09904

  25. [25]

    AgenTracer: Who Is Inducing Failure in the LLM Agentic Systems? arXiv

    https: //doi.org/10.48550/arXiv.2406.07358 • Zhang, G.; Wang, J.; Chen, J.; Zhou, W.; Wang, K.; Yan, S. AgenTracer: Who Is Inducing Failure in the LLM Agentic Systems? arXiv

    Pith/arXiv arXiv doi:10.48550/arxiv.2406.07358

  26. [26]

    03312 36

    https://doi.org/10.48550/arXiv.2509. 03312 36

    doi:10.48550/arxiv.2509