Artificial Intelligence · arXiv:2611.16493 v1 · submitted 12 Nov 2026 · 70 references
ReA-Bench: Benchmarking Responsible Agent Auditing
The paper says
“As agents perform tasks autonomously in local and cloud environments, auditing their execution trajectories can help service providers identify risks and improve agent harnesses and foundation models.”
more of the abstract
Our analysis of production and red-team trajectories shows that some risks are repeatedly missed across auditing configurations. Overall recall obscures which risks are missed and whether higher recall comes with more false positives on legitimate behavior. Existing benchmarks offer limited insight into these failures. Generated trajectories can contain inconsistent tool calls and responses, conspicuous risky actions, or repetitive attack patterns. Grouping results by risk type also makes it difficult to tell whether auditors overlooked evidence, accepted an unsupported conclusion, or lacked information needed to make a judgment. We introduce ReA-Bench, a diagnostic benchmark for auditing responsible agent behavior. It organizes auditing challenges into six tiers: internal safety knowledge, contradictions within trajectories, evidence retrieval in long contexts, verification of sources and reasoning, indirect steering, and missing information. It includes challenging benign cases in which actions resemble risky ones but are consistent with the task and its authorization. It also provides paired trajectory versions to test whether auditors detect the same risks more reliably when the trajectory is shortened or missing information is supplied. Most samples are drawn from real production trajectories, with scenarios manually selected and adapted to preserve realistic tasks and risks that could plausibly arise during execution.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 13 Nov 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up. A well-constructed diagnostic benchmark for agent trajectory auditing with real data and a six-tier framework—the attribution of recall differences to specific capabilities is partly confounded by trajectory length, but the paper acknowledges this limitation upfront. This line is wrong
2 major objections · 6 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 70 references say.
The paper says, at 4.1, Table 3
The review did not quote this passage.
Pith says: Objection 1 of 2 · major
The paper states that 'recall is generally higher on L1 than on L2 and L4' and interprets this as indicating that risks requiring contextual judgment are harder than those requiring internal knowledge. However, the paper does not report per-tier trajectory length (message count, token count) or control for length in this cross-tier comparison.
more of objection 1
If L2 and L4 samples are systematically longer or more complex than L1 samples, the recall differences could partially reflect general difficulty with long contexts rather than the specific auditing requirement being tested. The paper acknowledged in Section 6 that 'low recall for a category of trajectories may not be directly attributable to a deficiency in any single model capability,' but this caveat is not reflected in the interpretation of Table 3. The authors should report per-tier trajectory length statistics add a brief discussion of whether length confounds the tier-level comparisons.
The paper says, at 4.3
The review did not quote this passage.
Pith says: Objection 2 of 2 · major
The RQ3 analysis shows that shortening trajectories yields only modest gains (87.1-89.2% threshold-decision agreement between original and short versions), which is an important result. However, the paper does not report the per-tier breakdown of these gains. If shortening benefits some tiers more than others (e.g., improving L3 evidence-dilution cases more than L4 logic cases), this would provide further insight into the nature of the auditing failures. The authors should consider providing tier-level agreement statistics for the shortening experiments.
6 minor objections
The paper says, at 1.3
Pith says: minor objection 1 of 6
The contributions list is clear, but the third bullet ('Evaluation across models and methods') could explicitly mention the five methods and eight models for specificity.
The paper says, at 4.1, Table 5
Pith says: minor objection 2 of 6
The Recall@FPR≤10% metric is important, but it is computed on the benchmark's deliberately hard benign samples. The paper acknowledges in Section 6 that these samples do not represent production distributions, but the main text could state this caveat more prominently when discussing low-FPR results to avoid overinterpretation.
show 4 more
The paper says, at 6
Pith says: minor objection 3 of 6
The limitation about single rums per configuration (line b411) is appreciated. However, the paper recommends 'repeated experiments are needed to confirm the stability of small performance differences' without specifying how many rums or what effect size is considered meaningful. A brief quantitative suggestion (e.g., based on standard errors given sample size) would strengthen this discussion.
The paper says, at Appendix A1
Pith says: minor objection 4 of 6
Figure 6 (dataset distribution) mentions 'median message count' but the text of Appendix A1 does not provide the per-tier medians. Reporting these would support the analysis in my major comment above.
The paper says, at 2.5
Pith says: minor objection 5 of 6
The quality assurance section describes LLM-assisted review, human review, and programmatic checks. It would be helpful to state how many human reviewers were involved and whether inter-rater reliability was assessed for the risk/benign labels.
The paper says, at 4.2, Table 10
Pith says: minor objection 6 of 6
The five categories of frequently missed risks are insightful. The 'Count' column sums to 26, matching the stated sample count. However, the category descriptions (e.g., 'Evidence does not support the conclusion' vs. 'Overinterpreting facts') could be made more operational to guide future method development.
What would settle it
For each sample in L1 (198), L2 (79), and L4 (90), compute: (a) input token count, (b) number of messages, (c) number of tool calls. Use logistic regression predicting detection (score >= 5) for one model-method combination (e.g., Grok-high FA) with predictors: tier, log(token_count), number_of_messages, and number_of_tool_calls. Test the marginal effect of tier. If L2/L4 coefficients remain negative and significant (p<0.05) after controlling for these covariates, the tier-level differences reflect the auditing requirement beyond structural complexity. If coefficients are no longer significant, the diagnostic conclusions are confounded.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 13 Nov 2026
- Reader holds up
- Skeptic partly agrees
- Editor wrote the verdict above
- Referee minor revision: 2 major objections, 6 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
This is a solid benchmark paper. The main contribution is the six-tier diagnostic organization, which breaks down auditing challenges by the kind of judgment required (internal knowledge, contradictions, evidence dilution, provenance/ logic, indirect steering, missing information) rather than by risk category. That shift is genuine and useful for debugging why auditors fail. The use of real production trajectories as seeds, paired with hard benign cases and version controls (shortened vs. original, missing vs. supplied information), is a significant step beyond existing LLM-generated or staged benchmarks. The evaluation is thorough: 8 model configs × 4 (later 5) methods, ROC analysis, tier-level breakdowns, and error pattern analysis. The results are transparently reported, including where GPT models have high recall but also high FPR, and where GSA helps or hurts. The paper ships the benchmark after de-identification, which is important for reproducibility.
The soft spot the stress-test note flags is real but not secret: recall differences between L1, L2, and L4 may be partly driven by differences in trajectory length or complexity, not solely by the targeted auditing requirement. The paper acknowledges in Section 6 that 'low recall for a category of trajectories may not be directly attributable to a deficiency in any single model capability.' That said, the paper would be stronger if it reported per-tier token/message counts and showed that the tier-level patterns hold after controlling for length, or explicitly noted that it cannot yet disentangle them. The current analysis treats tier-level recall as diagnostic, which is a reasonable starting point but should be read with the caveat in mind.
The other limitations (single runs per configuration, uneven sample sizes across fine-grained categories, hard benign samples not representative of production distributions) are stated honestly. None are fatal; they are the standard trade-offs of a hand-built benchmark.
This paper is for anyone working on agent safety monitoring, especially those who want to understand why a given auditor misses certain risks. It deserves a serious referee. I would recommend accept with a minor revision asking for per-tier trajectory length statistics and a sentence clarifying how the attribute-to-capability claim is bounded by the confound.
For peer review: accept with minor revision.
The strongest objection
The paper's six-tier framework is designed to test distinct auditing requirements, and a key result is that recall is generally lower on L2 (contradictions) and L4 (provenance/logic) than on L1 (internal knowledge) (Table 3). However, L2 and L4 samples may systematically differ from L1 samples in trajectory length, number of steps, or tool calls. Longer trajectories are independently harder to audit (as the L3 analysis shows, though L3 controls within-sample via shortening, this does not address cross-tier differences). The paper reports an overall median message count of ~66 (Appendix A1) but does not provide per-tier distributions of token counts or message counts. If L2 and L4 samples are systematically longer or more complex than L1 samples, the recall differences could partially reflect general difficulty with long contexts rather than the specific auditing requirement of contradiction detection or provenance verification. The L3 analysis controls for length within L3 samples but does not extend to L1/L2/L4 comparisons. This concern is load-bearing because the central diagnostic claim — that the tier framework reveals which auditing challenges are harder — relies on face-valid recall differences across tiers. Without controlling for structural confounds, the attribution of low recall to the specific capability being tested is weakened. The paper acknowledges in Section 6 that 'low recall for a category of trajectories may not be directly attributable to a deficiency in any single model capability,' but this does not address whether tier-level recall differences are partly driven by trajectory length disparities.
The reader's weakest_assumption identified that low recall may not be attributable to a single capability. My concern is a specific, testable instance of that: structural properties (length, step count) may confound tier-level recall comparisons. The paper acknowledges attribution limitations generally but does not control for this confound. A simple regression using existing per-sample metadata would settle whether the tier framework provides independent diagnostic information or merely reflects that L2/L4 samples are harder because they are longer. This concern does not undermine the benchmark's overall value or the descriptive recall results; it qualifies the strength of the diagnostic attribution. The paper's conclusions about contextual judgment being harder are also supported within L1 (17 contextual vs. 33 explicit samples), so the central claim is not entirely dependent on cross-tier comparisons. Verdict remains UNCHANGED because the benchmark is still a substantive advance, but the concrete test would strengthen the paper if tier effects hold after control, and would clarify limitations if they do not.
The referee's report
If the benchmark results hold, ReA-Bench provides a valuable diagnostic tool for the agent safety community. Unlike existing benchmarks that group results mainly by risk type, the six-tier framework organizes samples by the judgment an auditor must make, enabling more targeted debugging of auditing failures. The use of real production trajectories, hard benign samples, and paired version controls is a methodological advance over purely synthetic benchmarks. The evaluation across eight diverse models and five methods provides practically useful comparisons. The finding that recall differences across tiers persist even after controlling for trajectory length (R03) suggests the tier framework captures genuine auditing challenges beyond simple context-length effects.
Pith's reading of the paper (a machine's summary, not the authors')
Autonomous agents execute tasks by calling tools, reading files, and running scripts. Auditors review these execution trajectories to identify risky actions. Existing benchmarks use LLM-generated or simulated trajectories with obvious attacks. ReA-Bench uses real production trajectories as seeds, preserving tool responses, business context, and subtle risks.
The benchmark organizes 587 trajectories into six tiers. L1 tests whether auditors recognize unsafe operations such as file overwrites and reverse shells. L2 tests detection of contradictions between what the agent does and what the user asked. L3 tests finding evidence buried in long contexts. L4 tests verifying whether evidence is authentic and supports conclusions. L5 tests detecting indirect steering over multiple turns. L6 tests cases where decisive information is missing from the trajectory.
Each tier includes hard benign samples where actions resemble risky ones but are legitimate. Paired versions (shortened trajectories, added information) test how context affects judgments. Evaluation of eight LLMs and five auditing methods shows that auditors often miss risks requiring contextual judgment while frequently flagging benign samples that resemble attacks. Adding evidence through the global-summary method improves discrimination for most configurations but increases false positives.
Core claim. ReA-Bench provides a diagnostic evaluation of agent trajectory auditors by organizing samples into six tiers (L1-L6) that test distinct auditing requirements: internal safety knowledge, contradictions, evidence dilution, provenance verification, indirect steering, and missing information. The paper claims this design reveals that risks requiring contextual judgment are missed more often than clearly dangerous actions (Table 3: recall lower on L2 and L4 than L1 across all models) and that hard benign samples are frequently flagged (Table 11: mean FPR of 45.2% on development/maintenance benign samples for six non-GPT configurations).
The premise it rests on. The assumption that the six-tier categorization provides a meaningful diagnostic decomposition of auditing failures, such that low recall on a tier can be attributed primarily to the specific capability being tested. The paper acknowledges (Section 6) that 'auditing a single trajectory may involve rule understanding, evidence association, and logical verification at the same time' and that 'low recall for a category of trajectories may not be directly attributable to a deficiency in any single model capability.' This is a structural limitation of any diagnostic benchmark that categorizes by judgment type and limits the strength of conclusions drawn from tier-level scores.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
As agents perform tasks autonomously in local and cloud environments, auditing their execution trajectories can help service providers identify risks and improve agent harnesses and foundation models. Our analysis of production and red-team trajectories shows that some risks are repeatedly missed across auditing configurations. Overall recall obscures which risks are missed and whether higher recall comes with more false positives on legitimate behavior. Existing benchmarks offer limited insight into these failures. Generated trajectories can contain inconsistent tool calls and responses, conspicuous risky actions, or repetitive attack patterns. Grouping results by risk type also makes it difficult to tell whether auditors overlooked evidence, accepted an unsupported conclusion, or lacked information needed to make a judgment. We introduce ReA-Bench, a diagnostic benchmark for auditing responsible agent behavior. It organizes auditing challenges into six tiers: internal safety knowledge, contradictions within trajectories, evidence retrieval in long contexts, verification of sources and reasoning, indirect steering, and missing information. It includes challenging benign cases in which actions resemble risky ones but are consistent with the task and its authorization. It also provides paired trajectory versions to test whether auditors detect the same risks more reliably when the trajectory is shortened or missing information is supplied. Most samples are drawn from real production trajectories, with scenarios manually selected and adapted to preserve realistic tasks and risks that could plausibly arise during execution.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-11-13
The authors' main claim, as the classifier picked it out 0.60
“We introduce ReA-Bench, a diagnostic benchmark for auditing responsible agent behavior.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.08 · states limitation 0.20 · human participants 0.16 · controlled comparison 0.27 · randomized assignment 0.02 · states quantitative result 0.14 · compares with prior results 0.29 · mentions code or data availability 0.05
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| As agents perform tasks autonomously in local and cloud environments, auditing their execution trajectories can help service providers identify risks and improve agent harnesses and foundation models. | background 1.00 | 0.06 | 0.04 | hedged possibility 0.70 |
| Our analysis of production and red-team trajectories shows that some risks are repeatedly missed across auditing configurations. | result 0.96 | 0.94 | 0.17 | asserted 0.99 |
| Overall recall obscures which risks are missed and whether higher recall comes with more false positives on legitimate behavior. | gap 0.61 | 0.38 | 0.09 | asserted 1.00 |
| Existing benchmarks offer limited insight into these failures. | gap 0.97 | 0.06 | 0.03 | asserted 0.99 |
| Generated trajectories can contain inconsistent tool calls and responses, conspicuous risky actions, or repetitive attack patterns. | background 0.62 | 0.33 | 0.03 | asserted 0.75 |
| Grouping results by risk type also makes it difficult to tell whether auditors overlooked evidence, accepted an unsupported conclusion, or lacked information needed to make a judgment. | gap 0.81 | 0.30 | 0.05 | asserted 0.97 |
| We introduce ReA-Bench, a diagnostic benchmark for auditing responsible agent behavior. | contribution 1.00 | 0.91 | 0.13 | asserted 1.00 |
| It organizes auditing challenges into six tiers: internal safety knowledge, contradictions within trajectories, evidence retrieval in long contexts, verification of sources and reasoning, indirect steering, and missing information. | contribution 0.50 | 0.87 | 0.08 | asserted 1.00 |
| It includes challenging benign cases in which actions resemble risky ones but are consistent with the task and its authorization. | contribution 0.62 | 0.84 | 0.04 | asserted 0.99 |
| It also provides paired trajectory versions to test whether auditors detect the same risks more reliably when the trajectory is shortened or missing information is supplied. | method 0.71 | 0.85 | 0.04 | asserted 0.97 |
| Most samples are drawn from real production trajectories, with scenarios manually selected and adapted to preserve realistic tasks and risks that could plausibly arise during execution. | method 0.99 | 0.89 | 0.04 | asserted 0.94 |
What kind of paper the classifier reports this is: dataset or benchmark 1.00 · evidence from analysis of existing data 0.76
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-13: 1 request, 8,055 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (29)
Showing 28 of the paper's 29 figures. Figures are the authors', reproduced from the paper.
References (70)
- [1] Alibaba Cloud. qwen3.8-flash Model Info. Model Studio documentation, 2026. URLhttps://www. alibabacloud.com/help/en/model-studio/qwen3-8-flash. Accessed November 7, 2026 on Pith
- [2] Alibaba Cloud. qwen3.8-max Model Info. Model Studio documentation, 2026. URLhttps://www. alibabacloud.com/help/en/model-studio/qwen3-8-max. Accessed November 7, 2026. 20 on Pith
- [3] Maksym Andriushchenko, Alexandra Souly, Mateusz Dziemian, Derek Duenas, Maxwell Lin, Justin Wang, et al. AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents. arXiv:2410.09024, 2024. URLhttps://arxiv.org/abs/2410.09024 on PithPith's reviewarXiv
- [4] Benjamin Arnav, Pablo Bernabeu-Pérez, Nathan Helm-Burger, Tim Kostolansky, Hannes Whit- tingham, and Mary Phuong. CoT Red-Handed: Stress Testing Chain-of-Thought Monitoring. arXiv:2505.23575, 2025. URLhttps://arxiv.org/abs/2505.23575 on PitharXiv
- [5] Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, et al. LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding. arXiv:2308.14508, 2023. URL https://arxiv.org/abs/2308.14508 on PithPith's reviewarXiv
- [6] Bowen Baker, Joost Huizinga, Leo Gao, Zehao Dou, Melody Y. Guan, Aleksander Madry, et al. Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation. arXiv:2503.11926, 2025. URLhttps://arxiv.org/abs/2503.11926 on PithPith's reviewarXiv
- [7] Daniel Ben-Levi, Judah Goldfeder, Mateusz Dziemian, and Raja Moreno. Back to Basics: Formal Analysis Hints Improve Weak-to-Strong Monitoring. arXiv:2610.34064, 2026. URLhttps://arxiv. org/abs/2610.34064 on PithPith's reviewarXiv
- [8] Shiqi Chen, Yiran Zhao, Jinghan Zhang, I-Chun Chern, Siyang Gao, Pengfei Liu, et al. FELM: Benchmarking Factuality Evaluation of Large Language Models. arXiv:2310.00741, 2023. URL https://arxiv.org/abs/2310.00741 on PithPith's reviewarXiv
- [9] Sizhe Chen, Julien Piet, Chawin Sitawarin, and David Wagner. StruQ: Defending Against Prompt Injection with Structured Queries. arXiv:2402.06363, 2024. URLhttps://arxiv.org/abs/2402.063 63 on PithPith's reviewarXiv
- [10] Zhaorun Chen, Zhen Xiang, Chaowei Xiao, Dawn Song, and Bo Li. AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases. arXiv:2407.12784, 2024. URLhttps: //arxiv.org/abs/2407.12784 on PithPith's reviewarXiv
- [11] Zhaorun Chen, Mintong Kang, and Bo Li. ShieldAgent: Shielding Agents via Verifiable Safety Policy Reasoning. arXiv:2503.22738, 2025. URLhttps://arxiv.org/abs/2503.22738 on PitharXiv
- [12] Jesse Davis and Mark Goadrich. The Relationship Between Precision-Recall and ROC Curves. In Proceedings of the 23rd International Conference on Machine Learning, pp. 233–240, 2006. doi: 10.1145/ 1143844.1143874. URLhttps://doi.org/10.1145/1143844.1143874 on Pithdoi:10.1145/1143844.1143874
- [13] Edoardo Debenedetti, Jie Zhang, Mislav Balunovi´ c, Luca Beurer-Kellner, Marc Fischer, and Florian Tramèr. AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents. arXiv:2406.13352, 2024. URLhttps://arxiv.org/abs/2406.13352 on PithPith's reviewarXiv
- [14] DeepSeek-AI. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. arXiv:2606.19348, 2026. URLhttps://arxiv.org/abs/2606.19348 on PitharXiv
- [15] DeepSeek-AI. DeepSeek-V4.1-Flash-1017: Wish-Driven Programming Made Real. Official model release, 2026. URLhttps://api-docs.deepseek.com/news/news261017/. Accessed November 7, 2026 on Pith
- [16] Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, et al. Chain-of-Verification Reduces Hallucination in Large Language Models. arXiv:2309.11495, 2023. URLhttps://arxiv.org/abs/2309.11495 on PithPith's reviewarXiv
- [17] Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H. Laradji, Manuel Del Verme, Tom Marty, et al. WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks? arXiv:2403.07718, 2024. URLhttps://arxiv.org/abs/2403.07718 on PithPith's reviewarXiv
- [18] Tom Fawcett. An introduction to ROC analysis.Pattern Recognition Letters, 27(8):861–874, 2006. doi: 10.1016/j.patrec.2005.10.010. URLhttps://doi.org/10.1016/j.patrec.2005.10.010 on Pithdoi:10.1016/j.patrec.2005.10.010
- [19] Ryan Greenblatt, Buck Shlegeris, Kshitij Sachan, and Fabien Roger. AI Control: Improving Safety Despite Intentional Subversion. arXiv:2312.06942, 2023. URLhttps://arxiv.org/abs/2312.06942 on PithPith's reviewarXiv
- [20] Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. Not what you’ve signed up for: Compromising Real-World LLM-Integrated Applications 21 with Indirect Prompt Injection. arXiv:2302.12173, 2023. URLhttps://arxiv.org/abs/2302.12173 on PithPith's reviewarXiv
show 50 more
- [21] Melody Y. Guan, Miles Wang, Micah Carroll, Zehao Dou, Annie Y. Wei, Marcus Williams, et al. Monitoring Monitorability. arXiv:2512.18311, 2025. URLhttps://arxiv.org/abs/2512.18311 on PitharXiv
- [23] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, et al. SWE- bench: Can Language Models Resolve Real-World GitHub Issues? arXiv:2310.06770, 2023. URL https://arxiv.org/abs/2310.06770 on PithPith's reviewarXiv
- [24] Monika Jotautait ˙e, Maria Angelica Martinez, Ollie Matthews, and Tyler Tracy. MonitoringBench: Semi-Automated Red-Teaming for Agent Monitoring. arXiv:2605.09684, 2026. URLhttps://arxi v.org/abs/2605.09684 on PithPith's reviewarXiv
- [25] Neil Kale, Chen Bo Calvin Zhang, Kevin Zhu, Ankit Aich, Paula Rodriguez, Scale Red Team, et al. Reliable Weak-to-Strong Monitoring of LLM Agents. arXiv:2508.19461, 2025. URLhttps://arxiv. org/abs/2508.19461 on PithPith's reviewarXiv
- [26] Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, et al. Prometheus: Inducing Fine-grained Evaluation Capability in Language Models. arXiv:2310.08491, 2023. URL https://arxiv.org/abs/2310.08491 on PithPith's reviewarXiv
- [27] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large Language Models are Zero-Shot Reasoners. arXiv:2205.11916, 2022. URLhttps://arxiv.org/abs/ 2205.11916 on PithPith's reviewarXiv
- [28] Jonathan Kutasov, Yuqi Sun, Paul Colognese, Teun van der Weij, Linda Petrini, Chen Bo Calvin Zhang, et al. SHADE-Arena: Evaluating Sabotage and Monitoring in LLM Agents. arXiv:2506.15740, 2025. URLhttps://arxiv.org/abs/2506.15740 on PithPith's reviewarXiv
- [29] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, et al. Efficient Memory Management for Large Language Model Serving with PagedAttention. arXiv:2309.06180, 2023. URLhttps://arxiv.org/abs/2309.06180 on PithPith's reviewarXiv
- [30] Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Her- nandez, et al. Measuring Faithfulness in Chain-of-Thought Reasoning. arXiv:2307.13702, 2023. URLhttps://arxiv.org/abs/2307.13702 on PithPith's reviewarXiv
- [31] Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. HaluEval: A Large- Scale Hallucination Evaluation Benchmark for Large Language Models. arXiv:2305.11747, 2023. URLhttps://arxiv.org/abs/2305.11747 on PithPith's reviewarXiv
- [33] Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, et al. Let’s Verify Step by Step. arXiv:2305.20050, 2023. URLhttps://arxiv.org/abs/2305.20050 on PithPith's reviewarXiv
- [34] Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring How Models Mimic Hu- man Falsehoods. arXiv:2109.07958, 2021. URLhttps://arxiv.org/abs/2109.07958 on PithPith's reviewarXiv
- [35] Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, et al. Lost in the Middle: How Language Models Use Long Contexts. arXiv:2307.03172, 2023. URL https://arxiv.org/abs/2307.03172 on PithPith's reviewarXiv
- [36] Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, et al. AgentBench: Evaluating LLMs as Agents. arXiv:2308.03688, 2023. URLhttps://arxiv.org/abs/2308.03688 on PithPith's reviewarXiv
- [37] Hanjun Luo, Shenyu Dai, Chiming Ni, Xinfeng Li, Guibin Zhang, Kun Wang, et al. AgentAuditor: Human-Level Safety and Security Evaluation for LLM Agents. arXiv:2506.00641, 2025. URLhttps: //arxiv.org/abs/2506.00641 on PitharXiv
- [38] Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas 22 Scialom. GAIA: a benchmark for General AI Assistants. arXiv:2311.12983, 2023. URLhttps: //arxiv.org/abs/2311.12983 on PithPith's reviewarXiv
- [39] Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen tau Yih, Pang Wei Koh, et al. FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation. arXiv:2305.14251, 2023. URLhttps://arxiv.org/abs/2305.14251 on PithPith's reviewarXiv
- [40] Elle Najt, Colin Toft, Tyler Tracy, Fabien Roger, and Joe Benton. SLEIGHT-Bench: A Benchmark of Evasion Attacks Against Agent Monitors. arXiv:2605.16626, 2026. URLhttps://arxiv.org/abs/ 2605.16626 on PithPith's reviewarXiv
- [41] OpenAI. GPT-6 Luna Model. API model documentation, 2026. URLhttps://developers.openai. com/api/docs/models/gpt-6-luna. Accessed November 7, 2026 on Pith
- [42] OpenAI. GPT-6 Sol Model. API model documentation, 2026. URLhttps://developers.openai.co m/api/docs/models/gpt-6-sol. Accessed November 7, 2026 on Pith
- [43] Arjun Panickssery, Samuel R. Bowman, and Shi Feng. LLM Evaluators Recognize and Favor Their Own Generations. arXiv:2404.13076, 2024. URLhttps://arxiv.org/abs/2404.13076 on PithPith's reviewarXiv
- [44] Asif Pinjari and Mithun Paul Saint-Germain. AgentDrift: A Step-Labeled Benchmark of Injection- Hijacked LLM Agent Trajectories. arXiv:2609.06972, 2026. URLhttps://arxiv.org/abs/2609.069 72 on PithPith's reviewarXiv
- [45] Aniruddh Pramod, James Oldfield, and Adel Bibi. Towards a Unified Misuse Monitoring Bench- mark. arXiv:2610.07089, 2026. URLhttps://arxiv.org/abs/2610.07089 on PithPith's reviewarXiv
- [46] Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, et al. ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs. arXiv:2307.16789, 2023. URLhttps: //arxiv.org/abs/2307.16789 on PithPith's reviewarXiv
- [47] Yangjun Ruan, Honghua Dong, Andrew Wang, Silviu Pitis, Yongchao Zhou, Jimmy Ba, et al. Iden- tifying the Risks of LM Agents with an LM-Emulated Sandbox. arXiv:2309.15817, 2023. URL https://arxiv.org/abs/2309.15817 on PithPith's reviewarXiv
- [48] Takaya Saito and Marc Rehmsmeier. The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets.PLOS ONE, 10(3):e0118432, 2015. doi: 10.1371/journal.pone.0118432. URLhttps://journals.plos.org/plosone/article?id=10.13 ... on Pithdoi:10.1371/journal.pone.0118432
- [49] Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, et al. Toolformer: Language Models Can Teach Themselves to Use Tools. arXiv:2302.04761, 2023. URLhttps://arxiv.org/abs/2302.04761 on PithPith's reviewarXiv
- [50] scikit-learn developers. average_precision_score. scikit-learn documentation, 2026. URLhttps:// scikit-learn.org/stable/modules/generated/sklearn.metrics.average_precision_score.html. Accessed November 7, 2026 on Pith
- [51] SpaceXSI. Introducing Grok 4.6. Official model release, 2026. URLhttps://x.ai/news/grok-4-6. Accessed November 7, 2026 on Pith
- [52] Mikhail Terekhov, Alexander Panfilov, Daniil Dzenhaliou, Caglar Gulcehre, Maksym An- driushchenko, Ameya Prabhu, et al. Adaptive Attacks on Trusted Monitors Subvert AI Control Protocols. arXiv:2510.09462, 2025. URLhttps://arxiv.org/abs/2510.09462 on PitharXiv
- [53] Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman. Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting. arXiv:2305.04388, 2023. URLhttps://arxiv.org/abs/2305.04388 on PithPith's reviewarXiv
- [54] Eric Wallace, Kai Xiao, Reimar Leike, Lilian Weng, Johannes Heidecke, and Alex Beutel. The In- struction Hierarchy: Training LLMs to Prioritize Privileged Instructions. arXiv:2404.13208, 2024. URLhttps://arxiv.org/abs/2404.13208 on PithPith's reviewarXiv
- [55] Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, et al. Large Language Models are not Fair Evaluators. arXiv:2305.17926, 2023. URLhttps://arxiv.org/abs/2305.17926 on PithPith's reviewarXiv
- [57] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, et al. Self- Consistency Improves Chain of Thought Reasoning in Language Models. arXiv:2203.11171, 2022. URLhttps://arxiv.org/abs/2203.11171 on PithPith's reviewarXiv
- [58] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, et al. Chain- of-Thought Prompting Elicits Reasoning in Large Language Models. arXiv:2201.11903, 2022. URL https://arxiv.org/abs/2201.11903 on PithPith's reviewarXiv
- [59] URLhttps://arxiv.org/abs/2407.16741 on PithPith's reviewarXiv
- [60] Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, et al. OS- World: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. arXiv:2404.07972, 2024. URLhttps://arxiv.org/abs/2404.07972 on PithPith's reviewarXiv
- [61] Frank F. Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, et al. TheAgent- Company: Benchmarking LLM Agents on Consequential Real World Tasks. arXiv:2412.14161, 2024. URLhttps://arxiv.org/abs/2412.14161 on PithPith's reviewarXiv
- [62] Zhen Xiang, Linzhi Zheng, Yanjie Li, Junyuan Hong, Qinbin Li, Han Xie, et al. GuardAgent: Safe- guard LLM Agents by a Guard Agent via Knowledge-Enabled Reasoning. arXiv:2406.09187, 2024. URLhttps://arxiv.org/abs/2406.09187 on PithPith's reviewarXiv
- [63] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, et al. ReAct: Synergizing Reasoning and Acting in Language Models. arXiv:2210.03629, 2022. URLhttps: //arxiv.org/abs/2210.03629 on PithPith's reviewarXiv
- [65] John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, et al. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. arXiv:2405.15793, 2024. URLhttps://arxiv.org/abs/2405.15793 on PithPith's reviewarXiv
- [66] Qiusi Zhan, Zhixiang Liang, Zifan Ying, and Daniel Kang. InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents. arXiv:2403.02691, 2024. URL https://arxiv.org/abs/2403.02691 on PithPith's reviewarXiv
- [67] Zhexin Zhang, Shiyao Cui, Yida Lu, Jingzhuo Zhou, Junxiao Yang, Hongning Wang, et al. Agent- SafetyBench: Evaluating the Safety of LLM Agents. arXiv:2412.14470, 2024. URLhttps://arxiv. org/abs/2412.14470 on PithPith's reviewarXiv
- [68] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv:2306.05685, 2023. URL https://arxiv.org/abs/2306.05685 on PithPith's reviewarXiv
- [69] Tongxin Yuan, Zhiwei He, Lingzhong Dong, Yiming Wang, Ruijie Zhao, Tian Xia, et al. R-Judge: Benchmarking Safety Risk Awareness for LLM Agents. arXiv:2401.10019, 2024. URLhttps://arxi v.org/abs/2401.10019 on PithPith's reviewarXiv
- [70] Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, et al. WebArena: A Realistic Web Environment for Building Autonomous Agents. arXiv:2307.13854, 2023. URL https://arxiv.org/abs/2307.13854. A Dataset Details and Complete Cases Appendix A reports dataset... on PithPith's reviewarXiv
- [73] Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, et al. SGLang: Efficient Execution of Structured Language Model Programs. arXiv:2312.07104, 2023. URLhttps://arxiv.org/abs/2312.07104 on PithPith's reviewarXiv
- [2023] URLhttps://arxiv.org/abs/2312.14197 on PithPith's reviewarXiv
- [2024] URLhttps://arxiv.org/abs/2404.06654 on PithPith's reviewarXiv
- [2026] URLhttps://arxiv.org/abs/2604.02022 on PithPith's reviewarXiv
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment