Pith.

Machine Learning · arXiv:2609.17930 v1 · submitted 15 Sep 2026 · 26 references

Locating Hidden Failures Makes Long-Horizon Agents More Reliable

Salman Rahman, Yubin Kim, Mihir Parmar, A. Ali Heydari, Genglin Liu, Simon A. Lee, Weizhi Zhang, Arian Hosseini

show 10 more authors

Ahmed A. Metwally, Yuzhe Yang, Baharan Mirzasoleiman, Xin Liu, Pavel Izmailov, Saadia Gabriel, Mark Malhotra, Shwetak Patel, Daniel McDuff, Hamid Palangi

The paper says

“As AI agents take on long, autonomous tasks, we increasingly oversee rather than perform the work, yet we still judge them almost entirely by whether they finally succeed.”

more of the abstract

An outcome cannot reveal where a run went wrong, whether the agent recovered, or the irreversible harm it caused along the way, and where long-horizon agents fail remains unmapped. We study 2518 agent trajectories across software engineering, computer use, and science, close to real deployment, and classify 6967 mistakes into 78 failure types. Failure follows a recurring signature: after its first mistake an agent often fails to recover and rarely catches the error itself, so the run continues unchecked while still looking correct; whether an agent recovers depends on the task and the environment's feedback, not on the agent framework running it. Long-horizon agents can do real harm on the way to a passing result: even runs scored as solved delete data, corrupt systems, or fabricate success rather than earning it. We release these human-verified annotations as Traverse, a benchmark on which six frontier judges struggle to locate failure regardless of scale: even the strongest correctly identifies the first mistake in fewer than a third of runs. Yet Scout, a 4B verifier we trained, locates failure far better than these judges and transfers to domains it never saw. Used at test time to select among an agent's candidate runs, it raises task success above the agent's own single-attempt performance, without retraining the agent. By making failure cheap to locate and correct, this work is a foundation for more trustworthy long-horizon agents that learn from their own mistakes, and a practical path to overseeing increasingly autonomous AI.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 17 Sep 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up. Thorough step-level failure analysis with a trained small verifier that outperforms frontier models, though the human annotation dependency and modest SWE gain warrant cautious interpretation. This line is wrong

3 major objections · 5 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 26 references say.

The paper says, at Abstract, Section 3.3

The review did not quote this passage.

Pith says: Objection 1 of 3 · major

The claim that Scout 'locates failure far better' than frontier judges is not accompanied by confidence intervals for the key metrics (exact match, MAE, test-time selection gain). On SWE-Bench Verified, the test-time improvement over the next-best verifier is only 0.6 pp (78.0% vs. 77.4%); without bootstrapped CIs or a significance test, this difference may not be statistically significant. The authors should provide uncertainty estimates for all main comparisons to support the strength of the claims.

The paper says, at Section 2.2, Limitations in Section 4

The review did not quote this passage.

Pith says: Objection 2 of 3 · major

The entire pipeline—failure characterization, Traverse benchmark, Scout training—rests on gold first-mistake labels produced by a single annotation process (though two annotators and a third resolver). Inter-annotator agreement of κ=0.73 on 'where' is substantial but not perfect, and 23% of failed runs have no localizable first mistake.

more of objection 2

The paper does not test whether Scout's relative advantage over frontier models is stable under alternative annotation rubrics or a separate pool of annotators. A sensitivity analysis (e.g., evaluating Scout against the secondary annotator's labels) would increase confidence that the advantage is not an artifact of annotation style.

The paper says, at Table 1, Section 3.2

The review did not quote this passage.

Pith says: Objection 3 of 3 · major

Table 1 reports localization accuracy for frontier models but does not include Scout's performance on the same held-out test set (258 trajectories). Since the central claim is that Scout is superior, the main comparison table should include Scout's exact match, ±1, ±3, and MAE on SWE-bench and TerminalBench (and BixBench for transfer) to allow direct reading.

5 minor objections

The paper says, at Abstract, Section 3.3

Pith says: minor objection 1 of 5

The phrase 'far better' is imprecise; replace with specific effect sizes or 'substantially better' where supported by the data.

The paper says, at Section 2.4, Eq. (1)

Pith says: minor objection 2 of 5

The reward weighting coefficients 0.7 and 0.3 appear without justification. A brief justification or a sensitivity analysis would help the reader understand their impact.

show 3 more

The paper says, at Figure 3

Pith says: minor objection 3 of 5

Panel labels P1, P2, P4 could be more descriptive directly in the caption (e.g., P1: non-recovery, P2: non-detection, P4: persistence). Currently the caption uses 'P1, P2, P4' without defining them in the figure text.

The paper says, at Section 3.1

Pith says: minor objection 4 of 5

The 23% of runs classified as 'diffuse failure' (no localizable first mistake) are mentioned but not further characterized. A brief description (e.g., gradual accumulation of small errors) would give a more complete picture of failure modes.

The paper says, at Appendix Table 3

Pith says: minor objection 5 of 5

The greyed-out categories (10 never observed) are clearly marked, but the table is long. Consider moving the full taxonomy to supplementary and keeping a concise summary in the main text.

What would settle it

Commission an independent annotation team (different institution, fresh guidelines) to re-label the first mistake on a random subset of 200 Traverse trajectories (100 SWE-bench, 100 TerminalBench). Compute inter-annotator agreement with the original labels. Then evaluate Scout (SFT+RL) and the best frontier model (Gemini 3.1 Pro) on the new labels. If Scout's exact-match advantage over Gemini 3.1 Pro on the new labels is significantly smaller (e.g., more than halved) than on the original labels, then the reported advantage is sensitive to annotation bias. Additionally, compute bootstrapped 95% confidence intervals for all test-time selection success rates; if the 78.0% vs 77.4% difference is not significant (p>0.05), the SWE-bench claim should be tempered.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 17 Sep 2026

  1. Reader holds up confidence high
  2. Skeptic partly agrees 56 s · 29,992 tokens read, 5,833 written
  3. Editor wrote the verdict above 23 s · 30,953 tokens read, 2,230 written
  4. Referee minor revision: 3 major objections, 5 minor 31 s · 30,390 tokens read, 3,074 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

Hey [Colleague],

This paper delivers the goods. It is the first holistic, step-level account of how long-horizon agents actually fail across multiple realistic domains (software engineering, computer use, science). The taxonomy (78 failure types) and the failure signature—after the first mistake, agents rarely recover, rarely self-detect, and keep going—are genuinely new and well-supported by the data. The Traverse benchmark with human-annotated first mistakes is a practical tool for the field. And Scout, a 4B verifier trained on these annotations, convincingly outperforms much larger frontier models at locating the first mistake and improves test-time task success, especially on Terminal-Bench.

The soft spots are real but not disqualifying. The entire pipeline depends on human annotations of the "first mistake." Inter-annotator agreement is good (kappa 0.73 on where) but not perfect, and 23% of failures lack a localizable first mistake. The paper acknowledges this, but the stress-test concern—that Scout might internalize annotation biases that zero-shot frontier models do not share—is valid. The test-time gain on SWE-Bench Verified is only 0.6 percentage points (78.0% vs. 77.4%) with no confidence intervals reported, so that particular advantage may not be statistically significant. On Terminal-Bench, the gain is larger and more believable.

Still, the central claims hold: long-horizon failure is characterizable, frontier models struggle to locate it, and a small model can be trained to do it better. The paper is transparent about its limitations, releases its data and model, and the code appears replicable. This is a solid contribution that moves the conversation from outcome-only evaluation to process-level oversight.

Who this is for: anyone working on agent evaluation, safety, reward modeling, or test-time scaling. It should definitely be sent to peer review. I would cite it in my own work, and I'd bring it to a reading group focused on agent reliability.

Recommendation: accept with minor revisions. Encourage the authors to add confidence intervals for the test-time selection results and to include an analysis of annotation bias (e.g., how performance changes under an alternative annotation protocol or a separate annotator pool). The rest is strong enough as is.

The strongest objection

The entire pipeline—failure characterization, Traverse benchmark, Scout training, and evaluation—rests on gold first-mistake labels produced by human annotators. Inter-annotator agreement (κ=0.73 on where) is substantial but not perfect, and 23% of failures lack a localizable first mistake, indicating inherent ambiguity. The paper does not test whether the relative performance of Scout versus frontier models is stable under alternative annotation rubrics or a separate pool of annotators. If human labels systematically reflect a particular judgment style (e.g., flagging action-level errors earlier than causally prior reasoning errors), Scout's supervised training could internalize those biases, while zero-shot frontier models might make different (potentially more correct) judgments. The test-time selection improvement on SWE-Bench Verified is also small (78.0% vs. 77.4% for the next-best verifier, a 0.6 pp gap) and no confidence intervals are reported, so the gain may not be statistically significant. Together, these concerns weaken the central claim that Scout objectively locates failure 'far better' than frontier models and that the benefit for test-time selection is robust.

The reader identified the human-annotation assumption as the weakest. I agree that it is load-bearing, but I push further: the paper does not test for systematic bias in the annotations, which is a concrete threat to validity, and the small SWE-bench improvement may not be statistically significant. However, the paper acknowledges the 23% diffuse-failure limitation, uses multiple judges and ranges for sensitive quantities, and releases the data and model. The central characterization of failure and the demonstration that a small model can be trained to locate mistakes are still valuable even if the exact ranking vs. frontier models shifts under re-annotation. Hence the verdict remains ACCEPT; no change needed.

The referee's report

If the claims hold, this work is a substantial contribution to scalable oversight of long-horizon AI agents. The failure taxonomy is the first holistic step-level account across multiple deployment-relevant domains. Traverse fills a critical gap as the first benchmark combining long trajectories, naturally occurring failures, and human-verified first-mistake localization. The demonstration that a small specialized verifier can outperform much larger frontier models at failure localization, and that this translates into test-time gains, opens a practical path toward more reliable autonomous agents. The safety audit (irreversible actions even in solved runs) is a valuable addition beyond prior failure studies.

Pith's reading of the paper (a machine's summary, not the authors')

Long-horizon AI agents working autonomously across hundreds of steps fail in a recurring pattern: after committing their first mistake, they rarely recover (69.5% of the time), rarely detect the error (38.5%), and keep acting regardless (72.6%). This failure signature means a single early reasoning or planning error compounds into a cascade of downstream action mistakes, while the run ends looking correct. The paper shows that frontier models including GPT-5.5, Claude Opus 4.6, and Gemini 3.1 Pro cannot reliably identify this first mistake, succeeding on fewer than a third of long trajectories. Yet a 4B-parameter verifier, Scout, trained on human-annotated first mistakes, locates failure better than any frontier model and, when used to select among an agent's candidate trajectories, raises task success above the agent's own single-attempt performance (from 81.8% to 90.2% on Terminal-Bench). This establishes that failure localization is a learnable skill that a lightweight verifier can acquire, offering a practical path to more trustworthy long-horizon agents.

What carries the argument. The central objects are the first mistake (the step where a run first goes wrong, human-annotated with Cohen's $\kappa=0.73$) and the failure signature (non-recovery, non-detection, persistence) that characterizes long-horizon agent failure. The key mechanism is a trained verifier, Scout, a 4B-parameter model fine-tuned on step-level correctness labels using supervised learning and GRPO reinforcement learning. Its reward design uses the product of recall on correct and incorrect steps ($R_c \cdot R_i$) to avoid the degenerate strategies of over-flagging or under-flagging that plague frontier judges.

Core claim. The paper's central discovery is that long-horizon agent failure is not a random accumulation of errors but follows a systematic and measurable signature: after the first mistake, agents recover in only 30.5% of runs, detect the error in only 38.5%, and persist in acting in 72.6%. Errors cascade, making a step 11\u201314 times more likely to be wrong after an incorrect step. The first mistake is typically a reasoning or planning fault, while the more numerous action mistakes are downstream symptoms. Frontier models, even the strongest (GPT-5.5, Gemini 3.1 Pro), locate this first mistake on fewer than a third of long trajectories, and all are systematically biased toward over- or under-flagging. However, a 4B-parameter verifier (Scout) trained via supervised fine-tuning and reinforcement learning on human-annotated first mistakes outperforms every frontier model on localization, transfers to an unseen scientific domain, and improves agent task success through best-of-N selection without retraining the agent.

The premise it rests on. The paper assumes that most failed runs have a single decisive first mistake that can be reliably identified by human annotators; it explicitly notes that 23% of failed runs have no localizable first mistake (diffuse failure), so the first-mistake analysis covers only a subset of failures.

If the paper is right (Pith's inferences, not the authors')

Agent training pipelines can use first-mistake localization as a dense reward signal for process-level reinforcement learning, replacing sparse outcome rewards.

Test-time selection with a small verifier can improve agent performance without retraining the agent, as shown by Scout's best-of-N gains on Terminal-Bench and SWE-Bench.

Safety audits of long-horizon agents should monitor the first mistake as the critical intervention point, since destructive actions often follow early errors.

Benchmarks for agent oversight should measure first-mistake localization accuracy, not just outcome pass rates, because frontier judges are biased at the process level.

The failure signature implies that building agents with built-in self-detection and recovery mechanisms could be more effective than reducing their raw error rate.

If failure localization can be learned by a 4B model, it could be distilled into the agent itself as a self-verification module, enabling agents to catch and correct their own mistakes mid-run without an external verifier.

The finding that 23% of failed runs have no localizable first mistake suggests a distinct failure mode (diffuse, gradual degradation) that may require a different oversight strategy, such as continuous monitoring of trajectory health rather than single-point detection.

The safety audit's identification of destructive actions in solved runs implies that deployment of long-horizon agents in high-stakes settings should mandate process-level auditing and not rely on outcome verification alone.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

As AI agents take on long, autonomous tasks, we increasingly oversee rather than perform the work, yet we still judge them almost entirely by whether they finally succeed. An outcome cannot reveal where a run went wrong, whether the agent recovered, or the irreversible harm it caused along the way, and where long-horizon agents fail remains unmapped. We study 2518 agent trajectories across software engineering, computer use, and science, close to real deployment, and classify 6967 mistakes into 78 failure types. Failure follows a recurring signature: after its first mistake an agent often fails to recover and rarely catches the error itself, so the run continues unchecked while still looking correct; whether an agent recovers depends on the task and the environment's feedback, not on the agent framework running it. Long-horizon agents can do real harm on the way to a passing result: even runs scored as solved delete data, corrupt systems, or fabricate success rather than earning it. We release these human-verified annotations as Traverse, a benchmark on which six frontier judges struggle to locate failure regardless of scale: even the strongest correctly identifies the first mistake in fewer than a third of runs. Yet Scout, a 4B verifier we trained, locates failure far better than these judges and transfers to domains it never saw. Used at test time to select among an agent's candidate runs, it raises task success above the agent's own single-attempt performance, without retraining the agent. By making failure cheap to locate and correct, this work is a foundation for more trustworthy long-horizon agents that learn from their own mistakes, and a practical path to overseeing increasingly autonomous AI.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-09-23

The authors' main claim, as the classifier picked it out 0.97

“By making failure cheap to locate and correct, this work is a foundation for more trustworthy long-horizon agents that learn from their own mistakes, and a practical path to overseeing increasingly autonomous AI.”

How the main claim is stated: asserted 0.93

What the abstract itself states, as probabilities: claims first 0.08 · states limitation 0.64 · human participants 0.36 · controlled comparison 0.75 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.93 · mentions code or data availability 0.83

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
As AI agents take on long, autonomous tasks, we increasingly oversee rather than perform the work, yet we still judge them almost entirely by whether they finally succeed. background 0.99 0.07 0.03 asserted 1.00
An outcome cannot reveal where a run went wrong, whether the agent recovered, or the irreversible harm it caused along the way, and where long-horizon agents fail remains unmapped. gap 0.88 0.09 0.07 asserted 1.00
We study $2518$ agent trajectories across software engineering, computer use, and science, close to real deployment, and classify $6967$ mistakes into $78$ failure types. method 0.70 0.97 0.04 asserted 1.00
Failure follows a recurring signature: after its first mistake an agent often fails to recover and rarely catches the error itself, so the run continues unchecked while still looking correct; whether an agent recovers depends on the task and the environment's feedback, not on the agent framework running it. result 0.97 0.90 0.20 asserted 0.90
Long-horizon agents can do real harm on the way to a passing result: even runs scored as solved delete data, corrupt systems, or fabricate success rather than earning it. result 0.96 0.79 0.08 asserted 0.71
We release these human-verified annotations as Traverse, a benchmark on which six frontier judges struggle to locate failure regardless of scale: even the strongest correctly identifies the first mistake in fewer than a third of runs. contribution 0.66 0.92 0.11 asserted 1.00
Yet Scout, a $4$B verifier we trained, locates failure far better than these judges and transfers to domains it never saw. contribution 0.43 0.95 0.16 asserted 1.00
Used at test time to select among an agent's candidate runs, it raises task success above the agent's own single-attempt performance, without retraining the agent. result 0.97 0.92 0.18 asserted 1.00
By making failure cheap to locate and correct, this work is a foundation for more trustworthy long-horizon agents that learn from their own mistakes, and a practical path to overseeing increasingly autonomous AI. interpretation 0.91 0.45 0.73 asserted 0.93

What kind of paper the classifier reports this is: dataset or benchmark 0.80 · evidence from observation or experiment 0.98

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 7,135 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

References (26)
  1. [4] H. Chen, D. Metelski, L. Qi, T. Xia, J. Lee, S. Brown, K. Riley, F. Wang, T. Liu, H. C. MD, et al. Chi-bench: Can ai agents automate end-to-end, long-horizon, policy-rich healthcare workflows?arXiv preprint arXiv:2605.16679, on PithPith's reviewarXiv
  2. [5] https://frontierswe.com/blog. X. Deng, J. Da, E. Pan, Y. Y. He, C. Ide, K. Garg, N. Lauffer, A. Park, N. Pasari, C. Rane, et al. Swe-bench pro: Can ai agents solve long-horizon software engineering tasks?arXiv preprint arXiv:2509.16941, on PithPith's reviewarXiv
  3. [7] Accessed: 2026-06-23. R. Geirhos, J.-H. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge, and F. A. Wichmann. Shortcut learning in deep neural networks.Nature Machine Intelligence, 2(11):665–673, on Pith
  4. [8] URLhttps: //arxiv.org/abs/2602.15763. 20 Locating Hidden Failures Makes Long-Horizon Agents More Reliable J. Gottweis, W.-H. Weng, A. Daryin, T. Tu, P. Sirkovic, A. Myaskovsky, G. Glowaty, F. Weissenberger, A. Orlandi, D. Popovici, et al. Accelerating scientific discovery with... on PithPith's reviewarXiv
  5. [9] R. Greenblatt, C. Denison, B. Wright, F. Roger, M. MacDiarmid, S. Marks, J. Treutlein, T. Be- lonax, J. Chen, D. Duvenaud, et al. Alignment faking in large language models.arXiv preprint arXiv:2412.14093, on PithPith's reviewarXiv
  6. [10] C.E.Jimenez, J.Yang, A.Wettig, S.Yao, K.Pei, O.Press, andK.Narasimhan. Swe-bench: Canlanguage models resolve real-world github issues? InInternational Conference on Learning Representations, volume 2024, pages 54107–54157, on Pith
  7. [11] Notion Blog. D. Li, Y. Yao, Z. Tan, H. Liu, and R. Guo. Toolprmbench: Evaluating and advancing process reward models for tool-using agents.arXiv preprint arXiv:2601.12294, 2026a. H. Li, Y. Yao, L. Zhu, R. Feng, H. Ye, J. Wang, Y. He, P. Zou, L. Zhang, X. Lei, et al. Codetracer... on PitharXiv
  8. [12] H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe. Let’s verify step by step. InInternational Conference on Learning Representations, volume 2024, pages 39578–39601, on Pith
  9. [13] 21 Locating Hidden Failures Makes Long-Horizon Agents More Reliable X. H. Lù, A. Kazemnejad, N. Meade, A. Patel, D. Shin, A. Zambrano, K. Stańczak, P. Shaw, C. J. Pal, and S. Reddy. Agentrewardbench: Evaluating automatic evaluations of web agent trajectories.arXiv preprint arX... on PitharXiv
  10. [14] Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long.857. URLhttps://aclanthology.org/2025.acl-long.857/. M. A. Merrill, A. G. Shaw, N. Carlini, B. Li, H. Raj, I. Bercovich, L. Shi, J. Y. Shin, T. Walshe, E. K. Buchanan, et al. Term... on PithPith's reviewarXiv
  11. [15] L. Mitchener, J. M. Laurent, A. Andonian, B. Tenmann, S. Narayanan, G. P. Wellawatte, A. White, L. Sani, and S. G. Rodriques. Bixbench: a comprehensive benchmark for llm-based agents in computational biology.arXiv preprint arXiv:2503.00096, on PitharXiv
  12. [17] URLhttps://arxiv.org/abs/ 2603.08640. M. S. Rashid, C. Bock, Y. Zhuang, A. Buchholz, T. Esler, S. Valentin, L. Franceschi, M. Wistuba, P. T. Sivaprasad, W. J. Kim, et al. Swe-polybench: A multi-language benchmark for repository level evaluation of coding agents.arXiv preprint ... on PitharXiv
  13. [18] J. Scheurer, M. Balesni, and M. Hobbhahn. Large language models can strategically deceive their users when put under pressure.arXiv preprint arXiv:2311.07590, on PithPith's reviewarXiv
  14. [19] 22 Locating Hidden Failures Makes Long-Horizon Agents More Reliable Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. Deepseek- math: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, on PithPith's reviewarXiv
  15. [20] G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, R. Zhang, Y. Peng, H. Lin, and C. Wu. Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv: 2409.19256, on PithPith's reviewarXiv
  16. [21] N. Singhi, H. Bansal, A. Hosseini, A. Grover, K.-W. Chang, M. Rohrbach, and A. Rohrbach. When to solve, when to verify: Compute-optimal problem solving and generative verification for llm reasoning.arXiv preprint arXiv:2504.01005, on PitharXiv
  17. [22] Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long.1230. URL https:// aclanthology.org/2025.acl-long.1230/. Y. Sun, X. Han, W. Zhang, Y. Pang, T. Wang, Y. Cao, Y. Huang, C. Duroiu, H. Zhang, J. Lin, et al. Agents’ last exam.arXiv ... on PitharXiv
  18. [23] URLhttps://arxiv.org/abs/2505.09388. B. Vidgen, A. Mann, A. Fennelly, J. W. Stanly, L. Rothman, M. Burstein, J. Benchek, D. Ostrofsky, A. Ravichandran, D. Sur, et al. Apex-agents.arXiv preprint arXiv:2601.14242, on PithPith's reviewarXiv
  19. [24] X. J. Wang, H. Bai, Y. Sun, H. Wang, S. Zhang, W. Hu, M. Schroder, B. Mutlu, D. Song, and R. D. Nowak. The long-horizon task mirage? diagnosing where and why agentic systems break.arXiv preprint arXiv:2604.11978, on PithPith's reviewarXiv
  20. [25] L. Zhang, A. Hosseini, H. Bansal, S. M. Kazemi, A. Kumar, and R. Agarwal. Generative verifiers: Reward modeling as next-token prediction. InInternational Conference on Learning Representations, volume 2025, pages 12476–12505, on Pith
show 6 more
  1. [26] K. Zhu, Z. Liu, B. Li, M. Tian, Y. Yang, J. Zhang, P. Han, Q. Xie, F. Cui, W. Zhang, et al. Where llm agents fail and how they can learn from failures.arXiv preprint arXiv:2509.25370, on PitharXiv
  2. [2022] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, on Pith
  3. [2023] S. R. Bowman, J. Hyun, E. Perez, E. Chen, C. Pettit, S. Heiner, K. Lukoši¯ut˙e, A. Askell, A. Jones, 19 Locating Hidden Failures Makes Long-Horizon Agents More Reliable A. Chen, et al. Measuring progress on scalable oversight for large language models.arXiv preprint arXiv:2211.03540, on PithPith's reviewarXiv
  4. [2024] S. Rahman, S. Gorantla, A. Gupta, S. Roy, N. Peng, and Y. Liu. Spark: Stepwise process-aware rewards for reference-free reinforcement learning.arXiv preprint arXiv:2512.03244, on PithPith's reviewarXiv
  5. [2025] D. Deshpande, V. Gangal, H. Mehta, J. Krishnan, A. Kannappan, and R. Qian. Trail: Trace reasoning and agentic issue localization.arXiv preprint arXiv:2505.08638, on PithPith's reviewarXiv
  6. [2026] Accessed: 2026-06-22. Y. Bengio, G. Hinton, A. Yao, D. Song, P. Abbeel, T. Darrell, Y. N. Harari, Y.-Q. Zhang, L. Xue, S. Shalev- Shwartz, et al. Managing extreme ai risks amid rapid progress.Science, 384(6698):842–845, on Pith
Cited by 5 Pith papers

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment