Pith.

Machine Learning · arXiv:2610.02994 v1 · submitted 2 Oct 2026 · 32 references

Sentry: Learning to Recover from LLM Agent Failures at Test Time

Changxiu Ji, Amy Lu, Qizheng Zhang, Kunle Olukotun

The paper says

“LLM agents often fail mid-task due to invalid tool calls, repeated actions, or poorly grounded reasoning, and learning from these failures is a path to reliability.”

more of the abstract

We find that how failure knowledge reaches the agent matters as much as what it contains. Failure lessons are conditional: kept in the agent's context, they misfire when their failure is absent, and removing them from an evolving playbook improves performance. Runtime interventions, in contrast, act only when a failure occurs but do not learn from their repairs. We argue that failure knowledge is conditional knowledge and should be conditionally exposed, and instantiate this principle in Sentry, a failure-management layer that runs alongside the agent. When Sentry detects a failure, it retrieves matching lessons from an external playbook to guide recovery, verifies without access to task rewards whether the agent recovered, and stores a new lesson only if it did; the full playbook never enters the agent's context. Across multiple agentic benchmarks, Sentry outperforms the strongest runtime-intervention baseline on every benchmark, by 37% on average, and the strongest context-evolution baseline by 39% on the two benchmarks where both are evaluated; combining Sentry with context evolution yields further gains. Learned lessons transfer to held-out tasks, and controlled experiments show that exposing the full playbook to the agent lowers performance even when relevant lessons remain available on demand.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 5 Oct 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. A genuinely careful empirical paper whose conditional-exposure claim mostly holds up; the load-bearing soft spot is the self-judging verifier with unmeasured error direction, and the gains are more WebShop-heavy than the abstract suggests. This line is wrong

2 major objections · 4 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 32 references say.

The paper says, at §3.3, §5.2, Appendix A.7

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.76):

“Sentry uses the same model for the task agent, detector, and verifier; although its judgments largely agree with human annotators, decoupled detector and verifier 9 models remain to be studied.”

Pith says: Objection 1 of 2 · major

The recovery verifier that gates every playbook admission and every recovery count (81.7%, Figure 3) is the same Qwen3.5-9B model that generated the repair being judged and that diagnosed the failure; the paper explicitly acknowledges the shared-model design as a limitation ([b149]).

more of objection 1

The human audit reports 81.8% agreement over only 99 soft-repair windows with both annotators drawn from the research team that designed the verifier rubric, and it reports agreement rates but not error direction. If the 18% of disagreements skew toward false 'resolved', a non-trivial fraction of admitted lessons come from episodes that did not genuinely recover — exactly the 'lesson from a failed repair misfires later' failure mode the paper motivates its design against — and these junk lessons then enter the frozen playbook used in Table 4 and the held-out transfer results. This is load-bearing for the transfer and recovery-rate claims, so the manuscript should add an error-direction analysis of the disagreements, ideally validated against an independent recovery signal such as downstream reward where available, or a decoupled verifier model on a held-out subset.

The paper says, at §2, §4.3

The review did not quote this passage.

Pith says: Objection 2 of 2 · major

The paper's central finding that 'failure lessons are conditional' rests on the pruning experiment: removing failure-mode lessons from ACE's playbook improves held-out performance from 0.335 to 0.391 on WebShop and from 0.562 to 0.643 on Mind2Web Replay. The provided text never specifies how 'failure-mode lessons' were identified in ACE's playbook, which lessons were removed, how many were removed, or the …

more of objection 2

… exact pruning rule; the only reference is to the held-out protocol of §4.3, which describes task splits but not the pruning procedure. Because this experiment is the empirical basis for the principle that the entire method instantiates, the procedure needs to be specified precisely so the finding can be reproduced and audited.

4 minor objections

The paper says, at Appendix A.5.1, Tables 6–9

Pith says: minor objection 1 of 4

Several table cells appear to have merged columns or missing separators, e.g., '7922793' in Table 6, '38.53±0.87 18766 19.61' in Table 7, '0.30±0.01 90 95' in Table 8, and '55435 770.410' in Table 9; these should be reformatted so the intervention counts and guided-task counts are legible.

The paper says, at §5.1, Tables 1–2

Pith says: minor objection 2 of 4

Because task sets are fixed across the five runs, paired comparisons over the same tasks are possible; the manuscript should report paired significance (e.g., bootstrap or Wilcoxon signed-rank over tasks) rather than relying on visual separation of the mean±std intervals, which is weak evidence for several adjacent entries.

show 2 more

The paper says, at §5.2, Table 11

Pith says: minor objection 3 of 4

The paired counterfactual results (74 higher final score, 101 ties, 25 worse out of 200 pairs) are consistent with the local-recovery claim but show that most pairs tie; the main-text sentence should state the local-recovery comparison as the primary evidence and characterize the final-score distribution without implying a uniform advantage.

The paper says, at §5.4, Appendix A.5.2

Pith says: minor objection 4 of 4

The GPT-OSS-120B robustness experiment covers only WebShop and Mind2Web Replay, and the paper concedes this at the end of the appendix; the main text's summary 'Sentry's performance gains remain robust across models' should carry the two-benchmark scope explicitly in the same sentence.

What would settle it

On the stored trajectories from the 939 detected failures, compute the eventual true task outcome (final WebShop reward, AppWorld pass rate, SWE-bench patch resolution, Mind2Web completion) per soft repair — using outcome labels never visible to the verifier — and compare, within benchmark and failure type, outcomes for episodes judged resolved versus unresolved. If judged-resolved episodes do not show materially better true outcomes than judged-unresolved ones, the verifier's gate carries no objective recovery signal and the playbook is polluted; also report the false-resolved versus false-unresolved split from a larger (≥300-window) annotation by annotators external to the project to confirm error direction. A cross-model swap (GPT-OSS-120B verifier with Qwen3.5-9B agent, rebuilt playbook, held-out evaluation) would independently pin the cause on same-model verification.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 5 Oct 2026

  1. Reader holds up, with conditions confidence moderate
  2. Skeptic agrees 3 min 39 s · 26,059 tokens read, 19,546 written
  3. Editor wrote the verdict above 27 s · 27,004 tokens read, 4,078 written
  4. Referee major revision: 2 major objections, 4 minor 34 s · 26,269 tokens read, 5,052 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

What you should know first: this is a solid, unusually well-ablated empirical paper, and I mostly believe the direction of the results. The central claim — failure knowledge is conditional and should be surfaced only when a matching failure is diagnosed, then admitted only after verified recovery — is supported by cleaner experiments than most work in this area.

The genuinely new pieces: pruning failure lessons from ACE's playbook improves held-out performance (0.335 to 0.391 on WebShop); the frozen-playbook control isolates exposure from evolution and shows persistent context placement lowers reward even when the same lessons stay retrievable on demand; and the design itself — external, failure-gated, verified playbook with top-k retrieval. The ablations are the real strength. Every component removal hurts, the intervention-need gate is the biggest lever (54.7% average drop), and the paired counterfactual forks (local recovery 33.5% to 60.5%, higher final score in 74 of 200 pairs) are the right kind of evidence. I buy the mechanism at the level of direction.

Now the soft spots, in proportion. The verifier worry is real, and the authors say so themselves ([b149]): Qwen3.5-9B is both the agent being repaired and the judge of whether the repair worked, with no reward access. The human audit covers 99 soft-repair windows with 81.8% agreement, same team, no error direction reported. The w/o-verification ablation (20% drop) proves gating matters, but removing the gate is not the same as calibrating it — an over-permissive verifier would still beat no gate. So the 81.7% recovery headline and the quality of the transferred playbook rest on an unmeasured error direction. This does not threaten the per-benchmark gains, which would survive an imperfect verifier; it does mean the mechanism claim, that reward-free verification is a valid recovery proxy, is not yet established. I would want error-direction analysis or a decoupled verifier before signing off on that.

Two smaller items. The statistics: five runs over a single fixed task set with randomized execution order. The ±std captures order noise, not task-sampling variance, so the headline numbers (77% on WebShop) are stated with more confidence than the design justifies. And the gains are uneven: WebShop carries the story, while Mind2Web Replay is +7.1% and its Replay scores are explicitly not comparable to official Mind2Web metrics — disclosed, but worth keeping in view when quoting the average.

Who this is for: anyone working on agent reliability, runtime intervention, or memory design. The exposure-versus-selection framing is a genuinely useful way to think about when lessons should reach an agent. I would bring it to reading group and would cite the pruning and exposure-control findings.

Send it to peer review. It is substantive, honestly written, and the ablation design deserves referee time. The revision ask should be verifier error direction and calibration, plus a statistical treatment that acknowledges the fixed-task-set sampling. None of that is a reject reason.

The strongest objection

Every playbook entry and every 'recovered' count in §5.2 (81.7% of 939 detected failures) is gated by Sentry's reward-free verifier (§3.3, Appendix A.8.3). The verifier is Qwen3.5-9B at temperature 0 — the same weights as the detector that diagnosed the failure and the task agent that produced the post-repair trajectory being judged. Its resolved/unresolved verdict, 'stops the diagnosed failure pattern and resumes task-relevant, observation-grounded progress' ([b71], Appendix A.8.3), is therefore a self-consistency check on the model's own repair, not an independent recovery signal; the model can be expected to recognize its own guidance in the continuation regardless of whether the task actually advances. The human audit (§5.2, Appendix A.7) mitigates this: two annotators, verdicts withheld, Cohen's κ=0.82–0.89, 81.8% agreement over 99 soft-repair windows. It does not settle it: the sample is small, the annotators are the research team that designed the verifier rubric, and only the agreement rate is reported — not the error direction. If the 18% disagreements skew toward false 'resolved,' a nontrivial fraction of admitted lessons come from episodes that did not genuinely recover, which is precisely the 'lesson from a failed repair misfires later' failure mode §2 motivates the design against; those junk lessons then enter the frozen playbook in Table 4 and the held-out transfer results. The w/o-verification ablation (20% drop, Table 3) shows gating matters, but removing the gate is not the same as calibrating it; an over-permissive gate would still beat no gate. The authors acknowledge the shared-model design ([b149]) but do not quantify the over-attribution risk. This does not threaten the direction of the objective per-benchmark gains (Tables 1–2), which survive even if the verifier is imperfect; it threatens the mechanism claim (reward-free verification is a valid recovery proxy), the 81.7% recovery headline, and the quality of the transferred playbook.

The reader's weakest assumption — that the shared-model reward-free verifier's resolved/unresolved judgment is a valid recovery proxy — is also the single most load-bearing point in the paper, and I agree with it. My formulation adds one sharpening: since the verifier shares weights with both the detector and the task agent whose continuation it judges, the verdict is a self-consistency check with a built-in prior that the model's own guidance worked; agreement on 99 same-team windows and κ statistics do not measure the error direction, which is exactly what would decide whether over-attribution pollutes the playbook. I considered two alternative concerns and found them secondary. (a) The persistent-exposure control (Table 4) confounds lesson content with added context length; but the retrieval ablations (§5.3, Table 14) hold the number of injected entries fixed while varying content (label-filtered 0.50/0.73 vs. random 0.33/0.55) and the w/o-intervention-need ablation shows timing alone can take WebShop below the base agent (0.136 vs. 0.170), so conditional exposure has independent, length-free support. (b) The '37%' aggregation is dominated by WebShop's +77% and SWE-bench rests on roughly a 3-resolved-task difference on a fixed 50-task set, so magnitude statements deserve caution, but the direction is positive on all four benchmarks and on the two-model GPT-OSS check. Credit where due: the internal arithmetic of Tables 1–3 and §5.3 all checks; every component ablation reduces performance; the paired-fork experiment compares objective final scores (74 wins vs. 25 losses); the human audit is two-annotator, blinded to verdicts, with κ=0.82–0.89; and the authors explicitly flag the shared-model design ([b149]) and Mind2Web score non-comparability ([b78]). None of this removes the over-attribution risk — it only narrows it. The stated fix (decoupled verifier plus, per my test, an outcome-calibration audit of existing logs) is cheap and decisive, so the reader's CONDITIONAL verdict stands as is.

The referee's report

If the conditional-exposure principle holds, it is a clear and falsifiable design guideline for test-time agent reliability and self-evolving agents, and the cross-benchmark consistency of the gains is substantial. The experimental design is unusually careful for this area: component ablations where every removal degrades performance, a frozen-playbook control that isolates exposure from memory evolution, paired counterfactual forks measuring causal local recovery, a deterministic label-based retrieval rule that is inspectable, fixed task sets across five runs, and human validation against the detector, verifier, and intervention helpfulness. The principal open risk is that every playbook admission and every headline recovery count is gated by a reward-free verifier that shares its weights with the task agent and the detector, so the transfer and recovery-rate claims stand or fall with the verifier's unbiasedness; the manuscript acknowledges the shared-model design as a limitation but the mitigation is partial.

Pith's reading of the paper (a machine's summary, not the authors')

The paper claims that recovery knowledge for LLM agents is conditional knowledge: a lesson like "after repeated empty searches, broaden the query" helps only while that failure is happening, and keeping such lessons permanently visible makes agents misapply them and perform worse. To test this, the authors build Sentry, a layer that monitors an agent mid-task, diagnoses failures, retrieves at most five matching lessons from an external playbook, guides recovery, and stores a new lesson only after verifying — without seeing task rewards — that the agent actually recovered. Across WebShop, AppWorld, SWE-bench Lite, and Mind2Web Replay, Sentry beats the strongest runtime-intervention baseline on every benchmark (37% on average, up to 77%) and the strongest context-evolution baseline by 39%. Removing failure lessons from a standard playbook improves held-out performance, and re-adding the full playbook persistently to context lowers it, supporting the central conditional-exposure principle.

What carries the argument. The load-bearing mechanism is the conditional-exposure design with three properties: surfacing (only a diagnosed failure triggers retrieval of at most k=5 matching lessons), verification (a lesson enters the playbook only after a verifier judges the triggering failure resolved over the next H=10 cycles, with no access to task rewards), and isolation (the full playbook never enters the agent's persistent context). Retrieval is label-filtered: entries must share the failure type and a fine-grained retrieval label with the diagnosis, ranked by label overlap with recency tie-breaking, and capped so no single failure subtype dominates. The companion results carry the argument: failing to gate — persistent context exposure, always-on intervention, or unverified admission — each measurably lowers performance in controlled ablations.

Core claim. The central claim is that how failure knowledge reaches an agent matters as much as what it contains. Context-evolution methods expose lessons too early — selected at task start and kept in context for the whole trajectory — so the agent must judge at every step whether each applies, and it often judges wrong. Runtime interventions act at the right moment but never learn from their rescues. Sentry combines the two: it maintains an evolving recovery playbook outside the agent's context, surfaces a few matching lessons only when it diagnoses a failure mid-trajectory, and admits a lesson only after a verifier, blind to task rewards, judges the recovery resolved. If the paper is right, the load-bearing result is that a failure-gated, verified, externalized memory reliably outperforms both always-on lesson exposure and intervention-without-learning across heterogeneous agentic tasks, and transferring lessons works on held-out tasks. The reported evidence includes a paired fork experiment showing local recovery rises from 33.5% without intervention to 60.5% with Sentry's soft repair.

The premise it rests on. Everything the playbook stores is gated on a verifier that shares its model with the agent and never sees task rewards, so the method's value depends on that model reliably telling whether a recovery actually happened.

If the paper is right (Pith's inferences, not the authors')

Failure-specific memory should be kept outside the agent's persistent context and exposed only on a diagnosed, matching failure; adding the full playbook persistently lowers WebShop reward by 0.068 even when matching lessons remain retrievable on demand.

Lesson admission should be gated on verified recovery, since a lesson learned from a failed repair would misfire on future tasks too; removing verification reduces performance by roughly 20% on average.

Task-level context evolution and failure-level recovery memory are complementary: combining them (ACE+Sentry) gives the best measured result on both benchmarks where both are evaluated.

Learned recovery lessons transfer across disjoint tasks: a frozen playbook initialized on 50 tasks improves held-out WebShop reward from 0.251 to 0.364 and Mind2Web Replay completion from 0.533 to 0.651, with further gains from continuing verified updates.

Recovery works best when retrieval is matched to the diagnosed failure; loosening retrieval to failure-type-only, unfiltered, or random selection degrades performance by roughly 11%, 19%, and 29% respectively.

The conditional-exposure principle likely generalizes beyond failure knowledge to any piece of advice an agent could misapply: the cost of a lesson lies not in what it says but in where the agent is forced to weigh it, which suggests a testable comparison of gating against simple pruning across non-failure advice as well.

Because the detector, verifier, and task agent use the same model, the measured gains may partly reflect the model being good at recovering from its own mistakes rather than recoverable failures in general; using decoupled models for detection and verification is the direct way to separate these effects.

The single-model verifier is the load-bearing point: 81.8% human agreement on 99 soft-repair windows is the evidence the paper offers, and the entire playbook is gated on that judgment, so an independent recovery signal is the most direct check of the headline numbers.

The playbook's steady growth (937 detected failures, 227 interventions in one setting) raises a question the paper flags: whether confidence-based admission, merging, or pruning of redundant entries would keep retrieval quality from degrading as the playbook ages.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

LLM agents often fail mid-task due to invalid tool calls, repeated actions, or poorly grounded reasoning, and learning from these failures is a path to reliability. We find that how failure knowledge reaches the agent matters as much as what it contains. Failure lessons are conditional: kept in the agent's context, they misfire when their failure is absent, and removing them from an evolving playbook improves performance. Runtime interventions, in contrast, act only when a failure occurs but do not learn from their repairs. We argue that failure knowledge is conditional knowledge and should be conditionally exposed, and instantiate this principle in Sentry, a failure-management layer that runs alongside the agent. When Sentry detects a failure, it retrieves matching lessons from an external playbook to guide recovery, verifies without access to task rewards whether the agent recovered, and stores a new lesson only if it did; the full playbook never enters the agent's context. Across multiple agentic benchmarks, Sentry outperforms the strongest runtime-intervention baseline on every benchmark, by 37% on average, and the strongest context-evolution baseline by 39% on the two benchmarks where both are evaluated; combining Sentry with context evolution yields further gains. Learned lessons transfer to held-out tasks, and controlled experiments show that exposing the full playbook to the agent lowers performance even when relevant lessons remain available on demand.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-10-05

The authors' main claim, as the classifier picked it out 0.80

“We argue that failure knowledge is conditional knowledge and should be conditionally exposed, and instantiate this principle in Sentry, a failure-management layer that runs alongside the agent.”

How the main claim is stated: asserted 0.95

What the abstract itself states, as probabilities: claims first 0.05 · states limitation 0.31 · human participants 0.04 · controlled comparison 0.97 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.98 · mentions code or data availability 0.03

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
LLM agents often fail mid-task due to invalid tool calls, repeated actions, or poorly grounded reasoning, and learning from these failures is a path to reliability. background 1.00 0.08 0.04 asserted 0.98
We find that how failure knowledge reaches the agent matters as much as what it contains. result 0.89 0.90 0.28 asserted 1.00
Failure lessons are conditional: kept in the agent's context, they misfire when their failure is absent, and removing them from an evolving playbook improves performance. result 0.91 0.83 0.34 asserted 1.00
Runtime interventions, in contrast, act only when a failure occurs but do not learn from their repairs. background 0.78 0.16 0.05 asserted 1.00
We argue that failure knowledge is conditional knowledge and should be conditionally exposed, and instantiate this principle in Sentry, a failure-management layer that runs alongside the agent. contribution 0.91 0.71 0.71 asserted 0.95
When Sentry detects a failure, it retrieves matching lessons from an external playbook to guide recovery, verifies without access to task rewards whether the agent recovered, and stores a new lesson only if it did; the full playbook never enters the agent's context. method 0.99 0.93 0.18 asserted 1.00
Across multiple agentic benchmarks, Sentry outperforms the strongest runtime-intervention baseline on every benchmark, by 37\% on average, and the strongest context-evolution baseline by 39\% on the two benchmarks where both are evaluated; combining Sentry with context evolution yields further gains. result 1.00 0.96 0.31 asserted 1.00
Learned lessons transfer to held-out tasks, and controlled experiments show that exposing the full playbook to the agent lowers performance even when relevant lessons remain available on demand. result 1.00 0.95 0.19 asserted 1.00

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.99

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-05: 1 request, 6,383 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (4)
Figure 1 of the paper
Figure 1. Two ways failure knowledge goes wrong (Qwen3.5-9B on WebShop). (a) Always exposed: recovery rules learned by ACE stay in the agent’s context; after finding a matching item, the agent keeps comparing alternatives and takes no action. (b) Never learned: AgentGuard’s fixed no-progress warning resolves a search loop, but no lesson is stored; in a later task, the same warning fires after valid option selections, and the agent leaves the page and loses them. two apart next time. In
Figure 2 of the paper
Figure 2. Sentry’s framework. Sentry routes invalid actions to hard repair and progress or reasoning– grounding failures to soft repair with retrieved playbook guidance. Hard repairs are checked immediately; soft repairs undergo delayed recovery verification. Only verified soft-repair recoveries are added to the external playbook for use in future interventions. Sentry first checks whether the action is executable under the environment’s required schema. If it is not, Sentry invokes hard repair: the task agent receives the invalid action, the relevant trajectory context, and the schema, and…
Figure 3 of the paper
Figure 3. Local recovery rates by failure category and benchmark. Recovery rates measure the fraction of detected failures judged resolved within each category and benchmark. Action-validity recovery requires a corrected action accepted by the environment; progress and reasoning–grounding recovery is assessed by Sentry’s verifier after (H=10) reasoning–action–observation cycles. Soft repair improves local recovery and final task performance. To further evaluate Sentry’s soft repair, we fork 200 soft-repair intervention checkpoints across the four benchmarks: from the same agent state, one…
Figure 4 of the paper
Figure 4. Detailed examples underlying

Figures are the authors', reproduced from the paper.

References (32)
  1. [1] Lakshya A. Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J. Ryan, Meng Jiang, Christopher Potts, Koushik Sen, Alexandros G. Dimakis, Ion Stoica, Dan Klein, Matei Zaharia, and Omar Khattab. GEPA: Reflec... on Pith
  2. [2] Mert Cemri, Melissa Z. Pan, Shuyi Yang, Lakshya A. Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, Matei Zaharia, Joseph E. Gonzalez, and Ion Stoica. Why do multi-agent LLM systems fail? InAdvances in Neural Information... on Pith
  3. [3] Mengzhuo Chen, Junjie Wang, Zhe Liu, Yawen Wang, and Qing Wang. From failed trajectories to reliable llm agents: Diagnosing and repairing harness flaws.arXiv preprint arXiv:2606.06324, 2026 on PithPith's reviewarXiv
  4. [4] Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2Web: Towards a generalist agent for the web. InAdvances in Neural Information Processing Systems (NeurIPS), 2023 on Pith
  5. [5] Sunny Dubey. Real-time detection and repair of llm agent failures.arXiv preprint arXiv:2608.02464, 2026 on PithPith's reviewarXiv
  6. [6] Zhibin Gou, Zhihong Shao, Yeyun Gong, Yujiu Yang, Nan Duan, Weizhu Chen, et al. Critic: Large language models can self-correct with tool-interactive critiquing. InInternational Conference on Learning Representations, volume 2024, pages 57734–57811, 2024 on Pith
  7. [7] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world github issues? InInternational Conference on Learning Representations (ICLR), 2024 on Pith
  8. [8] Roham Koohestani. AgentGuard: Runtime verification of AI agents.arXiv preprint arXiv:2509.23864, 2025 on PitharXiv
  9. [9] Xinghua Lou, Miguel Lázaro-Gredilla, Antoine Dedieu, Carter Wendelken, Wolfgang Lehrach, and Kevin P . Murphy. Autoharness: Improving llm agents by automatically synthesizing a code harness. arXiv preprint arXiv:2603.03329, 2026 on PitharXiv
  10. [10] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self- feedback.Advances in neural information processing systems, 36:46534–46594, 2023 on Pith
  11. [11] Mike Merrill, Alexander Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Shin, Thomas Walshe, E Kelly Buchanan, et al. Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces. InInternational Conference on Learning Re... on Pith
  12. [12] Hadar Mulian, Sergey Zeltyn, Ido Levy, Liane Galanti, Avi Yaeli, and Segev Shlomov. AgentFixer: From failure detection to fix recommendations in LLM agentic systems.arXiv preprint arXiv:2603.29848, 2026 on PitharXiv
  13. [13] Rahul Nanda, Chandra Maddila, Smriti Jha, Euna Mehnaz Khan, Matteo Paltenghi, and Satish Chandra. Wink: Recovering from misbehaviors in coding agents. InProceedings of the 3rd ACM International Conference on AI-Powered Software, pages 208–217, 2026 on Pith
  14. [14] OpenAI. Introducing gpt-oss.https://openai.com/index/introducing-gpt-oss/, 2025. 10 on Pith
  15. [15] Siru Ouyang, Jun Yan, I-Hung Hsu, Yanfei Chen, Ke Jiang, Zifeng Wang, Rujun Han, Long T. Le, Samira Daruki, Xiangru Tang, Vishy Tirumalashetty, George Lee, Mahsan Rofouei, Hangfei Lin, Jiawei Han, Chen-Yu Lee, and Tomas Pfister. ReasoningBank: Scaling agent self-evolving with ... on Pith
  16. [16] Qwen Team. Qwen3.5-9B.https://huggingface.co/Qwen/Qwen3.5-9B, 2026 on Pith
  17. [17] Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. InAdvances in Neural Information Processing Systems (NeurIPS), 2023 on Pith
  18. [18] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. InAdvances in Neural Information Processing Systems, volume 36, 2023 on Pith
  19. [19] Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal, and Niranjan Balasubramanian. Appworld: A controllable world of apps and people for benchmarking interactive coding agents. InProceedings of the 62nd Annual Mee... on Pith
  20. [20] Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models.arXiv preprint arXiv:2305.16291, 2023 on PithPith's reviewarXiv
show 12 more
  1. [21] Zora Zhiruo Wang, Jiayuan Mao, Daniel Fried, and Graham Neubig. Agent workflow memory. In International Conference on Machine Learning, 2025 on Pith
  2. [22] Cailin Winston and René Just. A taxonomy of failures in tool-augmented LLMs. InProceedings of the 2025 IEEE/ACM International Conference on Automation of Software Test (AST), pages 125–135, 2025 on Pith
  3. [23] Tianshi Xu, Huifeng Wen, and Meng Li. Adapting the interface, not the model: Runtime harness adaptation for deterministic llm agents.arXiv preprint arXiv:2605.22166, 2026 on PithPith's reviewarXiv
  4. [24] Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. WebShop: Towards scalable real- world web interaction with grounded language agents. InAdvances in Neural Information Processing Systems, 2022 on Pith
  5. [25] Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. τ-bench: A benchmark for tool-agent-user interaction in real-world domains.arXiv preprint arXiv:2406.12045, 2024 on PithPith's reviewarXiv
  6. [26] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. InInternational Conference on Learning Representations (ICLR), 2023 on Pith
  7. [27] Ye Yu, Xiaopeng Yuan, Haibo Jin, Heming Liu, Yaoning Yu, and Haohan Wang. Do self-evolving agents forget? capability degradation and preservation in lifelong llm agent adaptation.arXiv preprint arXiv:2605.09315, 2026 on PithPith's reviewarXiv
  8. [28] Boxuan Zhang, Jianing Zhu, Zeru Shi, Dongfang Liu, and Ruixiang Tang. AgentForesight: Online auditing for early failure prediction in multi-agent systems.arXiv preprint arXiv:2605.08715, 2026 on PithPith's reviewarXiv
  9. [29] Hangfan Zhang, Shao Zhang, Kangcong Li, Chen Zhang, Yang Chen, Yiqun Zhang, Lei Bai, and Shuyue Hu. Self-harness: Harnesses that improve themselves.arXiv preprint arXiv:2606.09498, 2026 on PithPith's reviewarXiv
  10. [30] Qizheng Zhang, Changran Hu, Shubhangi Upasani, Boyuan Ma, Fenglu Hong, Vamsidhar Kamanuru, Jay Rainton, Chen Wu, Mengmeng Ji, Hanchen Li, Urmish Thakker, James Zou, and Kunle Olukotun. Agentic context engineering: Evolving contexts for self-improving language models. InInterna... on Pith
  11. [31] Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. ExpeL: LLM Agents Are Experiential Learners. InProceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19632–19642, 2024 on Pith
  12. [32] Kunlun Zhu, Zijia Liu, Bingxuan Li, Muxin Tian, Yingxuan Yang, Jiaxun Zhang, Pengrui Han, Qipeng Xie, Fuyang Cui, Weijia Zhang, et al. Where llm agents fail and how they can learn from failures.arXiv preprint arXiv:2509.25370, 2025. 12 A Appendix A.1 Additional Related Work A.... on PitharXiv
Cited by 4 Pith papers

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment