Pith

open record

sign in

arxiv: 2607.06273 · v1 · pith:QCNYA3CS · submitted 2026-07-07 · cs.SE

AgentTether: Graph-Guided Diagnosis and Runtime Intervention for Reliable LLM Agent Operation

Chenyu Zhao , Shenglin Zhang , Wenwei Gu , Yongqian Sun , Dan Pei , Chetan Bansal , Saravan Rajmohan , Minghua Ma This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-07-08 11:09 UTCglm-5.2pith:QCNYA3CSrecord.jsonopen to challenge →

classification cs.SE
keywords LLM agent repairroot cause analysisruntime interventiongraph-guided diagnosisanomaly detectiontrajectory repairtool-use agents
0
0 comments X

The pith

Graph-Guided Repair Recovers 69% of Failed LLM Agent Tasks

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents AgentTether, a runtime repair framework for LLM agents that combines graph-guided root-cause diagnosis with guarded runtime intervention. It claims that effective agent repair requires two things beyond diagnosis: localizing the upstream decision that caused a failure via dependency-aware graph analysis, and keeping that correction active during re-execution through a supervised intervention harness. The central object is the Critical Transition Graph (CTG), which models each agent run as a sequence of Transition Units—decision–execution–feedback cycles—linked by both temporal order and information-flow dependencies. An offline heterogeneous graph transformer, trained on success-only trajectories, scores each unit for structural anomaly, complemented by a run-local Isolation Forest detector. An analyst LLM then converts the localized anomalous substructures into behavior-scoped repair guidance, which is injected into the next retry. A cross-iteration Repair Memory tracks what has been fixed versus what remains unresolved, and a runtime harness monitors re-execution at tool-return and text-response hooks, intervening only when the agent drifts from the repair plan. The paper evaluates this on 261 tau-bench tasks across three domains with Qwen3.7-max and cross-validates on Banking with GPT-5.4. AgentTether repairs 69.11% of initially failed tasks, improving over blind retry by 26.02 percentage points overall and 32.53 on Banking, while reducing average agent turns and end-to-end tokens. The paper argues that diagnosis alone is insufficient because agents drift from one-shot feedback during long tool-use trajectories, and that graph-guided attribution plus guarded intervention together close the repair loop.

Core claim

The paper's central claim is that agent repair is a two-part problem—localizing the upstream root cause via dependency-aware graph attribution, and enforcing the correction during re-execution—and that both parts are necessary. The offline HGT detector, trained on 21,143 success-only trajectories from unrelated domains, supplies a learned normal-behavior prior that the paper claims transfers across task domains by learning telemetry structure rather than specific tool policies. The ablation removing this detector drops overall repair from 69.11% to 46.34%, and removing Repair Memory drops it to 52.03%, indicating both components are load-bearing. The runtime intervention harness adds a net 8

What carries the argument

Critical Transition Graph (CTG): a graph where nodes are Transition Units (one observation–belief–action–feedback cycle each), temporal edges connect consecutive units, and dependency edges link units sharing artifacts or error signatures. An offline Heterogeneous Graph Transformer scores each unit's conformity to a learned normal-behavior model; a real-time Isolation Forest detector scores run-local anomalies. Their union forms anomalous substructures that an analyst LLM converts into a diagnosis with root cause, turning point, and recovery hints. A feedback builder turns the diagnosis into behavior-scoped guidance. A runtime harness checks for loop repetition, intent drift, expectation dev

If this is right

  • Production agent deployments could wrap existing agents with a diagnostic-and-intervention layer without retraining the underlying model, reducing wasted re-execution on tasks that fail due to upstream decisions.
  • The finding that one-shot feedback decays to below 50% adherence by step 13 suggests that any agent system relying on injected context for behavioral correction in long trajectories needs runtime enforcement, not just better prompts.
  • The cross-domain transfer claim for the offline HGT model, if validated beyond tau-bench, would imply that normal-execution structure is partially domain-agnostic, reducing the need for domain-specific training data in agent monitoring.
  • The hurt cases where excessive intervention causes re-planning instead of commitment suggest that intervention policy design is itself a non-trivial control problem, potentially requiring domain-specific calibration.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The offline HGT's transfer from TerminalBench and SWE-smith to tau-bench is the framework's most fragile link. If the learned prior encodes domain-specific tool-call patterns rather than abstract telemetry structure, deployment to a new domain with unfamiliar tool semantics could degrade diagnosis quality sharply, as the Banking ablation suggests.
  • The intervention harness resembles a lightweight runtime verifier for agent behavior, and its success on compliance-heavy Banking tasks versus neutrality on Airline suggests that the value of runtime intervention scales with the cost of irreversible state changes—a pattern that could generalize to other domains with destructive operations.
  • The Repair Memory mechanism implicitly assumes that failures are decomposable into independent fixes that accumulate without interaction; domains where fixes interact non-linearly (fixing one constraint breaks another) may require a more sophisticated dependency model between corrections.

Load-bearing premise

The offline HGT normal-behavior model, trained on 21,143 success-only trajectories from TerminalBench and SWE-smith (domains unrelated to tau-bench), is assumed to generalize to tau-bench's Retail, Airline, and Banking domains because it learns normal telemetry structure rather than specific tool policies. The ablation shows removing it drops Banking repair from 59.04% to 27.71%, making this cross-domain transfer load-bearing for the entire pipeline.

What would settle it

Deploy AgentTether on a new agent benchmark domain whose tool semantics and execution patterns differ substantially from both the training corpora (TerminalBench, SWE-smith) and the evaluation domains (tau-bench Retail, Airline, Banking). If the offline HGT detector's anomaly scores fail to localize the upstream root cause better than the real-time Isolation Forest alone, the cross-domain transfer claim is falsified.

Figures

Figures reproduced from arXiv: 2607.06273 by Chenyu Zhao, Chetan Bansal, Dan Pei, Minghua Ma, Saravan Rajmohan, Shenglin Zhang, Wenwei Gu, Yongqian Sun.

Figure 1
Figure 1. Figure 1: Agent self-repair versus human repair on a Banking task. (a) Missing [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Failure analysis of the 83 initially-failed Banking runs [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Feedback adherence by tool-call step after feedback [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Overview of AgentTether. Two repair tracks wrap a single Agent Run and interact only through it. Post-run (Observe & Diagnose): the completed trace is lifted into a Critical Transition Graph (CTG), where two complementary detectors isolate anomalous substructures that an LLM analyst turns into a diagnosis and repair guidance; a cross-iteration Repair Memory carries fixed/unresolved state across iterations.… view at source ↗
Figure 5
Figure 5. Figure 5: Banking transfer and repair dynamics. (a) Cross-model [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
read the original abstract

Large language model (LLM) agents are increasingly used for multi-step, stateful tool-use tasks, yet production reliability remains limited. Unlike static software repair, agent repair must recover dynamic trajectories whose early decisions can propagate into later errors and external state changes. Existing automatic remedies address only part of this problem: blind retry adds no diagnosis, outcome feedback says whether a run failed but not where or why, and self-reflection often lacks grounded evidence to prevent the same failure from recurring. We present AgentTether, a run-time repair framework that automates post-run diagnosis and guided recovery without modifying the underlying agent or environment. AgentTether abstracts each run into Transition Units, links them through a dependency-aware Critical Transition Graph, and localizes failure-critical subtrajectories by combining an offline normal-behavior model with a run-local graph detector. It then converts the localized cause into behavior-scoped guidance backed by cross-iteration Repair Memory, and can optionally apply guarded run-time intervention to keep the correction active during re-execution. The same design can be deployed as an offline diagnostic-and-guidance tool or as an online repair layer. We evaluate AgentTether on 261 tau-bench tasks across three domains with Qwen3.7-max, and test cross-model transfer on Banking with GPT-5.4. On the hardest Banking domain, AgentTether repairs 59.04% (49/83) of initially failed Qwen3.7-max tasks and 65.12% (56/86) of initially failed GPT-5.4 tasks. Overall, AgentTether improves repair effectiveness while reducing agent turns and end-to-end approach tokens, suggesting a practical reliability layer that can wrap existing agent deployments, reduce wasted re-execution, and improve recovery without retraining the agent.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 8 minor

Summary. The paper presents AgentTether, a runtime repair framework for LLM agents that combines graph-guided root-cause diagnosis over a Critical Transition Graph (CTG), cross-iteration Repair Memory, and guarded runtime intervention. The system abstracts agent traces into Transition Units (TUs), localizes failure-critical subtrajectories using an offline Heterogeneous Graph Transformer (HGT) trained on external success-only corpora plus a run-local Isolation Forest detector, and converts diagnoses into behavior-scoped guidance that is injected into subsequent repair iterations. The evaluation covers 261 τ-bench tasks across three domains (Retail, Airline, Banking) with Qwen3.7-max, plus a cross-model Banking study with GPT-5.4. The paper reports 69.11% repair of initially failed tasks overall, a 26.02 pp improvement over blind retry, with simultaneous reductions in agent turns and end-to-end tokens. The approach is evaluated against Blind retry, Outcome feedback, Reflexion, and a post-run-only ablation, with McNemar's exact tests for paired comparisons and component ablations isolating the offline HGT and Repair Memory.

Significance. The paper addresses a practically important problem: recovering failed LLM-agent runs without modifying the underlying agent or environment. The integrated pipeline—diagnosis, stateful guidance, and guarded intervention—is a reasonable framing of the repair loop that existing methods cover only partially. Strengths include the use of paired statistical tests (McNemar's exact), component-level ablations (Table II), a conservative localization proxy that acknowledges its own limitations (§IV-B.3), and the avoidance of train-test contamination by training the offline HGT on external corpora that exclude τ-bench. The release of source code and experimental setup via an anonymous repository supports reproducibility. The cross-model transfer study on Banking (GPT-5.4) provides initial evidence that the diagnosis pipeline is not tied to a single agent backbone. The efficiency results (fewer turns and tokens at higher repair rates) are a notable practical contribution.

major comments (3)
  1. §III-C.2 and Table II: The offline HGT is the single most impactful component—removing it drops Banking repair from 59.04% to 27.71% and overall from 69.11% to 46.34%. The HGT is trained on TerminalBench (command-line tasks) and SWE-smith (software engineering), which have fundamentally different behavioral patterns from τ-bench's customer-service tool calls. The paper asserts (§III-C.2) that the telemetry schema normalizes surface features so the HGT learns 'a normal-execution prior over telemetry structure rather than specific tool policies,' but this cross-domain transfer claim is load-bearing and not independently validated. The 71.8% coverage metric (§IV-B.3) is measured on the same τ-bench data used for the main evaluation, so it does not establish generalization to a new deployment domain. An in-domain HGT comparison (trained on τ-bench success trajectories) or a held-out domain (
  2. §IV-B.3: The 71.8% localization coverage is described as conservative because g marks the earliest externally visible violation while the HGT may select an earlier drift. However, this metric is computed only on the 78 behavioral Banking failures under Qwen3.7-max. The localization proxy is not reported for Retail or Airline, so the reader cannot assess whether graph-guided RCA generalizes across domains or is specific to Banking's failure patterns. Reporting the coverage metric for all three domains would strengthen the localization claim.
  3. Table I: The Retail (n=26) and Airline (n=14) sample sizes are small, and several between-approach differences in these domains are not statistically significant (e.g., Retail: AgentTether 96.15% vs. Blind retry 88.46%, McNemar p not reported but likely non-significant with n=26). The paper does not report p-values for the main Table I comparisons (only Table III reports McNemar p-values for the intervention ablation). Without per-comparison p-values in Table I, it is difficult to assess which differences are statistically meaningful versus attributable to small-sample noise. The overall claim of 26.02 pp improvement over blind retry is driven primarily by Banking (n=83); the Retail and Airline contributions should be qualified accordingly.
minor comments (8)
  1. §III-C.2: The HGT hidden dimension is 52, which is unusually small. A brief justification or sensitivity analysis would help the reader understand whether this is a principled choice or an arbitrary hyperparameter.
  2. Figure 4: The diagram is dense and some labels are difficult to read. Consider splitting into separate sub-figures for the post-run track and the run-time track, or enlarging key labels.
  3. §IV-A: The paper states that all hyperparameters are fixed across domains and models and are not tuned per benchmark. This is a strength, but the specific values (e.g., intent drift thresholds 0.35/0.50/0.55, EMA alpha=0.3) are listed without justification. A brief note on how these were selected would improve transparency.
  4. Table I: The 'Avg Dur.' column shows AgentTether (15.59 min) is slower than AgentTether (post-run only) (13.47 min) but faster than Blind retry (16.32 min). The text attributes this to auxiliary verification and intervention latency, but the magnitude of intervention overhead (2.12 min on Banking) should be discussed relative to the repair gains.
  5. §II, Figure 2b: The caption says 'median 4, max 26' for causal distance, but the x-axis bins (0, 1, 2-3, 4-5, >=6) make it hard to verify the median from the figure. Consider adding a dashed line or annotation for the median.
  6. Reference [26] (PROBE) appears to be by some of the same authors (Zhao et al.). The overlap in methodology or prior results should be disclosed explicitly to help the reader assess novelty.
  7. §III-D.2: The 'Minimal' guard chooses the least disruptive response, but the criteria for ranking disruptiveness are not specified. Is 'on tool return' always considered less disruptive than 'on text response'? A brief clarification would help.
  8. The paper uses model names (Qwen3.7-max, GPT-5.4, DeepSeek-V4-Pro) that appear to be future or hypothetical versions. If these are real models, the references should be complete; if hypothetical, this should be noted.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for a careful and constructive review. The three major comments all identify legitimate gaps in validation and statistical reporting. We address each below and commit to revisions for all three.

read point-by-point responses

  1. Referee: §III-C.2 and Table II: The offline HGT is the single most impactful component, but it is trained on TerminalBench and SWE-smith, which differ fundamentally from τ-bench's customer-service tool calls. The cross-domain transfer claim is load-bearing and not independently validated. The 71.8% coverage metric is measured on the same τ-bench data used for the main evaluation. Request: an in-domain HGT comparison or held-out domain validation.

    Authors: The referee is correct that the cross-domain transfer of the offline HGT is a load-bearing claim and that our current evidence does not independently validate it. We acknowledge two gaps: (1) the 71.8% coverage metric is computed on the same τ-bench Banking data used in the main evaluation, so it does not constitute an independent generalization test; (2) we do not include an in-domain HGT baseline (trained on τ-bench success trajectories), which would directly show whether the external-domain prior adds value beyond domain-matched training data. We will address both in the revision. Specifically, we will add an in-domain HGT comparison: training the HGT on τ-bench success trajectories from a held-out split (excluding the evaluated tasks) and comparing its repair rate against the current externally-trained HGT. This will show whether the telemetry-schema normalization genuinely enables cross-domain transfer or whether an in-domain model performs better. We will also add a held-out domain validation: training the HGT on two τ-bench domains and evaluating localization coverage on the third, providing an independent generalization measure not contaminated by the main evaluation data. We agree that if the in-domain HGT substantially outperforms the external one, the cross-domain transfer claim would need to be significantly qualified. We will report results transparently regardless of outcome. revision: yes

  2. Referee: §IV-B.3: The 71.8% localization coverage is computed only on the 78 behavioral Banking failures under Qwen3.7-max. Not reported for Retail or Airline. Request: report the coverage metric for all three domains.

    Authors: This is a fair observation. The localization proxy was reported only for Banking because that domain has the largest failure set (78 behavioral failures) and the richest gold-action annotations, making it the most informative single-domain analysis. However, the referee is right that reporting coverage for all three domains would let readers assess whether graph-guided RCA generalizes across domains or is specific to Banking's failure patterns. We will compute and report the localization coverage metric (anomalous substructures covering the earliest violated gold action) for Retail and Airline as well. We note that Retail (n=26) and Airline (n=14) have smaller failure sets, so the coverage estimates will have wider uncertainty, but reporting them is still informative and we will include appropriate caveats about sample size. If the coverage metric is substantially lower in Retail or Airline, we will discuss what this implies about the generality of the localization mechanism. revision: yes

  3. Referee: Table I: Retail (n=26) and Airline (n=14) sample sizes are small, several between-approach differences are likely non-significant, and p-values are not reported in Table I. The overall 26.02 pp improvement is driven primarily by Banking. Request: per-comparison p-values in Table I and qualification of Retail/Airline contributions.

    Authors: The referee is correct on both points. We did not include McNemar's exact p-values in Table I, which makes it difficult for readers to distinguish statistically meaningful differences from small-sample noise, particularly in Retail (n=26) and Airline (n=14). We will add McNemar's exact p-values for all pairwise comparisons in Table I (AgentTether vs. Blind retry, AgentTether vs. Outcome feedback, AgentTether vs. Reflexion, and AgentTether vs. AgentTether post-run only), for each domain and overall. We expect that several Retail and Airline comparisons will indeed be non-significant given the small samples, and we will state this explicitly rather than implying statistical significance where it does not hold. Second, we agree that the overall 26.02 pp improvement is driven primarily by Banking (n=83, +32.53 pp), and that the Retail (+7.69 pp) and Airline (+21.43 pp) contributions should be qualified accordingly. We will revise the discussion of Table I to explicitly note that the overall improvement is weighted heavily by Banking, that Retail and Airline have small sample sizes limiting statistical power, and that the domain-level results should be interpreted with appropriate caution. We will not claim statistical significance for domain-level differences where the p-value does not support it. revision: yes

Circularity Check

0 steps flagged

No significant circularity; one self-citation to prior work by overlapping authors is not load-bearing for the central empirical claims.

full rationale

The paper's central claim—69.11% repair rate on τ-bench—is evaluated against external ground truth (τ-bench gold action checks) and compared against independent baselines (Blind retry, Outcome feedback, Reflexion). The offline HGT is trained on external corpora (TerminalBench, SWE-smith) that explicitly exclude τ-bench domains, so its training data is not the evaluation data. The real-time Isolation Forest operates on run-local features without fitted parameters from the target benchmark. The ablation in Table II removes the HGT and shows performance drops, which is a legitimate test of component contribution, not a circular definition. The 71.8% coverage metric (§IV-B.3) is measured on τ-bench data, but it is used as a diagnostic localization check, not as the primary repair-rate claim. The one self-citation is to PROBE [26] (Zhao et al.), which is referenced as prior work the present paper improves upon; it is not invoked as a uniqueness theorem or as a load-bearing premise for the derivation. The cross-domain transfer of the HGT is a correctness/generalization risk (the reader's weakest-assumption concern), not a circularity problem: the HGT is not defined in terms of τ-bench outcomes, nor are its predictions equivalent to fitted τ-bench parameters by construction. No step in the derivation chain reduces to its inputs by definition or by self-citation.

Axiom & Free-Parameter Ledger

6 free parameters · 4 axioms · 3 invented entities

The paper presents a well-engineered agent repair framework with clear motivation, reasonable experimental design, and honest reporting of failure cases (hurt cases). The main concerns are: (1) the cross-domain transfer of the offline HGT model is load-bearing and not independently validated beyond τ-bench, (2) sample sizes are small in two of three domains, limiting statistical power for some claims, and (3) the system's diagnosis quality depends on auxiliary LLM judgments whose accuracy is only indirectly assessed. The code is publicly available and hyperparameters are fully specified, supporting reproducibility. The paper does not inflate claims and openly discusses limitations including intervention backfire cases.

free parameters (6)
  • HGT hidden dimension = 52
    Architecture choice for the offline normal-behavior model, specified in §IV-A.
  • Intent drift thresholds = 0.35 (destructive), 0.50 (default), 0.55 (information-gathering)
    Hand-set semantic dissimilarity thresholds for triggering intervention, specified in §IV-A.
  • EMA smoothing alpha = 0.3
    Exponential moving average parameter for intent drift scoring, specified in §IV-A.
  • Structural check parameters = 5-step minimum, 1 repeated reminder, 3-fire cap
    Hand-set parameters controlling when checkpoint reminders fire, specified in §IV-A.
  • Anomaly frontier cut = largest score drop in run's own distribution
    Data-driven but the cut criterion itself is a design choice, §III-C.2.
  • Repair iteration budget Gamma = 3
    Maximum repair iterations after initial run, §IV-A.
axioms (4)
  • domain assumption The offline HGT model trained on TerminalBench and SWE-smith success trajectories learns a normal-execution prior that transfers to τ-bench domains.
    Invoked in §III-C.2: the model 'learns normal TU context' from external corpora and applies it to τ-bench at inference. The ablation (Table II) confirms this is load-bearing.
  • domain assumption An analyst LLM (DeepSeek-V4-Pro) can produce accurate root-cause diagnoses from CTG-localized evidence packets.
    The entire diagnosis stage (§III-C.3) depends on the analyst LLM's judgment quality, which is not independently validated beyond the 71.8% localization coverage proxy.
  • domain assumption τ-bench's gold action checks correctly identify the earliest violated required step as the ground-truth root cause.
    Used in §II and §IV-B.3 to evaluate localization accuracy. The paper acknowledges this is a 'conservative proxy' but relies on it for the localization claim.
  • domain assumption Monkey-patching LLM SDK entry points captures the complete behavioral trace without altering agent behavior.
    Stated in §III-B.1: patches are 'transparent' and 'leave the agent's source and behavior unchanged.' No independent verification is provided.
invented entities (3)
  • Transition Unit (TU) independent evidence
    purpose: Abstracts a decision-execution-feedback cycle as a graph node, providing stable identity for evidence and correction attachment.
    TUs are a representational choice, not a new physical entity. Their effectiveness is empirically validated through the repair rate improvements in Table I.
  • Critical Transition Graph (CTG) independent evidence
    purpose: Models temporal order and dependency relationships among TUs for graph-guided root-cause attribution.
    The CTG is a data structure whose utility is tested through the localization experiment (71.8% coverage) and ablation (removing graph components degrades performance).
  • Repair Memory independent evidence
    purpose: Carries fixed/unresolved correction state across repair iterations to prevent regressions.
    Ablation in Table II shows removing Repair Memory drops overall repair from 69.11% to 52.03%, providing falsifiable evidence of its contribution.

pith-pipeline@v1.1.0-glm · 20750 in / 4444 out tokens · 357578 ms · 2026-07-08T11:09:27.449070+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

46 extracted references · 46 canonical work pages · 10 internal anchors

  1. [1]

    The rise and potential of large language model based agents: A survey,

    Z. Xi, W. Chen, X. Guo, W. He, Y . Ding, B. Hong, M. Zhang, J. Wang, S. Jin, E. Zhouet al., “The rise and potential of large language model based agents: A survey,”Science China Information Sciences, vol. 68, no. 2, p. 121101, 2025

    work page 2025

  2. [2]

    A survey on large language model based autonomous agents,

    L. Wang, C. Ma, X. Feng, Z. Zhang, H. Yang, J. Zhang, Z. Chen, J. Tang, X. Chen, Y . Linet al., “A survey on large language model based autonomous agents,”Frontiers of Computer Science, vol. 18, no. 6, p. 186345, 2024

    work page 2024

  3. [3]

    $\tau$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains

    S. Yao, N. Shinn, P. Razavi, and K. Narasimhan, “τ-bench: A benchmark for tool-agent-user interaction in real-world domains,” 2024. [Online]. Available: https://arxiv.org/abs/2406.12045

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  4. [4]

    ReAct: Synergizing Reasoning and Acting in Language Models

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y . Cao, “React: Synergizing reasoning and acting in language models,”arXiv preprint arXiv:2210.03629, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  5. [5]

    Toolformer: Language models can teach themselves to use tools,

    T. Schick, J. Dwivedi-Yu, R. Dess `ı, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom, “Toolformer: Language models can teach themselves to use tools,”Advances in neural informa- tion processing systems, vol. 36, pp. 68 539–68 551, 2023

    work page 2023

  6. [6]

    Tree of thoughts: Deliberate problem solving with large language models,

    S. Yao, D. Yu, J. Zhao, I. Shafran, T. Griffiths, Y . Cao, and K. Narasimhan, “Tree of thoughts: Deliberate problem solving with large language models,”Advances in neural information processing systems, vol. 36, pp. 11 809–11 822, 2023

    work page 2023

  7. [7]

    Voyager: An Open-Ended Embodied Agent with Large Language Models

    G. Wang, Y . Xie, Y . Jiang, A. Mandlekar, C. Xiao, Y . Zhu, L. Fan, and A. Anandkumar, “V oyager: An open-ended embodied agent with large language models,”arXiv preprint arXiv:2305.16291, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  8. [8]

    Webarena: A realistic web environment for building autonomous agents,

    S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, T. Ou, Y . Bisk, D. Friedet al., “Webarena: A realistic web environment for building autonomous agents,” inInternational Conference on Learning Representations, vol. 2024, 2024, pp. 15 585–15 606

    work page 2024

  9. [9]

    Agentbench: Evaluating llms as agents,

    X. Liu, H. Yu, H. Zhang, Y . Xu, X. Lei, H. Lai, Y . Gu, H. Ding, K. Men, K. Yanget al., “Agentbench: Evaluating llms as agents,” inInternational Conference on Learning Representations, vol. 2024, 2024, pp. 52 989– 53 046

    work page 2024

  10. [10]

    Swe-bench: Can language models resolve real-world github issues?

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan, “Swe-bench: Can language models resolve real-world github issues?” inInternational Conference on Learning Representa- tions, vol. 2024, 2024, pp. 54 107–54 157

    work page 2024

  11. [11]

    Toolllm: Facilitating large language models to master 16000+ real-world apis,

    Y . Qin, S. Liang, Y . Ye, K. Zhu, L. Yan, Y . Lu, Y . Lin, X. Cong, X. Tang, B. Qianet al., “Toolllm: Facilitating large language models to master 16000+ real-world apis,” inInternational Conference on Learning Representations, vol. 2024, 2024, pp. 9695–9717

    work page 2024

  12. [12]

    Gorilla: Large language model connected with massive apis,

    S. G. Patil, T. Zhang, X. Wang, and J. E. Gonzalez, “Gorilla: Large language model connected with massive apis,”Advances in Neural Information Processing Systems, vol. 37, pp. 126 544–126 565, 2024

    work page 2024

  13. [13]

    Swe-agent: Agent-computer interfaces enable automated soft- ware engineering,

    J. Yang, C. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press, “Swe-agent: Agent-computer interfaces enable automated soft- ware engineering,”Advances in Neural Information Processing Systems, vol. 37, pp. 50 528–50 652, 2024

    work page 2024

  14. [14]

    AI Agents That Matter

    S. Kapoor, B. Stroebl, Z. S. Siegel, N. Nadgir, and A. Narayanan, “Ai agents that matter,”arXiv preprint arXiv:2407.01502, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  15. [15]

    Why do multi-agent llm systems fail?

    M. Cemri, M. Z. Pan, S. Yang, L. A. Agrawal, B. Chopra, R. Tiwari, K. Keutzer, A. Parameswaran, D. Klein, K. Ramchandranet al., “Why do multi-agent llm systems fail?”Advances in Neural Information Processing Systems, vol. 38, 2026

    work page 2026

  16. [16]

    Identifying the risks of lm agents with an lm-emulated sandbox,

    Y . Ruan, H. Dong, A. Wang, S. Pitis, Y . Zhou, J. Ba, Y . Dubois, C. Maddison, and T. Hashimoto, “Identifying the risks of lm agents with an lm-emulated sandbox,” inInternational Conference on Learning Representations, vol. 2024, 2024, pp. 27 031–27 098

    work page 2024

  17. [17]

    Air canada’s chatbot illustrates persistent agency and responsibility gap problems for ai,

    J. L. Brand, “Air canada’s chatbot illustrates persistent agency and responsibility gap problems for ai,”AI & SOCIETY, vol. 40, no. 5, pp. 3361–3363, 2025

    work page 2025

  18. [18]

    NYC’s AI chatbot tells businesses to break the law,

    C. Lecher, “NYC’s AI chatbot tells businesses to break the law,” The Markup, 2024. [On- line]. Available: https://themarkup.org/artificial-intelligence/2024/03/29/ nycs-ai-chatbot-tells-businesses-to-break-the-law

    work page 2024

  19. [19]

    Automated program repair,

    C. Le Goues, M. Pradel, and A. Roychoudhury, “Automated program repair,”Communications of the ACM, vol. 62, no. 12, pp. 56–65, 2019

    work page 2019

  20. [20]

    Automated program repair in the era of large pre-trained language models,

    C. S. Xia, Y . Wei, and L. Zhang, “Automated program repair in the era of large pre-trained language models,” in2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2023, pp. 1482–1494

    work page 2023

  21. [21]

    Teaching large language models to self-debug,

    X. Chen, M. Lin, N. Sch ¨arli, and D. Zhou, “Teaching large language models to self-debug,” inInternational Conference on Learning Repre- sentations, vol. 2024, 2024, pp. 8746–8825

    work page 2024

  22. [22]

    Let’s verify step by step,

    H. Lightman, V . Kosaraju, Y . Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe, “Let’s verify step by step,” inInternational Conference on Learning Representations, vol. 2024, 2024, pp. 39 578–39 601

    work page 2024

  23. [23]

    Re- flexion: Language agents with verbal reinforcement learning,

    N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao, “Re- flexion: Language agents with verbal reinforcement learning,”Advances in neural information processing systems, vol. 36, pp. 8634–8652, 2023

    work page 2023

  24. [24]

    Self-refine: Iter- ative refinement with self-feedback,

    A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y . Yanget al., “Self-refine: Iter- ative refinement with self-feedback,”Advances in neural information processing systems, vol. 36, pp. 46 534–46 594, 2023

    work page 2023

  25. [25]

    Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems

    S. Zhang, M. Yin, J. Zhang, J. Liu, Z. Han, J. Zhang, B. Li, C. Wang, H. Wang, Y . Chenet al., “Which agent causes task failures and when? on automated failure attribution of llm multi-agent systems,”arXiv preprint arXiv:2505.00212, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  26. [26]

    Debugging the Debuggers: Failure-Anchored Structured Recovery for Software Engineering Agents

    C. Zhao, S. Zhang, Y . Lin, W. Gu, Z. Chen, Y . Sun, D. Pei, C. Bansal, S. Rajmohan, and M. Ma, “Debugging the debuggers: Failure-anchored structured recovery for software engineering agents,”arXiv preprint arXiv:2605.08717, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  27. [27]

    Qwen3.7: The agent frontier,

    Qwen Team, “Qwen3.7: The agent frontier,” May 2026, aPI model identifier:qwen3.7-max. [Online]. Available: https://qwen.ai/blog? id=qwen3.7

    work page 2026

  28. [28]

    Introducing GPT-5.4,

    OpenAI, “Introducing GPT-5.4,” Mar. 2026, aPI model identifier:gpt-5.4. [Online]. Available: https://openai.com/index/ introducing-gpt-5-4/

    work page 2026

  29. [29]

    Dapper, a large-scale distributed systems tracing infrastructure,

    B. H. Sigelman, L. A. Barroso, M. Burrows, P. Stephenson, M. Plakal, D. Beaver, S. Jaspan, and C. Shanbhag, “Dapper, a large-scale distributed systems tracing infrastructure,” Google, Tech. Rep., 2010

    work page 2010

  30. [30]

    Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces

    M. A. Merrill, A. G. Shaw, N. Carlini, B. Li, H. Raj, I. Bercovich, L. Shi, J. Y . Shin, T. Walshe, E. K. Buchananet al., “Terminal- bench: Benchmarking agents on hard, realistic tasks in command line interfaces,”arXiv preprint arXiv:2601.11868, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  31. [31]

    Swe-smith: Scaling data for software engineering agents,

    J. Yang, K. Lieret, C. Jimenez, A. Wettig, K. Khandpur, Y . Zhang, B. Hui, O. Press, L. Schmidt, and D. Yang, “Swe-smith: Scaling data for software engineering agents,”Advances in Neural Information Processing Systems, vol. 38, 2026

    work page 2026

  32. [32]

    Heterogeneous graph trans- former,

    Z. Hu, Y . Dong, K. Wang, and Y . Sun, “Heterogeneous graph trans- former,” inProceedings of the web conference 2020, 2020, pp. 2704– 2710

    work page 2020

  33. [33]

    Isolation-based anomaly detection,

    F. T. Liu, K. M. Ting, and Z.-H. Zhou, “Isolation-based anomaly detection,”ACM Transactions on Knowledge Discovery from Data (TKDD), vol. 6, no. 1, pp. 1–39, 2012

    work page 2012

  34. [34]

    https://arxiv.org/abs/2606.19348

    A. Xu, B. Lin, B. Xue, B. Wang, B. Xu, B. Wu, B. Zhang, C. Lin, C. Dong, C. Linget al., “Deepseek-v4: Towards highly efficient million- token context intelligence,”arXiv preprint arXiv:2606.19348, 2026

    work page arXiv 2026

  35. [35]

    CRITIC: Large language models can self-correct with tool-interactive critiquing,

    Z. Gou, Z. Shao, Y . Gong, Y . Shen, Y . Yang, N. Duan, and W. Chen, “CRITIC: Large language models can self-correct with tool-interactive critiquing,” inInternational Conference on Learning Representations, 2024

    work page 2024

  36. [36]

    Language models can solve computer tasks,

    G. Kim, P. Baldi, and S. McAleer, “Language models can solve computer tasks,”Advances in Neural Information Processing Systems, vol. 36, pp. 39 648–39 677, 2023

    work page 2023

  37. [37]

    When can llms actually correct their own mistakes? a critical survey of self-correction of llms,

    R. Kamoi, Y . Zhang, N. Zhang, J. Han, and R. Zhang, “When can llms actually correct their own mistakes? a critical survey of self-correction of llms,”Transactions of the Association for Computational Linguistics, vol. 12, pp. 1417–1440, 2024

    work page 2024

  38. [38]

    Inner Monologue: Embodied Reasoning through Planning with Language Models

    W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng, J. Tompson, I. Mordatch, Y . Chebotaret al., “Inner monologue: Embod- ied reasoning through planning with language models,”arXiv preprint arXiv:2207.05608, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  39. [39]

    Large language models cannot self-correct reasoning yet,

    J. Huang, X. Chen, S. Mishra, H. S. Zheng, A. Yu, X. Song, and D. Zhou, “Large language models cannot self-correct reasoning yet,” in International conference on learning representations, vol. 2024, 2024, pp. 32 808–32 824

    work page 2024

  40. [40]

    DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines

    O. Khattab, A. Singhvi, P. Maheshwari, Z. Zhang, K. Santhanam, S. Vardhamanan, S. Haq, A. Sharma, T. T. Joshi, H. Moazamet al., “DSPy: Compiling declarative language model calls into self-improving pipelines,”arXiv preprint arXiv:2310.03714, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  41. [41]

    Automatic prompt optimization with “gradient descent

    R. Pryzant, D. Iter, J. Li, Y . Lee, C. Zhu, and M. Zeng, “Automatic prompt optimization with “gradient descent” and beam search,” in Proceedings of the 2023 conference on empirical methods in natural language processing, 2023, pp. 7957–7968

    work page 2023

  42. [42]

    A survey on software fault localization,

    W. E. Wong, R. Gao, Y . Li, R. Abreu, and F. Wotawa, “A survey on software fault localization,”IEEE Transactions on Software Engineering, vol. 42, no. 8, pp. 707–740, 2016

    work page 2016

  43. [43]

    Empirical evaluation of the tarantula automatic fault-localization technique,

    J. A. Jones and M. J. Harrold, “Empirical evaluation of the tarantula automatic fault-localization technique,” inProceedings of the 20th IEEE/ACM international Conference on Automated software engineer- ing, 2005, pp. 273–282

    work page 2005

  44. [44]

    Survey on Models and Techniques for Root-Cause Analysis

    M. Sol ´e, V . Munt ´es-Mulero, A. I. Rana, and G. Estrada, “Survey on models and techniques for root-cause analysis,”arXiv preprint arXiv:1701.08546, 2017

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  45. [45]

    Specification-based monitoring of cyber-physical systems: a survey on theory, tools and applications,

    E. Bartocci, J. Deshmukh, A. Donz ´e, G. Fainekos, O. Maler, D. Niˇckovi´c, and S. Sankaranarayanan, “Specification-based monitoring of cyber-physical systems: a survey on theory, tools and applications,” inLectures on Runtime Verification: Introductory and Advanced Topics. Springer, 2018, pp. 135–175

    work page 2018

  46. [46]

    Process mining: Overview and opportunities,

    W. Van Der Aalst, “Process mining: Overview and opportunities,”ACM Transactions on Management Information Systems (TMIS), vol. 3, no. 2, pp. 1–17, 2012

    work page 2012