Pith. sign in

REVIEW 4 major objections 5 minor 27 references

TRAJDEBUG: Tracing Error Lifecycle to Identify Critical Failures in Long-Horizon Agent Trajectories

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that finding the failure-responsible step in long agent runs should be decomposed into trigger detection, error-state classification, and attribution.

desk verdict A genuinely new error-lifecycle classifier with solid experiments, but the new benchmark's labels may be partly anchored to the method's own taxonomy; worth publishing with a clean re-annotation. read the letter →

arxiv 2608.06346 v1 pith:RZHVSG22 submitted 2026-08-06 cs.AI

Yunjia Qi , Zehua Yin , Xintong Shi , Hao Peng , Songyuanyi Lu , Yixian Liu , Richeng Xuan , Yuhong Liu
show 5 more authors
Zhichao Hu Xiaozhi Wang Lei Hou Bin Xu Juanzi Li
This is my paper · ORCID
classification cs.AI
keywords criticalerrordetectionlifecycleagenttrajectorydebugginglong-horizonLLMagentsattributionrepairfailurememoryTrajErrBench
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Long agent trajectories fail in cascades, and the paper's claim is that the step truly responsible for the failure can be found by following each local error through its lifecycle instead of asking an LLM to judge the whole trajectory at once. TrajDebug first detects per-step 'error triggers'—wrong commitments that conflict verbatim with a task constraint, a prior observation, or the agent's own reasoning—then groups repeated triggers into error instances anchored to the same violated reference. It classifies each instance by whether the wrong commitment was repaired and whether it left an observable terminal footprint, so repaired or dormant errors drop out and only terminal-relevant candidates reach a final attribution step. The authors build TrajErrBench, 486 manually annotated failed trajectories from tool-use and coding domains, and report that this staged pipeline reaches 34.11% macro-average exact-step accuracy, the best in their comparison, with the largest edge on long trajectories. They also show the diagnoses are actionable: per-trajectory repair with TrajDebug's feedback lifts average success from 78.07% to 88.87%.

What carries the argument

The load-bearing object is the error instance $E=(E,O)$, a cluster of triggers that share the same violated reference object $O$; each trigger is a tuple $(t,c,p,q_w,q_r)$ recording the step, reference category, execution phase, the wrong commitment, and the verbatim violated reference. Multi-granularity compression builds high-, medium-, and low-detail views of each step so that distant history stays compact while local verification can cite verbatim evidence. The decisive mechanism is the two-axis error-state classification (resolved vs. active; terminal footprint vs. none), which yields four states—Clean Resolution, Costly Resolution, Manifest Active, Latent Active—and restricts the final LLM attribution head to the candidate set $F(\tau)$ of costly-resolved and manifest-active instances.

What would settle it

Re-annotate a random sample of TrajErrBench trajectories from the raw failed trajectories alone, without model pre-annotations and without the framework's category names; if the resulting majority-vote labels diverge substantially from the released labels, or if TrajDebug's margin over direct prompting shrinks when scored against the blind labels, the reported advantage is partly an artifact of label anchoring.

Watch

Extended reading notes

Core claim

The paper's central claim is that critical error detection in long-horizon agent trajectories should be modeled as error-lifecycle tracing: an error is born as an evidence-grounded trigger at a specific step, persists as an instance tied to a violated reference object, and either gets repaired, stays active, leaves a terminal footprint, or does not. TrajDebug implements this as three stages—trigger detection, state classification, and candidate-set-guided causal attribution—and the paper reports that this decomposition outperforms both direct LLM prompting and multi-agent diagnostic baselines on exact critical-step accuracy across seven datasets. The strongest result is on long horizons: on SWE-Bench Pro (average 119.7 steps), TrajDebug reaches 24.41% exact-step accuracy versus 17.44% for the best direct-prompting baseline, and it retains over 20% accuracy on the longest length bucket where baselines fall to about 14%. In application experiments, feeding TrajDebug's diagnosis back to the executing agent improves re-execution success by 10.80 points on average, and a transfer setting that aggregates diagnoses into failure memory improves held-out tasks by 5.70 points.

Load-bearing premise

The evaluation stands on the assumption that TrajErrBench's human ground-truth labels are causally correct, even though annotators were given model pre-annotations and a guideline whose error categories match TrajDebug's own taxonomy, which could make the benchmark partly measure how well the method reproduces its annotation conventions.

Editorial extensions

If this is right

  • Exact critical-step detection on failed agent trajectories can be improved by roughly 8.4 accuracy points over holistic prompting with the same backbone, without extra inference budget, across embodied, web, tool-use, and coding domains.
  • On trajectories longer than 100 steps, TrajDebug degrades more gracefully than direct prompting—remaining above 20% exact-step accuracy where the best baseline falls to around 14%.
  • Critical-error diagnoses translate into measurable agent improvement: re-running the same task with the diagnosis injected raises average success from 78.07% to 88.87%, and aggregated failure memory raises held-out task success by 5.70 points on average.
  • Ablation evidence indicates that multi-granularity history compression is the largest single contributor, removing it costs 21.02 average-accuracy points, so preserving evidence across distance matters more than adding more LLM calls.
  • Because the method retains the true critical step in its candidate set only 42.0% of the time, further gains depend mainly on improving trigger recall and final attribution, not on state filtering.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: If TrajErrBench's human labels are partly anchored by the model pre-annotations and shared taxonomy used in annotation, the reported head-to-head margins may overstate TrajDebug's independent causal accuracy; blind re-annotation would resolve this.
  • Editorial inference: The lifecycle-state schema (resolved vs active, footprint vs none) is a transferable traceability vocabulary that could be used as a standard for agent observability and failure postmortems beyond the specific detector.
  • Editorial inference: A controlled stress-test of the method's attribution stage would replace the LLM attribution head with a deterministic rule, such as 'earliest manifest-active candidate,' to quantify how much of the gain comes from the final LLM judgment versus earlier evidence filtering.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes TrajDebug, a three-stage framework for critical error detection in long-horizon LLM agent trajectories: multi-granularity history compression, evidence-grounded error trigger detection, error state classification (Clean/Costly Resolution, Manifest/Latent Active), and candidate-set-guided causal attribution. It also introduces TrajErrBench, a benchmark of 486 manually annotated failed trajectories from τ2-Bench and SWE-Bench Pro. Experiments on seven benchmark subsets report a macro-average exact critical-step accuracy of 34.11%, the best among direct-prompting and multi-agent baselines, and application studies report that TrajDebug diagnoses improve downstream agent success by 10.80 points in per-trajectory repair and 5.70 points in failure-memory transfer. The paper includes ablations, length-based analysis, a compute-matched majority-voting baseline, a format-matched feedback comparison, and a stage-wise error decomposition.

Significance. If the results hold, the paper makes a useful contribution: it reframes critical error detection as an error-lifecycle tracing problem, provides a new long-horizon benchmark in tool-use and code domains, and demonstrates a practical feedback loop for improving agents. The paper also follows several good evaluation practices: it includes a compute-matched direct-prompting baseline (Table 10), a format-matched feedback comparison (Table 11), a component ablation (Table 2), and a detailed error analysis (Tables 12–13). These design choices strengthen the empirical core. The main risk is that the new benchmark's ground truth may be influenced by the authors' own taxonomy and by LLM pre-annotations, which would partially undermine the headline comparison.

major comments (4)
  1. [Appendix B.2, Table 4] The model-assisted pre-annotation protocol raises a load-bearing anchoring concern. Exact-match rates between pre-annotations and final human-majority labels reach 76.8% (Gemini-3.1-Pro on τ2-Bench) and are 53–77% across models, and the annotation guideline's criteria ('earliest evidence-grounded mistake,' 'failure-responsible,' reference categories, execution phases) closely match TrajDebug's trigger and state taxonomy in §3.2–3.3. The paper argues that no single model dominates, but that does not rule out collective anchoring: the final labels may substantially reflect the same LLM judgments and annotation conventions that TrajDebug is designed to reproduce. Because 486 of 869 evaluated trajectories come from TrajErrBench, and the SWE-Bench Pro long-horizon result (24.41%) and the headline macro-average (34.11%) depend on it, the benchmark may partly measure reproduction of its own annotation conventions. I recommend a clean re-annotation of a random subset without pre-annotations, a comparison of label stability under both protocols, and reporting Table 1 results with TrajErrBench excluded or with alternative label sets.
  2. [Table 1, Table 3] The central comparative claims lack uncertainty quantification. The macro-average margin over Gemini-3.1-Pro is 2.82 points, on per-dataset samples as small as 50 (GAIA, WebShop) and 58 (WhoAndWhen hand-crafted); no confidence intervals, bootstrap estimates, or paired significance tests are reported. Likewise, the per-trajectory repair gains in Table 3 (e.g., +10.00 vs +8.00 for Vanilla on Airline) may be within sampling noise. Please report confidence intervals or significance tests, or at least per-subset bootstrap intervals, for the main comparison and the application study.
  3. [§3.3, Appendix D] The method relies on ad-hoc thresholds — k=50 for budget debt, th1/th2/th3 compression caps (3000/1200/600 characters), and the 800-character attribution truncation — with no sensitivity analysis. Since Table 2 shows that removing multi-granularity compression causes the largest drop (21.02 points), the headline results could depend on these specific values. Please add a sensitivity sweep over k (e.g., 30/50/70) and at least one compression cap, and report whether the ranking in Table 1 is stable.
  4. [Appendix B.2] The SWE-Bench Pro evaluation set is the 86 of 110 initially annotated trajectories on which at least two annotators agreed on the critical step; the 24 discarded ambiguous cases are not included in any result, and inter-annotator agreement on the retained set is only κ=0.67. Excluding the hardest cases likely makes the benchmark easier and may inflate TrajDebug's 24.41% figure relative to a deployment distribution. Please report results on the full 110-trajectory set (or analyze sensitivity to the majority-vote threshold) and discuss how exclusion affects the long-horizon claim.
minor comments (5)
  1. [Table 1] The column header 'A VG' should be 'AVG' for clarity.
  2. [§3.1, Appendix D] The compression tiers th1/th2/th3 are introduced in prose but only concretely defined in Appendix D; consider defining them in §3.1 where they are first used.
  3. [§3.2, Appendix B.2] The 'Environment Anomaly' trigger category in TrajDebug is not part of the TrajErrBench annotation schema, which only lists Task Conflict, History Conflict, and Intra-Step Conflict; please clarify how env-anomaly triggers are evaluated against the benchmark's ground-truth labels.
  4. [Figure 2] The y-axis label 'Individual Model Step Accuracy (%)' is ambiguous; the caption indicates this is the average across seven models, so please align the axis label with the caption.
  5. [Appendix E.3, Table 8] The accuracy values 26.00, 20.02, 24.73 are non-monotonic as the per-step limit increases; a sentence interpreting this non-monotonicity would help readers understand the trade-off.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: TrajDebug's reported gains are not forced by construction; the TrajErrBench pre-annotation concern is a benchmark-validity caveat, not a derivational circularity.

full rationale

The paper's central claims are empirical evaluations, not derivations from first principles, and I find no step where a prediction reduces to its inputs by construction. TrajDebug is a zero-shot prompting pipeline whose three stages (trigger detection, state classification, causal attribution) are defined from the task formalization in Section 2.1 and implemented with independent prompt instructions; the benchmark labels are not produced by TrajDebug. TrajErrBench's ground truth comes from human majority vote (at least two of three annotators per trajectory), with model-assisted pre-annotations coming from GPT-5.4, Gemini-3.1-Pro, and Claude-Opus-4-6, not from TrajDebug (Appendix B.2). The taxonomy overlap between TrajDebug's reference categories/execution phases and the annotation guideline is real, but the main metric is exact critical-step index matching, and no equation or fitted parameter makes TrajDebug's selected step equal to the human label by construction. The paper also reports agreement between pre-annotations and final labels (Table 4) and shows no single model consistently determines the labels, which directly addresses the anchoring worry. Independent external benchmarks (WhoAndWhen, AgentDebugBench) also contribute to the macro-average and TrajDebug is competitive there, so the overall result does not rest solely on an internally constructed benchmark. The per-trajectory repair and failure-memory transfer experiments use held-out execution settings and compare against vanilla and self-reflection baselines; improvement is measured by task success, not by any quantity TrajDebug is fitted to. The only legitimate concern is that TrajErrBench may partly encode the authors' annotation conventions and LLM pre-annotation biases, but that is a benchmark-validity caveat rather than the kind of self-definitional, fitted-input, or self-citation reduction this pass is asked to identify. No circular step can be quoted from the paper with a concrete reduction, so the appropriate finding is no significant circularity.

Assumptions & free parameters 3 free parameters · 6 assumptions · 3 invented entities

The ledger records two hand-set thresholds (k=50 and compression character caps) plus several domain assumptions, notably the counterfactual decisive-error definition, fidelity of LLM quote extraction, sufficiency of the terminal-footprint taxonomy, reliability of majority-vote annotations, and non-anchoring of pre-annotations. The method is empirical engineering rather than a parameter-free derivation.

free parameters (3)
  • k (budget-debt threshold) = 50
    Sets whether an error resolved only after consuming more than 50% of the trajectory is classified as Costly Resolution (Section 3.3). Hand-chosen; no sensitivity analysis is reported.
  • Compression character caps (th1, th2, th3) = 3000 / 1200 / 600 chars
    Multi-granularity view sizes in Appendix D; chosen without a reported sweep and directly affect how much evidence survives compression.
  • Attribution-stage truncation limit = 800 chars
    Additional truncation limit for the full trajectory in the causal attribution stage (Appendix D); affects the amount of context the final judge sees.
assumptions (6)
  • domain assumption A step is decisive if replacing its action with a correct one, leaving prior steps unchanged, would turn failure into success (counterfactual definition).
    Adopted in Section 2.1 from Zhang et al. (2025c); the benchmark and metric depend on this causal counterfactual notion, which is not empirically verified.
  • domain assumption LLM-extracted verbatim quotes are faithful to the trajectory text.
    Stage 1 trigger detection requires wrong_content_quote and reference_quote to be explicitly citable; if the LLM fabricates or misquotes, triggers can be spurious (Section 3.2, Appendix D).
  • ad hoc to paper Terminal footprint (irreversible, semantic, budget debt) is a sufficient proxy for failure relevance.
    Section 3.3 defines F(tau) using only these three channels; the sufficiency of this taxonomy is asserted, not derived.
  • domain assumption Three-annotator majority vote yields reliable critical-step labels despite moderate agreement on SWE-Bench Pro (Fleiss kappa = 0.67).
    Appendix B.2 uses labels with at least 2 of 3 annotator agreement as ground truth; agreement is substantial on tau2-Bench but only moderate on code-heavy trajectories.
  • domain assumption Model pre-annotations provided to annotators do not systematically bias final labels.
    Appendix B.2 reports exact-match rates between pre-annotations and final labels (Table 4) but cannot rule out anchoring; the paper argues from the variability of match rates.
  • domain assumption Multi-granularity compression preserves all evidence needed for error judgment.
    Compression prompts require preserving constraints, plans, and verbatim tokens, but semantic degradation is possible; the large ablation drop without compression supports its importance.
invented entities (3)
  • Error instance E = (E, O)
    purpose: Groups triggers that violate the same reference object O to avoid overcounting repeated manifestations of one wrong commitment.
    Conceptual grouping introduced in Section 3.3; a methodological construct with no external falsifiable handle.
  • Error state labels (Clean Resolution, Costly Resolution, Manifest Active, Latent Active)
    purpose: Classify each instance by resolution status and terminal footprint in order to filter candidates before attribution.
    Taxonomy defined in Section 3.3; internal decision categories not validated against an external measure of error severity.
  • Terminal footprint channels (irreversible, semantic, budget debt)
    purpose: Operationalize the observable terminal impact of an error instance.
    Defined in Section 3.3; the choice of channels is a modeling decision rather than an independently evidenced entity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TRAJDEBUG: Tracing Error Lifecycle to Identify Critical Failures in Long-Horizon Agent Trajectories." pith.science (2026). https://pith.science/paper/RZHVSG22

@misc{pith2026260806346,
  author       = {Pith},
  title        = {Pith review of: TRAJDEBUG: Tracing Error Lifecycle to Identify Critical Failures in Long-Horizon Agent Trajectories},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RZHVSG22}},
  note         = {Machine review of arXiv:2608.06346}
}
read the original abstract

LLM-based agentic systems have shown remarkable capabilities in complex domains, while suffering from cascading errors and difficulty in debugging. Critical error detection aims to locate the earliest error step in a failed trajectory that is responsible for the final failure. However, progress faces two main challenges. First, long trajectories make it difficult to identify individual errors, since the evidence for judging a step may be scattered across distant instructions, observations, and prior context. Second, failed trajectories often contain multiple local errors with different downstream effects, only some of which remain responsible for the final failure. In this work, we propose TrajDebug, an error-lifecycle tracing framework that addresses long-trajectory error discovery with multi-granularity history compression and evidence-based error identification, and supports critical attribution by tracing each error's resolution status and terminal impact. We further construct TrajErrBench, a benchmark of 486 manually annotated failed trajectories from Tau2Bench and SWE-Bench Pro, covering realistic tool-use and coding scenarios. Experiments across diverse agent benchmarks show that TrajDebug achieves the best overall performance over existing baselines, and application studies further demonstrate that its diagnoses provide actionable feedback for improving downstream agent success. We will release the codes and data to facilitate further research.

Figures

Figures reproduced from arXiv: 2608.06346 by the authors.

Figure 1
Figure 1. Critical error detection requires grounding er [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Critical error detection accuracy and local [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. An overview of the framework of TRAJDEBUG. instruction, prior trajectory, environment feedback, or the same step. To prevent subjective drift and hallucinated diagnoses, each trigger must satisfy a verbatim evidence condition: both qw and qr must be explicitly citable; otherwise, the trigger is dis￾carded. A step may contain multiple triggers when it violates different references or expresses differ￾ent erroneous co… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Relative position of the critical error step [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Accuracy across trajectory length buckets. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Distribution of critical errors in τ 2 -Bench. Flows go from execution phase (left) to error subtype (middle) to reference category (right). flict and History Conflict in both subsets, sug￾gesting that strengthening evidence-grounded rea￾soning is the single highest-le…
Figure 7
Figure 7. Figure 7: Distribution of critical errors in SWE-Bench Pro. Flows go from execution phase (left) to error subtype [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

27 extracted references · 19 canonical work pages

  1. [1]

    The core meaning of EVERY sentence must survive

    PRESERVE SEMANTIC CORE: Compression removes ONLY redundant phrasing, verbose boilerplate, repetitive expressions, and filler words. The core meaning of EVERY sentence must survive. When in doubt, keep more rather than less

    work page

  2. [2]

    do NOT do X

    CONSTRAINTS AND PLANS ARE CRITICAL: If the content contains ANY of the following, their key information MUST be preserved across ALL three tiers: - Constraints (time limits, budget limits, "do NOT do X" prohibitions, "must do Y first" preconditions, required formats, allowed/disallowed actions) - Plans (specific action items, ordered steps, goals, sub- go...

    work page

  3. [3]

    Shengda Fan, Xuyan Ye, Yupeng Huo, Zhi-Yuan Chen, Yiju Guo, Shenzhi Yang, Wenkai Yang, Shuqi Ye, Jingwen Chen, Haotian Chen, and 1 others

    Trail: Trace reasoning and agentic issue local- ization.arXiv preprint arXiv:2505.08638. Shengda Fan, Xuyan Ye, Yupeng Huo, Zhi-Yuan Chen, Yiju Guo, Shenzhi Yang, Wenkai Yang, Shuqi Ye, Jingwen Chen, Haotian Chen, and 1 others

    arXiv

  4. [4]

    Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen

    Agentprocessbench: Diagnosing step-level process quality in tool-using agents.arXiv preprint arXiv:2603.14465. Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen

    arXiv

  5. [5]

    If the strategy was reasonable but the execution result was wrong, attribute the failure to the first execution step that produced the incorrect result

    If a flawed strategy is repeated across multiple steps, attribute the failure to the step where that strategy was first introduced. If the strategy was reasonable but the execution result was wrong, attribute the failure to the first execution step that produced the incorrect result

    work page

  6. [6]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shi- rong Ma, Peiyi Wang, Xiao Bi, and 1 others

    Who is introducing the failure? automatically attributing failures of multi-agent systems via spec- trum analysis.arXiv preprint arXiv:2509.13782. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shi- rong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforce...

    arXiv 2025

  7. [7]

    arXiv preprint arXiv:2602.23701

    From flat logs to causal graphs: Hierarchical failure attribution for llm-based multi-agent systems. arXiv preprint arXiv:2602.23701. Zhaohui Geoffrey Wang. 2026. Agenttrace: Causal graph tracing for root cause analysis in de- ployed multi-agent systems.arXiv preprint arXiv:2603.14688. xAI. 2025. Grok 4 fast model card. https://data.x. ai/2025-09-19-grok-...

    arXiv 2026

  8. [8]

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan

    Qwen3 technical report.arXiv preprint arXiv:2505.09388. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan

    arXiv

Show all 27 references
  1. [9]

    the agent seems off-track

    Tree of thoughts: Deliberate problem solving with large language models.Advances in neural information processing systems, 36:11809–11822. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting ...

    2022 arXiv

  2. [10]

    Pairwise

    from the OpenHands scaffold (Wang et al., 2025). The two subsets average 29.3 and 119.7 steps per trajectory, respectively. B.2 Annotation Details Our annotation protocol follows the decisive-error annotation practice of WhoAndWhen (Zhang et al., 2025c). Annotators are given t...

    2025

  3. [13]

    Disney",

    THREE-TIER COMPRESSION LEVELS: - th1 (detailed, <={th1_max} chars per field): Preserve all meaningful details. Remove only obvious redundancy, repeated phrasings, and decorative formatting. Keep every distinct fact, constraint, action, and observation. - th2 (moderate, <={th2_...

    1976

  4. [14]

    cat-1 instance MUST NOT have origin_commitment_status = pure_exploration

  5. [15]

    HARD ENV RULES:

    cat-1 instance MUST NOT have state = dormant. HARD ENV RULES:

  6. [16]

    An env instance MUST NOT have origin_commitment_status = pure_exploration UNLESS BOTH: (a) the environment self-recovers on the next judgable step, AND (b) the agent's next action is consistent with the recovered output

  7. [17]

    irreversible

    For env instances, wasted steps should be counted against the VIOLATED sub-goal object remaining unresolved. === TASK 2 - REPAIR / STATE DETERMINATION === 20 fixed_at_step_<N> There is a step N with anchor < N <= T at which the agent' s behavior satisfies the repair criterion ...

  8. [18]

    Consider the ENTIRE trajectory from a global perspective - understand the task goal and how the agent's path diverged from success

  9. [19]

    Early exploration steps (steps 1-3) are often normal and should NOT be marked as critical unless they clearly introduce the root cause

  10. [20]

    This includes producing an incorrect output, misinterpreting or misusing prior information, or turning a correct intermediate state into an incorrect one

    Prefer the first step where the agent had enough reasonable information to proceed correctly but nevertheless introduced the error locally. This includes producing an incorrect output, misinterpreting or misusing prior information, or turning a correct intermediate state into ...

  11. [21]

    If a later step only repeats, propagates, or amplifies an earlier mistake, select the earlier step where the mistake originated

    Do not choose a later step merely because the failure becomes more visible there. If a later step only repeats, propagates, or amplifies an earlier mistake, select the earlier step where the mistake originated

  12. [23]

    critical_step

    Use recoverability only as a tie-breaker: among otherwise similar candidates, prefer the earliest step whose error made recovery unlikely or blocked the trajectory from returning to a successful path. You are given: - the TASK, - the FULL TRAJECTORY, - a list of SUSPICIOUS STE...

  13. [24]

    Identify the earliest step which directly leads the agent off track or repeats ineffective behaviour

  14. [25]

    Do not shift to later steps of that error

    Reference that exact step number as shown in the trajectory. Do not shift to later steps of that error

  15. [26]

    Explain why the chosen step is wrong, citing relevant observation/action details

  16. [27]

    Suggest a concrete alternative for that same step that would move the agent toward success (e.g., a specific action to take instead). Respond strictly in the following format (single spaces around colons, no extra text): step:<number> reason:<one concise, specific sentence> su...

  17. [2023]

    InEmpirical Methods in Natural Language Processing (EMNLP)

    Enabling large language models to generate text with citations. InEmpirical Methods in Natural Language Processing (EMNLP). Yu Ge, Linna Xie, Zhong Li, Yu Pei, and Tian Zhang

  18. [2025]

    Darshan Deshpande, Varun Gangal, Hersh Mehta, Jitin Krishnan, Anand Kannappan, and Rebecca Qian

    Swe-bench pro: Can ai agents solve long- horizon software engineering tasks?arXiv preprint arXiv:2509.16941. Darshan Deshpande, Varun Gangal, Hersh Mehta, Jitin Krishnan, Anand Kannappan, and Rebecca Qian

    arXiv

  19. [2026]

    arXiv preprint arXiv:2604.22708

    Seeing the whole elephant: A benchmark for failure attribution in llm-based multi-agent systems. arXiv preprint arXiv:2604.22708. Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, An- drew Park, Nitin Pasari, Chetan Rane, and 1 others. 9

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.