{"id":"41448673-efd9-41f9-aeb6-61a54cafa983","arxiv_id":"2608.06346","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A three-stage error-lifecycle tracing framework localizes the earliest failure-responsible step in long agent trajectories and outperforms prior debuggers on a new 486-trajectory benchmark.","lead":"TrajDebug breaks failed long-horizon agent runs into evidence-backed error triggers, tracks which errors are repaired or remain active, and attributes the final failure to the earliest responsible step. It also ships TrajErrBench, 486 annotated failed trajectories, and reports gains over prior diagnostic methods and improvements in downstream agent success.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"TrajErrBench ground truth may be anchored by model pre-annotations and TrajDebug-aligned taxonomy; Table 1's 34.11% average and the +10.80 repair gain rest substantially on this benchmark.","rationale":"I read the paper as making two connected claims: (1) TrajDebug localizes the critical error step better than existing methods on a broad suite, and (2) those diagnoses translate into measurable downstream agent improvement. The second claim is supported by consistent, format-matched gains across three settings, though without significance tests. The first claim is the vulnerable one. The new benchmark provides the largest share of evaluation trajectories and the only long-horizon coding data, yet its ground truth was produced with model pre-annotations and a taxonomy aligned with TrajDebug. This does not make the result circular in a formal sense, but it creates a real risk that the benchmark rewards methods that mirror its annotation conventions. Because the reader already identified this as the weakest assumption and recommended a conditional verdict, I do not change the verdict; I would make the re-annotation check an explicit acceptance condition. Existing-benchmark wins and the ablation study are genuinely useful independent evidence, but they do not cover the long-horizon SWE-Bench Pro regime, so the anchoring question remains the main open risk.","tokens_in":28132,"tokens_out":7388,"duration_ms":78304,"concrete_test":"Take a stratified random sample of 100 TrajErrBench trajectories (50 τ2-Bench, 50 SWE-Bench Pro) and re-annotate them with fresh annotators who receive no model pre-annotations and only a neutral counterfactual critical-error definition, with no mention of TrajDebug's taxonomy. Compute exact-step and ±3-step agreement between the new labels and the released labels. If exact agreement on the retained subset is below about 80% (or ±3 agreement below about 90%), the anchor hypothesis is supported; then recompute Table 1 on the re-annotated subset for TrajDebug, Gemini-3.1-Pro, and Claude-Opus-4.6. If TrajDebug's margin over the best baseline shrinks below the noise floor or reverses, the full benchmark must be re-labeled before the central claim can be accepted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing comparison in Table 1 rests on TrajErrBench: 486 of 869 evaluated trajectories come from it, and it is the only source for the long-horizon tool-use and SWE-Bench Pro claims. Appendix B.2 shows annotators received model-assisted pre-annotations from GPT-5.4, Gemini-3.1-Pro, and Claude-Opus-4-6, plus a guideline whose reference categories and execution-phase concepts match TrajDebug's trigger taxonomy. Table 4 reports exact-match agreement up to 76.8%, but that only shows no single model dominates; it cannot rule out a collective anchor. If pre-annotations bias humans toward steps a taxonomy-driven detector would also find, the benchmark partly measures reproduction of its own annotation conventions. This is load-bearing because the headline 34.11% macro-average and the +10.80 per-trajectory repair gain depend on TrajDebug's high scores on exactly these two new datasets (52.75% and 24.41%); a bias of a few points could move the margin over Gemini-3.1-Pro (31.29%) or Claude-Opus-4.6 (29.40%). TrajDebug's wins on existing benchmarks give partial independent support, but the long-horizon SWE-Bench Pro result has no independent ground truth. A clean re-annotation experiment is required to settle whether the benchmark is anchored.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TrajDebug, a three-stage framework for critical error detection in long-horizon LLM agent trajectories: multi-granularity history compression, evidence-grounded error trigger detection, error state classification (Clean/Costly Resolution, Manifest/Latent Active), and candidate-set-guided causal attribution. It also introduces TrajErrBench, a benchmark of 486 manually annotated failed trajectories from τ2-Bench and SWE-Bench Pro. Experiments on seven benchmark subsets report a macro-average exact critical-step accuracy of 34.11%, the best among direct-prompting and multi-agent baselines, and application studies report that TrajDebug diagnoses improve downstream agent success by 10.80 points in per-trajectory repair and 5.70 points in failure-memory transfer. The paper includes ablations, length-based analysis, a compute-matched majority-voting baseline, a format-matched feedback comparison, and a stage-wise error decomposition.","tokens_in":28482,"tokens_out":8039,"duration_ms":84102,"significance":"If the results hold, the paper makes a useful contribution: it reframes critical error detection as an error-lifecycle tracing problem, provides a new long-horizon benchmark in tool-use and code domains, and demonstrates a practical feedback loop for improving agents. The paper also follows several good evaluation practices: it includes a compute-matched direct-prompting baseline (Table 10), a format-matched feedback comparison (Table 11), a component ablation (Table 2), and a detailed error analysis (Tables 12–13). These design choices strengthen the empirical core. The main risk is that the new benchmark's ground truth may be influenced by the authors' own taxonomy and by LLM pre-annotations, which would partially undermine the headline comparison.","major_comments":[{"comment":"The model-assisted pre-annotation protocol raises a load-bearing anchoring concern. Exact-match rates between pre-annotations and final human-majority labels reach 76.8% (Gemini-3.1-Pro on τ2-Bench) and are 53–77% across models, and the annotation guideline's criteria ('earliest evidence-grounded mistake,' 'failure-responsible,' reference categories, execution phases) closely match TrajDebug's trigger and state taxonomy in §3.2–3.3. The paper argues that no single model dominates, but that does not rule out collective anchoring: the final labels may substantially reflect the same LLM judgments and annotation conventions that TrajDebug is designed to reproduce. Because 486 of 869 evaluated trajectories come from TrajErrBench, and the SWE-Bench Pro long-horizon result (24.41%) and the headline macro-average (34.11%) depend on it, the benchmark may partly measure reproduction of its own annotation conventions. I recommend a clean re-annotation of a random subset without pre-annotations, a comparison of label stability under both protocols, and reporting Table 1 results with TrajErrBench excluded or with alternative label sets.","section":"Appendix B.2, Table 4"},{"comment":"The central comparative claims lack uncertainty quantification. The macro-average margin over Gemini-3.1-Pro is 2.82 points, on per-dataset samples as small as 50 (GAIA, WebShop) and 58 (WhoAndWhen hand-crafted); no confidence intervals, bootstrap estimates, or paired significance tests are reported. Likewise, the per-trajectory repair gains in Table 3 (e.g., +10.00 vs +8.00 for Vanilla on Airline) may be within sampling noise. Please report confidence intervals or significance tests, or at least per-subset bootstrap intervals, for the main comparison and the application study.","section":"Table 1, Table 3"},{"comment":"The method relies on ad-hoc thresholds — k=50 for budget debt, th1/th2/th3 compression caps (3000/1200/600 characters), and the 800-character attribution truncation — with no sensitivity analysis. Since Table 2 shows that removing multi-granularity compression causes the largest drop (21.02 points), the headline results could depend on these specific values. Please add a sensitivity sweep over k (e.g., 30/50/70) and at least one compression cap, and report whether the ranking in Table 1 is stable.","section":"§3.3, Appendix D"},{"comment":"The SWE-Bench Pro evaluation set is the 86 of 110 initially annotated trajectories on which at least two annotators agreed on the critical step; the 24 discarded ambiguous cases are not included in any result, and inter-annotator agreement on the retained set is only κ=0.67. Excluding the hardest cases likely makes the benchmark easier and may inflate TrajDebug's 24.41% figure relative to a deployment distribution. Please report results on the full 110-trajectory set (or analyze sensitivity to the majority-vote threshold) and discuss how exclusion affects the long-horizon claim.","section":"Appendix B.2"}],"minor_comments":[{"comment":"The column header 'A VG' should be 'AVG' for clarity.","section":"Table 1"},{"comment":"The compression tiers th1/th2/th3 are introduced in prose but only concretely defined in Appendix D; consider defining them in §3.1 where they are first used.","section":"§3.1, Appendix D"},{"comment":"The 'Environment Anomaly' trigger category in TrajDebug is not part of the TrajErrBench annotation schema, which only lists Task Conflict, History Conflict, and Intra-Step Conflict; please clarify how env-anomaly triggers are evaluated against the benchmark's ground-truth labels.","section":"§3.2, Appendix B.2"},{"comment":"The y-axis label 'Individual Model Step Accuracy (%)' is ambiguous; the caption indicates this is the average across seven models, so please align the axis label with the caption.","section":"Figure 2"},{"comment":"The accuracy values 26.00, 20.02, 24.73 are non-monotonic as the per-step limit increases; a sentence interpreting this non-monotonicity would help readers understand the trade-off.","section":"Appendix E.3, Table 8"}],"recommendation":"major_revision","confidential_remarks":"The benchmark-anchoring concern is the main risk to the paper's central claim. I would not reject, because TrajDebug also wins on several independent existing benchmarks and the downstream success experiments do not depend on TrajErrBench labels, but the headline macro-average and the long-horizon SWE-Bench Pro result do depend on the new benchmark. A clean re-annotation study or a sensitivity analysis excluding TrajErrBench would substantially de-risk the contribution. The paper is otherwise well executed and clearly written."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper to know about: TrajDebug, a three-stage pipeline for locating the earliest failure-responsible step in long agent trajectories. The genuinely new piece is the error-lifecycle state classifier—resolved vs active, terminal footprint vs not—and the ablations show it does real work. That moves the field past the taxonomy flagging and causal-graph attribution of prior work.\n\nWhat's solid: the empirical core. Seven datasets, full prompts in the appendix, a compute-matched direct-prompting majority-vote baseline (40x tokens, still 4.5 points behind), length analysis, failure decomposition, and two application studies. The downstream success numbers are the strongest evidence: +10.8 points per-trajectory repair and +5.7 transfer are measured by actual re-execution, not by the benchmark labels. The format-matched control in Table 11 makes the repair comparison fair. And the paper is transparent about limitations.\n\nThe soft spots. First, TrajErrBench annotation: annotators were given pre-annotations from three frontier models plus a guideline whose categories (task/history/intra-step conflict, failure-responsible, earliest decisive step) mirror TrajDebug's own taxonomy. Table 4 shows no single model dominates, but it cannot rule out collective anchoring. This is load-bearing mainly for the SWE-Bench Pro result (24.41% vs 17.44% for the best direct baseline), which has no independent ground truth. The wins on ALFWorld, GAIA, and WhoAndWhen-Algo are independent of this benchmark, so the method isn't riding entirely on its own annotation conventions. Second, no significance testing; the 34.11% macro-average over Gemini's 31.29% could be noise. Third, code and data are promised but not released, so nothing is machine-checkable yet. Fourth, exact-step accuracy is low in absolute terms, and token consumption is ~40x a single direct prompt. The compute-matched baseline mitigates that, but cost matters for deployment.\n\nI disagree with the stress-test on one point: it claims the +10.8 repair gain depends on the anchored benchmark. It doesn't. That experiment re-executes tasks with feedback, independent of TrajErrBench labels. Still, the anchoring question is real and testable. A clean re-annotation without pre-annotations, or an intervention study that verifies the detector's fix, would settle it.\n\nBottom line: this is for the agent-reliability subfield, and it deserves a serious referee. I'd send it out with a request for the re-annotation and significance testing. I'd also bring it to the next reading group—the benchmark protocol is a good case study.","headline":"A genuinely new error-lifecycle classifier with solid experiments, but the new benchmark's labels may be partly anchored to the method's own taxonomy; worth publishing with a clean re-annotation.","tokens_in":29035,"tokens_out":5421,"would_cite":true,"duration_ms":53655,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that finding the failure-responsible step in long agent runs should be decomposed into trigger detection, error-state classification, and attribution.","keywords":["critical error detection","error lifecycle","agent trajectory debugging","long-horizon LLM agents","error attribution","trajectory repair","failure memory","TrajErrBench"],"falsifier":"Re-annotate a random sample of TrajErrBench trajectories from the raw failed trajectories alone, without model pre-annotations and without the framework's category names; if the resulting majority-vote labels diverge substantially from the released labels, or if TrajDebug's margin over direct prompting shrinks when scored against the blind labels, the reported advantage is partly an artifact of label anchoring.","tokens_in":27942,"feed_emoji":"🐞","tokens_out":9334,"duration_ms":93468,"temperature":0.7,"pith_summary":"Long agent trajectories fail in cascades, and the paper's claim is that the step truly responsible for the failure can be found by following each local error through its lifecycle instead of asking an LLM to judge the whole trajectory at once. TrajDebug first detects per-step 'error triggers'—wrong commitments that conflict verbatim with a task constraint, a prior observation, or the agent's own reasoning—then groups repeated triggers into error instances anchored to the same violated reference. It classifies each instance by whether the wrong commitment was repaired and whether it left an observable terminal footprint, so repaired or dormant errors drop out and only terminal-relevant candidates reach a final attribution step. The authors build TrajErrBench, 486 manually annotated failed trajectories from tool-use and coding domains, and report that this staged pipeline reaches 34.11% macro-average exact-step accuracy, the best in their comparison, with the largest edge on long trajectories. They also show the diagnoses are actionable: per-trajectory repair with TrajDebug's feedback lifts average success from 78.07% to 88.87%.","feed_headline":"TrajDebug finds the earliest lethal error in agent runs","feed_subtitle":"Tracing each mistake's lifecycle beats direct prompting and lifts repaired-run success from 78% to 89%.","key_machinery":"The load-bearing object is the error instance $E=(E,O)$, a cluster of triggers that share the same violated reference object $O$; each trigger is a tuple $(t,c,p,q_w,q_r)$ recording the step, reference category, execution phase, the wrong commitment, and the verbatim violated reference. Multi-granularity compression builds high-, medium-, and low-detail views of each step so that distant history stays compact while local verification can cite verbatim evidence. The decisive mechanism is the two-axis error-state classification (resolved vs. active; terminal footprint vs. none), which yields four states—Clean Resolution, Costly Resolution, Manifest Active, Latent Active—and restricts the final LLM attribution head to the candidate set $F(\\tau)$ of costly-resolved and manifest-active instances.","core_discovery":"The paper's central claim is that critical error detection in long-horizon agent trajectories should be modeled as error-lifecycle tracing: an error is born as an evidence-grounded trigger at a specific step, persists as an instance tied to a violated reference object, and either gets repaired, stays active, leaves a terminal footprint, or does not. TrajDebug implements this as three stages—trigger detection, state classification, and candidate-set-guided causal attribution—and the paper reports that this decomposition outperforms both direct LLM prompting and multi-agent diagnostic baselines on exact critical-step accuracy across seven datasets. The strongest result is on long horizons: on SWE-Bench Pro (average 119.7 steps), TrajDebug reaches 24.41% exact-step accuracy versus 17.44% for the best direct-prompting baseline, and it retains over 20% accuracy on the longest length bucket where baselines fall to about 14%. In application experiments, feeding TrajDebug's diagnosis back to the executing agent improves re-execution success by 10.80 points on average, and a transfer setting that aggregates diagnoses into failure memory improves held-out tasks by 5.70 points.","pith_inferences":["Editorial inference: If TrajErrBench's human labels are partly anchored by the model pre-annotations and shared taxonomy used in annotation, the reported head-to-head margins may overstate TrajDebug's independent causal accuracy; blind re-annotation would resolve this.","Editorial inference: The lifecycle-state schema (resolved vs active, footprint vs none) is a transferable traceability vocabulary that could be used as a standard for agent observability and failure postmortems beyond the specific detector.","Editorial inference: A controlled stress-test of the method's attribution stage would replace the LLM attribution head with a deterministic rule, such as 'earliest manifest-active candidate,' to quantify how much of the gain comes from the final LLM judgment versus earlier evidence filtering."],"forward_implications":["Exact critical-step detection on failed agent trajectories can be improved by roughly 8.4 accuracy points over holistic prompting with the same backbone, without extra inference budget, across embodied, web, tool-use, and coding domains.","On trajectories longer than 100 steps, TrajDebug degrades more gracefully than direct prompting—remaining above 20% exact-step accuracy where the best baseline falls to around 14%.","Critical-error diagnoses translate into measurable agent improvement: re-running the same task with the diagnosis injected raises average success from 78.07% to 88.87%, and aggregated failure memory raises held-out task success by 5.70 points on average.","Ablation evidence indicates that multi-granularity history compression is the largest single contributor, removing it costs 21.02 average-accuracy points, so preserving evidence across distance matters more than adding more LLM calls.","Because the method retains the true critical step in its candidate set only 42.0% of the time, further gains depend mainly on improving trigger recall and final attribution, not on state filtering."],"supporting_citations":[{"why":"Defines critical error detection and the decisive-error counterfactual that TrajDebug adopts, and supplies the WhoAndWhen benchmark plus the annotation convention used for TrajErrBench.","marker":"[Zhang et al., 2025c]"},{"why":"Provides AgentDebugBench trajectories and the AgentDebugger taxonomy baseline against which TrajDebug is compared.","marker":"[Zhu et al., 2025]"},{"why":"Source of the 400 τ2-Bench trajectories in TrajErrBench and of the airline/retail application scenarios used for repair feedback.","marker":"[Barres et al., 2025]"},{"why":"Source of the 86 SWE-Bench Pro trajectories in TrajErrBench, the longest-horizon coding evaluations.","marker":"[Deng et al., 2025]"},{"why":"AgentRX baseline that flags constraint-violating steps; its constraint-based approach is the main alternative TrajDebug must beat.","marker":"[Barke et al., 2026]"},{"why":"CHIEF baseline, a hierarchical causal-graph counterfactual method that TrajDebug compares against and improves on average.","marker":"[Wang et al., 2026]"},{"why":"Self-Reflection baseline in the application studies; isolates the contribution of localization versus generic reflection.","marker":"[Renze and Guven, 2024]"}],"fun_headline_variants":["Trace error lifecycles to find critical agent failures","Error lifecycle tracing spots earliest lethal agent errors","TrajDebug: error lifecycle beats direct prompting for agent debug","Find the fatal step with error lifecycle tracing"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation stands on the assumption that TrajErrBench's human ground-truth labels are causally correct, even though annotators were given model pre-annotations and a guideline whose error categories match TrajDebug's own taxonomy, which could make the benchmark partly measure how well the method reproduces its annotation conventions.","fun_headline_variants_meta":{"raw":{"variants":["Trace error lifecycles to find critical agent failures","Error lifecycle tracing spots earliest lethal agent errors","TrajDebug: error lifecycle beats direct prompting for agent debug","Find the fatal step with error lifecycle tracing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000627,"raw_usage":{"total_tokens":2933,"prompt_tokens":1011,"completion_tokens":1922,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":627,"completion_tokens_details":{"reasoning_tokens":1862}},"tokens_in":627,"tokens_out":1922,"duration_ms":15308,"temperature":1.0,"reasoning_tokens":1862,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:43:24.683470+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate a random sample of TrajErrBench trajectories from the raw failed trajectories alone, without model pre-annotations and without the framework's category names; if the resulting majority-vote labels diverge substantially from the released labels, or if TrajDebug's margin over direct prompting shrinks when scored against the blind labels, the reported advantage is partly an artifact of label anchoring.","supporting_citations":[],"review_version":1}