Artificial Intelligence · arXiv:2610.01256 v1 · submitted 1 Oct 2026 · 57 references
DeFA: Dependency-Guided Failure Attribution for LLM Agents
The paper says
“Errors in LLM agent executions and their visible consequences can be separated by many steps, making decisive-error localization a matter of understanding both step content and step dependencies. We introduce DeFA, a dependency-guided framework for agent failure attribution.”
more of the abstract
DeFA first combines protocol relations and semantic dependencies into an event dependency graph spanning the trajectory. It then identifies events that may violate task requirements and traces their sources and subsequent effects to construct a failure propagation graph. Finally, DeFA uses step evidence and the steps' roles in failure propagation to identify the decisive error, responsible agent, and error category. To support long trajectories, DeFA partitions executions into segments and combines the current segment's detailed content with summaries of the other segments, giving local diagnosis access to global execution context. Across Who and When and the Who and When Pro text subset, DeFA achieves the highest responsible-agent and exact step accuracy with all evaluated backbones, and the highest failure-mode accuracy among taxonomy-aligned methods on Pro. Further experiments on image and video trajectories demonstrate its applicability to multimodal failure attribution. Ablations support the contributions of segmentation, the event dependency graph, and the failure propagation graph. Using DeFA's diagnostic feedback for skill evolution in Trace2Skill improves downstream task accuracy by 6-15 percentage points over the native pipeline, showing that the diagnoses can also support agent improvement on subsequent tasks.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 2 Oct 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. Genuinely new dependency-graph attribution method with consistent directional gains and a useful downstream result, but the 'highest with all three backbones' claim is not statistically established per backbone and the paper needs significance testing and table repairs before the numbers can be taken at face value. This line is wrong
3 major objections · 6 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 57 references say.
The paper says, at Sections 4.2-4.3, Tables 1-2
The review did not quote this passage.
Pith says: Objection 1 of 3 · major
No measure of uncertainty (error bars, confidence intervals, or significance tests) is reported for any accuracy number, yet the abstract and Sections 4.2-4.3 claim that DeFA achieves the highest responsible-agent and exact-step accuracy 'with all three backbones.' Two-proportion checks against the numbers in Tables 1-2 show that most per-backbone gaps are within sampling noise: on Who&When (n=184) DeFA 52.7 vs …
more of objection 1
… ErrorProbe 43.5 gives roughly p≈0.08, DeepSeek 48.9 vs 39.1 gives p≈0.06, and Qwen 50.0 vs 42.4 gives p≈0.14; on Pro (n=300) only the GLM comparison (80.3 vs 72.0, p≈0.016) reaches conventional significance, while the DeepSeek and Qwen comparisons do not. The evidence is directionally consistent and the Pro-GLM and Trace2Skill results are individually significant, so this does not refute the method, but the central per-backbone claim is not statistically established. The authors should report bootstrap confidence intervals or a significance test (at least for the exact-Step metric) for each backbone and temper the 'all three backbones' phrasing accordingly.
The paper says, at Section 4.4, Figure 2c,d
The review did not quote this passage.
Pith says: Objection 2 of 3 · major
DeFA averages 8.73 LLM calls per trajectory versus 6.35-6.93 for the strongest baselines, roughly a 26% increase, and no compute-matched comparison is reported. As presented, part of DeFA's margin could reflect additional LLM calls rather than the dependency structure itself. A compute-matched or cost-normalized contrast (e.g., a budget-matched version of the strongest baseline, or Step accuracy per LLM call) is needed to support the attribution of the gains to the dependency-graph design.
The paper says, at Section 4.5, Table 3
The review did not quote this passage.
Pith says: Objection 3 of 3 · major
On the multimodal split, Agent accuracy is computed on only 17 trajectories, where DeFA 76.5 vs CockpitHAT 70.6 is a difference of a single trajectory; the claim of 'leading Agent ... with both backbones' (and similarly the 6-7 point Step gains on n=100) therefore rests on very small samples. This should be stated as a preliminary result, or the per-trajectory detail reported, rather than presented at the same confidence level as the text results.
6 minor objections
The paper says, at Table 1 (and Table 2, Table 3)
Pith says: minor objection 1 of 6
Several table entries run together without separators (for example Table 1 shows '46.272.8' and Table 2 shows '24.075.3' and '80.091.0'), making values hard to read; format the numbers so each cell is cleanly separated.
The paper says, at Section 4.1, Mode metric
Pith says: minor objection 2 of 6
The comparison set for Mode is only the three Who&When judges plus DeFA, since the other baselines do not adopt the Pro taxonomy; the 'taxonomy-aligned' qualifier used in the abstract and conclusions should also be applied consistently in the Section 4.3 discussion to avoid the impression of a broader comparison.
show 4 more
The paper says, at Figure 2
Pith says: minor objection 3 of 6
Figure 2c,d plots only a subset of the eight methods (three markers in 2c, one in 2d) with no legend identifying symbols; clarify which symbols correspond to which methods, or explain why the subset was selected.
The paper says, at Abstract and Section 5
Pith says: minor objection 4 of 6
The long-trajectory result (exact Step of only 17.2% on trajectories of 33-130 steps, acknowledged in Section 4.4) is striking and arguably belongs in the abstract alongside the headline gains, since it qualifies the method's practical range.
The paper says, at Section 5, Limitations
Pith says: minor objection 5 of 6
The paper acknowledges the dependence of semantic dependency recovery and failure propagation analysis on the model's trajectory interpretation (b163); an independent check of the recovered relation quality, even a small human-annotated sample of the dependency and propagation edges, would materially strengthen the paper's support for what is otherwise an internally consistent design.
The paper says, at Throughout
Pith says: minor objection 6 of 6
The benchmark is referred to as 'Who&When' in most places but 'Who and When' in the abstract and Section 1; unify the naming.
What would settle it
Recompute, per backbone and benchmark, Wilson 95% confidence intervals and exact two-proportion (or bootstrap, clustered by framework) p-values for DeFA versus the strongest baseline on Step, Agent, and Mode; and run a compute-matched DeFA capped at ≤7 LLM calls per trajectory against ErrorProbe and CockpitHAT. If, as a quick two-proportion check of the reported values suggests, most per-backbone Step gaps are not significant at 5% (Who&When: p≈0.06–0.14 on all three backbones; Pro: p≈0.13–0.26 on two of three) while the compute-matched DeFA loses its margin, then the 'highest accuracy with all three backbones' claim is not established, and the paper should report explicit uncertainty bounds and state which gains are significant.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 2 Oct 2026
- Reader holds up, with conditions
- Skeptic partly agrees
- Editor wrote the verdict above
- Referee major revision: 3 major objections, 6 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
Read this one for the method, not the headline numbers. DeFA combines deterministic protocol edges with LLM-induced semantic edges into an event dependency graph, builds a failure propagation graph, and adjudicates over segmented trajectories using cross-segment summaries. That combination is new relative to CockpitHAT and the other baselines, and it is clearly described. The most interesting result is downstream: DeFA's diagnoses plugged into Trace2Skill improve held-out test accuracy by 6–15 points (63→78 in the best cell, significant on n=100), showing the diagnoses are usable, not just better on a benchmark.
The ablations are decent. The same-coverage contrast (52.7 vs 40.2 Step at equal 83.2% coverage, Table 4) isolates the propagation edges' contribution from extra adjudication passes — the right control. The authors openly acknowledge the interpretation dependence (b163) and the weak long-trajectory localization (17.2%, b124). Citation pattern looks solid; related work covers the space.
Where it is soft:
- No error bars, confidence intervals, or significance tests anywhere. That is load-bearing. On Who&When (n=184), none of the three per-backbone exact-step gains clears 5% (DeepSeek p≈0.06, GLM p≈0.08, Qwen p≈0.14). On Pro (n=300), GLM is significant (p≈0.016) and Qwen is too against the right baseline (77.7 vs 69.3, p≈0.02) — the stress-test note used 72.3 there, which is DeepSeek's baseline, not Qwen's. DeepSeek (76.3 vs 72.3, p≈0.26) is not. So 'highest with all three backbones' is directionally consistent but only partially established.
- Compute: DeFA uses 8.73 LLM calls per trajectory vs 6.35–6.93 for the strongest baselines, and no compute-matched comparison is reported. The same-coverage ablation mitigates this for propagation edges specifically, but part of the overall margin is extra compute.
- Multimodal Agent accuracy rests on n=17; 76.5 vs 70.6 is a one-trajectory difference. Thin.
- Mode accuracy is modest in absolute terms (34–41%) and the comparison is only against the three Who&When judges, since the other baselines do not adopt the Pro taxonomy.
- Tables 1–3 have garbled merged cells ('46.272.8', '75.371.7'); values cannot be fully audited until fixed. Presentation, but real.
Who it is for: people working on LLM agent debugging, diagnosis, and agent evaluators. A solid within-subfield contribution, not a field-changer, with an honest limitations section. It deserves a real referee; the revision needs significance testing, table repairs, and ideally a compute-matched head-to-head before the headline claims are taken at face value.
The strongest objection
The central claim — DeFA achieves the highest responsible-agent and exact step accuracy with all three backbones on Who&When and Pro text — rests entirely on point estimates, with no error bars, confidence intervals, or significance tests reported anywhere. A quick two-proportion check of the stated values shows most per-backbone headline gaps are within noise. Who&When (n=184): DeFA 52.7 vs ErrorProbe 43.5 (z≈1.77, p≈0.08); DeepSeek 48.9 vs DeepDebug 39.1 (z≈1.90, p≈0.06); Qwen 50.0 vs CockpitHAT 42.4 (z≈1.47, p≈0.14). Pro (n=300): GLM 80.3 vs 72.0 is significant (p≈0.016), but DeepSeek 76.3 vs 72.3 (p≈0.26) and Qwen 77.7 vs 72.3 (p≈0.13) are not. Multimodal Agent accuracy is computed on n=17, where DeFA 76.5 vs CockpitHAT 70.6 is a one-trajectory difference. Complicating interpretation, DeFA averages 8.73 LLM calls per trajectory versus 6.35–6.93 for the strongest baselines (Figure 2c,d), so part of the margin could reflect extra compute rather than the dependency structure, and no compute-matched comparison is reported. The evidence is directionally consistent, and the Pro GLM Step gain and the Trace2Skill 63→78 (n=100) result are individually significant, so this does not refute the method; but the assertion 'highest with all three backbones' is not statistically established per backbone. The acknowledged interpretation dependence (b163) is real but secondary: the Table 4 same-coverage ablation (52.7 vs 40.2 Step at equal 83.2% coverage) reasonably supports the failure propagation graph's role, though no dependency-edge quality metric is reported, leaving graph correctness versus extra adjudicator passes as an untested confound.
The paper proposes a dependency-guided failure attribution framework and claims, as its central empirical result, the highest responsible-agent and exact step accuracy with all three evaluated backbones on Who&When and the Pro text subset, plus a downstream 6–15 point gain in Trace2Skill. For that claim to hold, the reported accuracy differences must be real and robust. My most load-bearing concern is that the reported numbers are bare point estimates with no uncertainty quantification, and a back-of-envelope two-proportion check of the stated values shows most per-backbone headline gaps are within conventional noise. Who&When (n=184) yields p≈0.08, 0.06, and 0.14 for the three backbones; on Pro (n=300) only the GLM exact-step margin reaches significance (p≈0.016), while DeepSeek and Qwen margins (p≈0.26, 0.13) do not. Multimodal Agent accuracy on n=17 turns the largest reported Agent gap into a one-trajectory difference. This is the least secure condition for the central claim, and it is concrete: recomputation with CIs and significance tests would settle it. I also note DeFA uses more LLM calls per trajectory (8.73 vs 6.35–6.93 for the strongest baselines), so a compute-matched comparison is needed before attributing the margin to the dependency structure; the paper reports the call counts transparently, so this is a fair-comparison gap rather than a misstatement. Directional consistency across three backbones, two benchmarks, multimodal data, and the downstream experiment, plus the individually significant Pro GLM and Trace2Skill gains, keeps this from overturning the paper. Credit is due for the Table 4 same-coverage ablation (52.7 vs 40.2 Step at equal 83.2% coverage), which is strong internal evidence that the propagation graph matters, and the authors openly acknowledge interpretation dependence in b163. The reader's identified weakest assumption (backbone's trajectory interpretation) is related but distinct; my focus is on the statistical robustness of the empirical claim itself, which is the more decisive gate for accepting the headline numbers, while the acknowledged mechanism dependence and the missing edge-quality metric remain as secondary, untested confounds. This supports keeping a CONDITIONAL verdict (require uncertainty bounds, significance statements, and a compute-matched comparison) rather than changing it.
The referee's report
If the results hold, DeFA is a useful contribution to agent failure attribution: the dependency-graph formulation is a principled way to connect a local deviation to the execution that made it consequential, and the design has real strengths. The controlled ablation in Table 4 (removing only propagation edges while holding the adjudication set and 83.2% coverage fixed still drops Step from 52.7 to 40.2) is a clean, interpretable experiment. The downstream Trace2Skill result (63 to 78 percentage points on held-out tasks with Qwen3.5-Plus) is measured on tasks not used for diagnosis, so the headline downstream gain is not circular. Evaluation across three backbones and two text benchmarks plus a multimodal extension broadens the evidence base, and the scoring follows the official benchmark protocols. The main risk is statistical: almost every per-backbone headline gap in the central claim is consistent with sampling noise at the reported sample sizes, and the method uses about 26% more LLM calls than the strongest baselines, so the compute-normalized advantage is not established.
Pith's reading of the paper (a machine's summary, not the authors')
DeFA is a method for locating the decisive error in a failed LLM agent execution: the earliest wrong step whose correction would make the task succeed, along with the responsible agent and the error's category. The paper's thesis is that a step's content alone cannot pin down this error when cause and effect are separated by many steps; what matters is the step's place in the flow of information, meaning where it received its inputs and what later steps did with its outputs. To capture that, DeFA builds an event dependency graph from protocol and semantic relations, flags steps that may violate the task, and traces their sources and effects into a failure propagation graph used for final adjudication. The claimed payoff is concrete: across three backbones DeFA beats seven baselines in responsible-agent and exact-step accuracy on Who&When and the Who&When Pro text subset, improving exact-step localization by 7.6 to 9.8 points and 4.0 to 8.3 points respectively, and its diagnostic feedback improves downstream skill evolution by 6 to 15 points.
What carries the argument. The operating mechanism is a pair of directed graphs over trajectory steps. The event dependency graph combines protocol edges, such as task delegation to its execution or a tool request to its result, which are recovered by a deterministic interpreter from LLM-induced rules, with semantic edges capturing later steps using earlier content or conclusions, including cross-segment edges that connect a step in one segment to later segments that continue using its information. The failure propagation graph then selects the relevant dependency edges and infers additional propagation edges that explain how a suspicious event's information was used, amplified, or corrected by later execution. For long trajectories, DeFA partitions execution into segments and gives each segment's analysis a view pairing that segment's full content with narrative summaries of all other segments, so local diagnosis still sees global context. The final adjudicator selects the decisive step from the propagation graph's vertex set under the criterion that correcting it, with later execution continuing from the corrected state, would turn failure into success.
Core claim. The paper's central claim is that failure attribution for LLM agents is best done along the dependency structure of the execution rather than on isolated steps. DeFA reconstructs an event dependency graph that records how steps pass and use instructions, results, and state, traces candidate error events backward to their sources and forward to their consequences to form a failure propagation graph, and then adjudicates the decisive error by comparing how steps introduce, transmit, or adopt errors rather than by local suspiciousness alone. Across three backbones the method reports the highest responsible-agent and exact-step accuracy on Who&When and the Who&When Pro text subset, the highest failure-mode accuracy among taxonomy-aligned methods on Pro, and the same advantage on multimodal image and video trajectories. The ablation the authors treat as decisive is that removing only the propagation edges, while holding the adjudication set fixed, drops exact step accuracy from 52.7% to 40.2%, which they take as supporting the role of propagation relations in identifying the decisive error.
The premise it rests on. The whole pipeline depends on the backbone LLM correctly recovering the semantic dependencies and failure-propagation edges from the trajectory text, and the authors acknowledge that missing or misidentified relations can break attribution, especially in long executions where evidence is dispersed, yet the paper offers no independent check on relation quality.
If the paper is right (Pith's inferences, not the authors')
A dependency-structured judge is a better basis for decisive-error localization than global, sequential, or binary-search judging, improving exact step accuracy by 7.6 to 9.8 points on Who&When and 4.0 to 8.3 points on the Pro text subset across three backbones.
The propagation graph, not candidate recall alone, carries much of the gain: removing only the propagation edges drops exact step accuracy by 12.5 points while keeping coverage at 83.2%.
Dependency-guided attribution transfers to trajectories with image and video evidence, improving exact step accuracy by 6.0 to 7.0 points on multimodal Pro trajectories.
Diagnoses produced this way are usable for agent improvement, raising downstream test accuracy by 6 to 15 percentage points over the native Trace2Skill pipeline.
The advantage holds across trajectory lengths, with the largest gains at intermediate lengths, while exact localization on the longest trajectories remains hard at 17.2%.
The Mode results (34-41%) sit far below the Step results, suggesting that error-category classification, not localization, is the current frontier; a natural extension is to attach failure-type evidence to each propagation edge instead of classifying only the decisive step's category.
Because relation quality is never independently checked, the method's ceiling may be set by the backbone's dependency recovery; a testable extension is to score recovered edges against a small gold dependency set and use that score to gate or re-rank adjudication.
The adjudicator's criterion that correcting the chosen step would make execution succeed could be operationalized directly with patch-and-replay evidence from intervention-based attribution methods, giving an independent check on the decisive step.
The segmentation-with-summaries trick is likely what keeps the method cheap enough to compete on long traces; whether full cross-segment content would close the 17.2% gap on the longest trajectories remains an open question the paper does not test.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Errors in LLM agent executions and their visible consequences can be separated by many steps, making decisive-error localization a matter of understanding both step content and step dependencies. We introduce DeFA, a dependency-guided framework for agent failure attribution. DeFA first combines protocol relations and semantic dependencies into an event dependency graph spanning the trajectory. It then identifies events that may violate task requirements and traces their sources and subsequent effects to construct a failure propagation graph. Finally, DeFA uses step evidence and the steps' roles in failure propagation to identify the decisive error, responsible agent, and error category. To support long trajectories, DeFA partitions executions into segments and combines the current segment's detailed content with summaries of the other segments, giving local diagnosis access to global execution context. Across Who and When and the Who and When Pro text subset, DeFA achieves the highest responsible-agent and exact step accuracy with all evaluated backbones, and the highest failure-mode accuracy among taxonomy-aligned methods on Pro. Further experiments on image and video trajectories demonstrate its applicability to multimodal failure attribution. Ablations support the contributions of segmentation, the event dependency graph, and the failure propagation graph. Using DeFA's diagnostic feedback for skill evolution in Trace2Skill improves downstream task accuracy by 6-15 percentage points over the native pipeline, showing that the diagnoses can also support agent improvement on subsequent tasks.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-10-02
The authors' main claim, as the classifier picked it out 0.84
“Across Who and When and the Who and When Pro text subset, DeFA achieves the highest responsible-agent and exact step accuracy with all evaluated backbones, and the highest failure-mode accuracy among taxonomy-aligned methods on Pro.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.07 · states limitation 0.10 · human participants 0.05 · controlled comparison 0.82 · randomized assignment 0.02 · states quantitative result 0.98 · compares with prior results 0.94 · mentions code or data availability 0.03
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Errors in LLM agent executions and their visible consequences can be separated by many steps, making decisive-error localization a matter of understanding both step content and step dependencies. | background 0.97 | 0.08 | 0.04 | asserted 0.99 |
| We introduce DeFA, a dependency-guided framework for agent failure attribution. | contribution 1.00 | 0.90 | 0.07 | asserted 0.99 |
| DeFA first combines protocol relations and semantic dependencies into an event dependency graph spanning the trajectory. | method 0.99 | 0.84 | 0.03 | asserted 1.00 |
| It then identifies events that may violate task requirements and traces their sources and subsequent effects to construct a failure propagation graph. | method 1.00 | 0.85 | 0.04 | asserted 0.98 |
| Finally, DeFA uses step evidence and the steps' roles in failure propagation to identify the decisive error, responsible agent, and error category. | method 0.99 | 0.89 | 0.12 | asserted 1.00 |
| To support long trajectories, DeFA partitions executions into segments and combines the current segment's detailed content with summaries of the other segments, giving local diagnosis access to global execution context. | method 0.98 | 0.87 | 0.04 | asserted 0.99 |
| Across Who and When and the Who and When Pro text subset, DeFA achieves the highest responsible-agent and exact step accuracy with all evaluated backbones, and the highest failure-mode accuracy among taxonomy-aligned methods on Pro. | result 1.00 | 0.96 | 0.42 | asserted 1.00 |
| Further experiments on image and video trajectories demonstrate its applicability to multimodal failure attribution. | result 0.90 | 0.93 | 0.05 | asserted 1.00 |
| Ablations support the contributions of segmentation, the event dependency graph, and the failure propagation graph. | result 0.94 | 0.93 | 0.06 | asserted 1.00 |
| Using DeFA's diagnostic feedback for skill evolution in Trace2Skill improves downstream task accuracy by 6-15 percentage points over the native pipeline, showing that the diagnoses can also support agent improvement on subsequent tasks. | result 0.85 | 0.94 | 0.21 | asserted 1.00 |
What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.98
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-02: 1 request, 7,504 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (3)
Figures are the authors', reproduced from the paper.
References (57)
- [1] Shinn, Noah and Cassano, Federico and Gopinath, Ashwin and Narasimhan, Karthik and Yao, Shunyu , booktitle =. 2023 , doi = on Pith
- [2] Madaan, Aman and Tandon, Niket and Gupta, Prakhar and Hallinan, Skyler and Gao, Luyu and Wiegreffe, Sarah and Alon, Uri and Dziri, Nouha and Prabhumoye, Shrimai and Yang, Yiming and Gupta, Shashank and Majumder, Bodhisattwa Prasad and Hermann, Katherine and Welleck, Sean and Y... on Pith
- [3] Ni, Jingwei and Liu, Yihao and Liu, Xinpeng and Sun, Yutao and Zhou, Mengyu and Cheng, Pengyu and Wang, Dexin and Zhao, Erchao and Jiang, Xiaoxi and Jiang, Guanjun , journal =. 2026 , doi = on Pith
- [4] Ma, Zeyao and Zhang, Bohan and Zhang, Jing and Yu, Jifan and Zhang, Xiaokang and Zhang, Xiaohan and Luo, Sijia and Wang, Xi and Tang, Jie , booktitle =. 2024 , doi = on Pith
- [5] Lost in the Middle: How Language Models Use Long Contexts , author =. Transactions of the Association for Computational Linguistics , volume =. 2024 , doi = on Pith
- [6] Wang, Wei and Liu, Shuanghe and Zhuo, Zhu and Zhong, Jiaqi and Zhao, Xiaozhao and Zuo, Xiaojie and Su, Jie , journal =. 2026 , doi = on Pith
- [7] Yao, Shunyu and Zhao, Jeffrey and Yu, Dian and Du, Nan and Shafran, Izhak and Narasimhan, Karthik and Cao, Yuan , booktitle =. 2023 , url = on Pith
- [8] Wu, Qingyun and Bansal, Gagan and Zhang, Jieyu and Wu, Yiran and Li, Beibin and Zhu, Erkang and Jiang, Li and Zhang, Xiaoyun and Zhang, Shaokun and Liu, Jiale and Awadallah, Ahmed and White, Ryen W. and Burger, Doug and Wang, Chi , booktitle =. 2024 , url = on Pith
- [9] Cemri, Mert and Pan, Melissa Z. and Yang, Shuyi and Agrawal, Lakshya A. and Chopra, Bhavya and Tiwari, Rishabh and Keutzer, Kurt and Parameswaran, Aditya and Klein, Dan and Ramchandran, Kannan and Zaharia, Matei A. and Gonzalez, Joseph E. and Stoica, Ion , booktitle =. Why Do ... on Pith
- [10] Zheng, Lianmin and Chiang, Wei-Lin and Sheng, Ying and Zhuang, Siyuan and Wu, Zhanghao and Zhuang, Yonghao and Lin, Zi and Li, Zhuohan and Li, Dacheng and Xing, Eric P. and Zhang, Hao and Gonzalez, Joseph E. and Stoica, Ion , booktitle =. Judging. 2023 , doi = on Pith
- [11] Let's Verify Step by Step , author =. International Conference on Learning Representations , year = on Pith
- [12] Zheng, Chujie and Zhang, Zhenru and Zhang, Beichen and Lin, Runji and Lu, Keming and Yu, Bowen and Liu, Dayiheng and Zhou, Jingren and Lin, Junyang , booktitle =. 2025 , doi = on Pith
- [13] Ruan, Yangjun and Dong, Honghua and Wang, Andrew and Pitis, Silviu and Zhou, Yongchao and Ba, Jimmy and Dubois, Yann and Maddison, Chris J. and Hashimoto, Tatsunori , booktitle =. Identifying the Risks of. 2024 , url = on Pith
- [14] Agent-as-a-Judge: Evaluate Agents with Agents , author =. Proceedings of the 42nd International Conference on Machine Learning , pages =. 2025 , url = on Pith
- [15] Ou, Tianyue and Guo, Wanyao and Gandhi, Apurva and Neubig, Graham and Yue, Xiang , booktitle =. 2025 , doi = on Pith
- [16] Zhang, Shaokun and Yin, Ming and Zhang, Jieyu and Liu, Jiale and Han, Zhiguang and Zhang, Jingyang and Li, Beibin and Wang, Chi and Wang, Huazheng and Chen, Yiran and Wu, Qingyun , booktitle =. Which Agent Causes Task Failures and When? On Automated Failure Attribution of. 202... on Pith
- [17] Deshpande, Darshan and Gangal, Varun and Mehta, Hersh and Krishnan, Jitin and Kannappan, Anand and Qian, Rebecca , journal =. 2025 , doi = on Pith
- [18] Barke, Shraddha and Goyal, Arnav and Khare, Alind and Singh, Avaljot and Nath, Suman and Bansal, Chetan , journal =. 2026 , doi = on Pith
- [19] Liu, Jiale and Xi, Huajun and Zhang, Shaokun and Zeng, Yifan and Yue, Tianwei and Wang, Chi and Kang, Jian and Wu, Qingyun and Wang, Huazheng , journal =. 2026 , doi = on Pith
- [20] Zhang, Guibin and Wang, Junhao and Chen, Junjie and Zhou, Wangchunshu and Wang, Kun and Yan, Shuicheng , booktitle =. 2026 , url = on Pith
show 37 more
- [21] Spectrum-Based Failure Attribution for Multi-agent Systems , author =. Proceedings of the ACM on Software Engineering , volume =. 2026 , doi = on Pith
- [22] Towards Self-Improving Error Diagnosis in Multi-Agent Systems , author =. Findings of the Association for Computational Linguistics: ACL 2026 , pages =. 2026 , doi = on Pith
- [23] Qiao, Hezhe and Tong, Hanghang and Lim, Ee-Peng and Liu, Bing and Pang, Guansong , journal =. 2026 , doi = on Pith
- [24] Zhu, Kunlun and Ye, Xuyan and Han, Zhiguang and Zhao, Yuchen and Li, Bingxuan and Zhang, Weijia and Tian, Muxin and Tang, Xiangru and Lu, Pan and Zou, James and You, Jiaxuan and Ji, Heng , journal =. 2026 , doi = on Pith
- [25] Lin, Xiaofeng and Wang, Yingxu and Kwok, Tung Sum Thomas and Guo, Daniel and Nale, Sahil Arun and Fleming, Charles and Cheng, Guang , journal =. 2026 , doi = on Pith
- [26] Bonagiri, Akash and Borkar, Devang and Anderias, Gerard Janno and Rafatirad, Setareh and Homayoun, Houman , journal =. 2026 , doi = on Pith
- [27] Guo, Dongxin and Wu, Jikun and Yiu, Siu Ming , journal =. 2026 , doi = on Pith
- [28] Zhao, Xiangxin and Li, Han and Li, Shuaiting and Zhao, Tianyi and Barr, Earl T. and Sarro, Federica and Ye, He , journal =. Failure as a Process: An Anatomy of. 2026 , doi = on Pith
- [29] Hou, Jun and Pitre, Priya and Fang, Yi and Wang, Xuan , journal =. 2026 , url = on Pith
- [30] Shraddha Barke, Arnav Goyal, Alind Khare, Avaljot Singh, Suman Nath, and Chetan Bansal. AgentRx : Diagnosing AI agent failures from execution trajectories. arXiv preprint arXiv:2602.02475, 2026. doi:10.48550/arXiv.2602.02475. URL https://arxiv.org/abs/2602.02475. Accepted to F... on PithPith's reviewarXivdoi:10.48550/arxiv.2602.02475
- [31] Akash Bonagiri, Devang Borkar, Gerard Janno Anderias, Setareh Rafatirad, and Houman Homayoun. CausalFlow : Causal attribution and counterfactual repair for LLM agent failures. arXiv preprint arXiv:2605.25338, 2026. doi:10.48550/arXiv.2605.25338. URL https://arxiv.org/abs/2605.25338 on PithPith's reviewarXivdoi:10.48550/arxiv.2605.25338
- [32] Mert Cemri, Melissa Z. Pan, Shuyi Yang, Lakshya A. Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, Matei A. Zaharia, Joseph E. Gonzalez, and Ion Stoica. Why do multi-agent LLM systems fail? In Advances in Neural Informa... on PithPith's reviewarXiv
- [33] Darshan Deshpande, Varun Gangal, Hersh Mehta, Jitin Krishnan, Anand Kannappan, and Rebecca Qian. TRAIL : Trace reasoning and agentic issue localization. arXiv preprint arXiv:2505.08638, 2025. doi:10.48550/arXiv.2505.08638. URL https://arxiv.org/abs/2505.08638 on PithPith's reviewarXivdoi:10.48550/arxiv.2505.08638
- [34] Yu Ge, Linna Xie, Zhong Li, Yu Pei, and Tian Zhang. Spectrum-based failure attribution for multi-agent systems. Proceedings of the ACM on Software Engineering, 3 0 (FSE): 0 1888--1909, 2026. doi:10.1145/3797113. URL https://doi.org/10.1145/3797113 on Pithdoi:10.1145/3797113
- [35] Jun Hou, Priya Pitre, Yi Fang, and Xuan Wang. EDGE : Error dependency graph-guided multi-error attribution in multi-agent LLM systems. arXiv preprint arXiv:2609.01360, 2026. URL https://arxiv.org/abs/2609.01360 on PithPith's reviewarXiv
- [36] Jiazheng Li, Emine Yilmaz, Bei Chen, and Thu Le. Towards self-improving error diagnosis in multi-agent systems. In Findings of the Association for Computational Linguistics: ACL 2026, pp.\ 2063--2077. Association for Computational Linguistics, 2026. doi:10.18653/v1/2026.findin... on Pithdoi:10.18653/v1/2026.findings-acl.98
- [37] Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=v8L0pN6EOi on Pith
- [38] Xiaofeng Lin, Yingxu Wang, Tung Sum Thomas Kwok, Daniel Guo, Sahil Arun Nale, Charles Fleming, and Guang Cheng. REFLECT : Intervention-supported error attribution for silent failures in LLM agent traces. arXiv preprint arXiv:2606.09071, 2026. doi:10.48550/arXiv.2606.09071. URL... on PithPith's reviewarXivdoi:10.48550/arxiv.2606.09071
- [39] Jiale Liu, Huajun Xi, Shaokun Zhang, Yifan Zeng, Tianwei Yue, Chi Wang, Jian Kang, Qingyun Wu, and Huazheng Wang. Who&When Pro : Can LLM s really attribute failures in AI agents? arXiv preprint arXiv:2607.09996, 2026. doi:10.48550/arXiv.2607.09996. URL https://arxiv.org/abs/2607.09996 on PithPith's reviewarXivdoi:10.48550/arxiv.2607.09996
- [40] Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12: 0 157--173, 2024. doi:10.1162/tacl_a_00638... on Pithdoi:10.1162/tacl_a_00638
- [41] Zeyao Ma, Bohan Zhang, Jing Zhang, Jifan Yu, Xiaokang Zhang, Xiaohan Zhang, Sijia Luo, Xi Wang, and Jie Tang. SpreadsheetBench : Towards challenging real world spreadsheet manipulation. In Advances in Neural Information Processing Systems, volume 37, 2024. doi:10.52202/079017-... on Pithdoi:10.52202/079017-3007
- [42] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-Refine : Iterati... on Pithdoi:10.52202/075280-2019
- [43] Jingwei Ni, Yihao Liu, Xinpeng Liu, Yutao Sun, Mengyu Zhou, Pengyu Cheng, Dexin Wang, Erchao Zhao, Xiaoxi Jiang, and Guanjun Jiang. Trace2Skill : Distill trajectory-local lessons into transferable agent skills. arXiv preprint arXiv:2603.25158, 2026. doi:10.48550/arXiv.2603.251... on PithPith's reviewarXivdoi:10.48550/arxiv.2603.25158
- [44] Tianyue Ou, Wanyao Guo, Apurva Gandhi, Graham Neubig, and Xiang Yue. AgentDiagnose : An open toolkit for diagnosing LLM agent trajectories. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp.\ 207--215. Associat... on Pithdoi:10.18653/v1/2025.emnlp-demos.15
- [45] Hezhe Qiao, Hanghang Tong, Ee-Peng Lim, Bing Liu, and Guansong Pang. VerifyMAS : Hypothesis verification for failure attribution in LLM multi-agent systems. arXiv preprint arXiv:2605.17467, 2026. doi:10.48550/arXiv.2605.17467. URL https://arxiv.org/abs/2605.17467 on PithPith's reviewarXivdoi:10.48550/arxiv.2605.17467
- [46] Yangjun Ruan, Honghua Dong, Andrew Wang, Silviu Pitis, Yongchao Zhou, Jimmy Ba, Yann Dubois, Chris J. Maddison, and Tatsunori Hashimoto. Identifying the risks of LM agents with an LM -emulated sandbox. In International Conference on Learning Representations, 2024. URL https://... on Pith
- [47] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion : Language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems, volume 36, 2023. doi:10.52202/075280-0377. URL https://papers.nips.cc/paper_... on Pithdoi:10.52202/075280-0377
- [48] Wei Wang, Shuanghe Liu, Zhu Zhuo, Jiaqi Zhong, Xiaozhao Zhao, Xiaojie Zuo, and Jie Su. CockpitHAT : Dependency-graph-driven hierarchical attribution for embodied multi-agent cockpits. arXiv preprint arXiv:2608.01805, 2026. doi:10.48550/arXiv.2608.01805. URL https://arxiv.org/a... on PithPith's reviewarXivdoi:10.48550/arxiv.2608.01805
- [49] Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Awadallah, Ryen W. White, Doug Burger, and Chi Wang. AutoGen : Enabling next-gen LLM applications via multi-agent conversations. In First Conference... on Pith
- [50] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct : Synergizing reasoning and acting in language models. In International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=WE_vluYUL-X on Pith
- [51] Guibin Zhang, Junhao Wang, Junjie Chen, Wangchunshu Zhou, Kun Wang, and Shuicheng Yan. AgenTracer : Who is inducing failure in the LLM agentic systems? In International Conference on Learning Representations, 2026. URL https://proceedings.iclr.cc/paper_files/paper/2026/hash/13... on Pith
- [52] Shaokun Zhang, Ming Yin, Jieyu Zhang, Jiale Liu, Zhiguang Han, Jingyang Zhang, Beibin Li, Chi Wang, Huazheng Wang, Yiran Chen, and Qingyun Wu. Which agent causes task failures and when? on automated failure attribution of LLM multi-agent systems. In Proceedings of the 42nd Int... on Pith
- [53] Xiangxin Zhao, Han Li, Shuaiting Li, Tianyi Zhao, Earl T. Barr, Federica Sarro, and He Ye. Failure as a process: An anatomy of CLI coding agent trajectories. arXiv preprint arXiv:2607.09510, 2026. doi:10.48550/arXiv.2607.09510. URL https://arxiv.org/abs/2607.09510 on PithPith's reviewarXivdoi:10.48550/arxiv.2607.09510
- [54] Chujie Zheng, Zhenru Zhang, Beichen Zhang, Runji Lin, Keming Lu, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. ProcessBench : Identifying process errors in mathematical reasoning. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguisti... on Pithdoi:10.18653/v1/2025.acl-long.50
- [55] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM -as-a-judge with MT -bench and chatbot arena. In Advances in Neural Information Proce... on PithPith's reviewarXiv
- [56] Kunlun Zhu, Xuyan Ye, Zhiguang Han, Yuchen Zhao, Bingxuan Li, Weijia Zhang, Muxin Tian, Xiangru Tang, Pan Lu, James Zou, Jiaxuan You, and Heng Ji. AgentDebugX : An open-source toolkit for failure observability, attribution, and recovery in LLM agents. arXiv preprint arXiv:2607... on PithPith's reviewarXivdoi:10.48550/arxiv.2607.18754
- [57] Mingchen Zhuge, Changsheng Zhao, Dylan R. Ashley, Wenyi Wang, Dmitrii Khizbullin, Yunyang Xiong, Zechun Liu, Ernie Chang, Raghuraman Krishnamoorthi, Yuandong Tian, Yangyang Shi, Vikas Chandra, and J \"u rgen Schmidhuber. Agent-as-a-judge: Evaluate agents with agents. In Procee... on Pith
Cited by 3 Pith papers
Reviewed papers on Pith that cite this one. The full citation record
- Self-Evolving Agents Can Repair Failures but Struggle to Generalize from Them cs.CL · Oct 2026
- Securing and Auditing Long-Horizon LLM Agents: State-Transfer Auditing and Controls in the Agent Harness cs.CR · Nov 2026
- Dependable Long-Horizon LLM Agents: A Trajectory-Centered Empirical Study of Where Errors Enter and Which Controls Reach Them cs.AI · Nov 2026
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment