Pith. sign in

REVIEW 3 major objections 5 minor 23 references

A warm-start injection pipeline builds 12,326 golden-labeled agent failure traces across text, image, and video, and under that test frontier LLMs still cannot reliably name who failed, when, and why.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 01:09 UTC pith:65Z2O6G3

load-bearing objection Warm-start injection at 12k multimodal scale is a real methodological step up; models still fail hard on failure-mode diagnosis, and the “golden earliest step” claim is a bit cleaner in the paper than under Zhang’s strict counterfactual reading. the 3 major comments →

arxiv 2607.09996 v1 pith:65Z2O6G3 submitted 2026-07-10 cs.AI cs.MA

Who&When Pro: Can LLMs Really Attribute Failures in AI Agents?

Jiale Liu , Huajun Xi , Shaokun Zhang , Yifan Zeng , Tianwei Yue , Chi Wang , Jian Kang , Qingyun Wu
show 1 more author
Huazheng Wang
This is my paper
classification cs.AI cs.MA
keywords failure attributionLLM agentsmulti-agent systemserror injectionmultimodal agentsdecisive errorself-evolving agentsagent debugging
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

As AI agents get stronger, their mistakes get subtler, so the field needs machines that can say which agent went wrong, at which step, and in what way. This paper’s answer is Who&When Pro: take a trajectory that already succeeded, restore the exact state just before a chosen step, force a single taxonomy-conditioned bad action, then let the real agent run on—and keep only the runs that now fail. Because the only controlled change was that one action, the decisive agent, step, and error mode become golden labels. The resulting 12,326 traces span 26 source benchmarks, nine task families, and three modalities. When frontier closed- and open-weight models try to attribute those failures, step localization peaks around 74% on text but error-mode diagnosis stays near 22% F1, joint accuracy is lower still, and behavior shifts systematically with modality, protocol, and model family. The authors treat the benchmark and those patterns as a foundation for building better automatic debuggers and self-improving agents.

Core claim

Strict warm-start error injection after exact replay of a successful prefix yields large-scale golden labels for decisive agent, step, and failure mode; evaluated on the resulting 12,326 multimodal traces, current LLMs remain far from reliable automated failure attribution, with especially weak mode classification and clear difficulty on long and video traces.

What carries the argument

Warm-start decisive-error injection: from a successful seed trajectory, restore agent and environment state through step t−1 (via cache or fidelity-checked replay), substitute one taxonomy-conditioned erroneous action, resume the original system, and retain only post-injection failures—so the injected step is, by construction, the earliest decisive error.

Load-bearing premise

That restoring state and forcing a single later failure is enough to guarantee the injected step was truly the earliest decisive error, and that these synthetic modes fairly represent the failures an attribution system must catch in the wild.

What would settle it

A larger human audit or counterfactual “fix the injected action and re-run” study showing that a substantial fraction of retained traces already had an earlier decisive error, or that judges trained on Who&When Pro systematically fail on naturally occurring (non-injected) agent failures.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Full-trajectory (all-at-once) attribution should be preferred over step-by-step or binary-search protocols for both accuracy and cost.
  • Open-weight models can sit on the cost–accuracy Pareto frontier for practical attribution backbones.
  • Step localization is hardest on video and long traces; mode classification is relatively easier once visual or behavioral cues appear.
  • Providing the task’s ground-truth answer helps diagnose perception errors but can degrade diagnosis of reasoning errors by inviting outcome shortcuts.
  • Future self-evolving agents can use externally verified step-level labels of this form rather than coarse terminal or self-graded feedback.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If warm-start fidelity ever leaks pre-injection drift at scale, the “golden” labels become approximate and the entire ranking of models on the benchmark softens.
  • The surface-symptom bias (planning/verification/coordination misread as reasoning) suggests attribution judges may need explicit causal or counterfactual prompting, not only larger models.
  • A natural next stress test is transfer: train or prompt on injected traces and measure recovery of natural failures from the same agent frameworks.
  • Taxonomy extensibility matters: new embodied or long-horizon collaborative settings will likely require new modes before the same pipeline stays complete.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces WHO&WHENPRO, a large-scale benchmark for automated failure attribution in LLM agentic systems. Using a warm-start pipeline that replays a successful trajectory prefix, injects a single taxonomy-conditioned erroneous action, and continues execution, the authors construct 12,326 failed traces with labels for decisive agent, step, and failure mode across 26 source benchmarks, 9 task categories, and text/image/video modalities. They evaluate closed- and open-weight frontier models under all-at-once, step-by-step, and binary-search protocols, reporting that attribution remains difficult (e.g., best text step accuracy ~73.9% but error-mode F1 only ~22.2%), with systematic effects of modality, protocol, trace length, and ground-truth availability, plus cost–performance trade-offs favoring open-weight models under all-at-once evaluation.

Significance. If the label construction is accepted as high-fidelity, this is a substantial contribution: prior failure-attribution benchmarks are small and almost entirely text-only, while WHO&WHENPRO expands scale, modality coverage, and topology coverage in a way that matches modern agent deployments. The warm-start design is a clear methodological improvement over re-rollout injection that can drift before the injection point. The empirical findings—modality-dependent bottlenecks, symptom-vs-root-cause confusion, full-trajectory superiority, and open-weight cost efficiency—are actionable for both attribution systems and self-evolving agents. Strengths include a carefully engineered pipeline (tool caching, fidelity checks), stratified human review with substantial agreement, multi-protocol evaluation, and public release of traces/labels/code.

major comments (3)
  1. [§3.3, Fig. 2, decisive-error definition] §3.3 and the decisive-error definition (opening of §3, following Zhang et al.): the construction shows that replacing the injected action ~a_t with the original a_t and restoring the successful continuation recovers success. That establishes the injection as a controlled cause of failure, but it is not identical to the formal earliest-decisive-step claim on the observed failed trajectory (a1..a_{t-1}, ~a_t, a'_{t+1}, …). Post-injection rollouts can introduce additional independent errors, especially in stateful browser/code settings (Appendix F), so correcting only ~a_t while keeping later observed actions need not yield success, and the earliest correctable index could differ. The paper should either (i) reframe labels as “first controlled deviation / injection locus under warm-start construction” rather than unqualified golden earliest-decisive labels, or (ii) provide counterfactual ch
  2. [§3.5, Table 3] §3.5 / Table 3: human validation covers 100 stratified traces (~0.8% of 12,326). Annotators largely accept the generated step/agent/mode labels (94%/90%/90%, Fleiss κ=0.73), which supports recognizability, but the protocol does not test counterfactual earliest-correctability or measure how often an earlier latent fragility or a later independent failure would change the label. Given that “golden labels” are the central selling point relative to prior automatic pipelines (Table 1), the manuscript should expand validation—at least on a larger stratified sample and with an explicit counterfactual or “would fixing this step alone recover success?” criterion—or clearly bound the claim to human-recognized injection loci rather than full decisive-error semantics.
  3. [§3.2–3.3, Table 4, Fig. 4] §3.2–3.3 and experimental interpretation: failure modes are profiled from natural failures then re-injected into successful seeds. This is a reasonable scalability design, but the main results (Table 4, Fig. 4) are then used to conclude how models attribute failures in agentic systems generally. The paper should quantify or bound distribution shift between natural and injected failures (e.g., side-by-side human comparison of natural vs injected traces for the same modes, or attribution performance on a held-out natural-failure subset with expert labels). Without that, claims about “systematic patterns” risk being specific to synthetic injection signatures rather than organic agent failures.
minor comments (5)
  1. [§4.1 Metrics, Table 4] Table 4 and §4.2: Joint is described as requiring Agent, Step, and Error all correct, but Agent is only defined on multi-agent traces. Clarify how Joint is computed on single-agent traces (76% of the data per Fig. 3)—e.g., whether Agent is treated as vacuously correct.
  2. [Fig. 4b, Fig. 10] Fig. 4b / Fig. 10: confusion matrices average over models; also report per-model or at least variance, since the text claims a general symptom-vs-root-cause pattern.
  3. [§4.3, Table 6] §4.3: the ablation subset of 1,444 traces is stratified, but the main paper should state the sampling procedure and confirm that Table 6 trends match full-set all-at-once results for the overlapping setting.
  4. [Appendix F] Appendix F fidelity checks: report quantitative replay-abort rates by environment class (static tools vs browser vs code) so readers can assess residual selection bias in retained traces.
  5. [Table 5, Appendix J] Minor polish: consistent naming (WHO&WHENPRO / Who&When Pro), and ensure all 18 mode codes in Table 5 align with the evaluation prompt taxonomy block without renumbering ambiguity.

Circularity Check

1 steps flagged

No load-bearing circularity: golden labels are explicitly by construction of warm-start injection, while reported LLM metrics are external measurements against those fixed labels; only mild self-citation of the decisive-error definition from overlapping authors.

specific steps
  1. self definitional [§3.3 Scalable Failure Trace Generation (and Abstract / Fig. 2)]
    "If the resulting trajectory fails, it enters the benchmark: reverting ˜at recovers the successful seed, so t is the decisive step by definition. ... Since the injected error is the only controlled change that turns a successful trajectory into a failed one, the resulting task failure can be attributed exactly to the injected agent and step, thus yielding golden labels under the decisive-error definition by (Zhang et al., 2025c)."

    The paper equates the injected step with the golden decisive step purely by the construction of the pipeline (warm-start + single injection + post-injection failure). This is definitional rather than an independent derivation; however, because the paper presents the labels as synthetic ground truth for a new benchmark (not as a predicted scientific quantity), the circularity is only mild and does not force the subsequent LLM accuracy numbers, which remain external matches against those labels.

full rationale

This is an empirical benchmark-construction and evaluation paper, not a first-principles derivation of a physical or mathematical prediction. The decisive-error labels (agent, step, mode) are defined operationally: a successful seed is warm-started, a single taxonomy-conditioned action is substituted at step t, and any resulting task failure is retained with t as the golden decisive step because “reverting ˜at recovers the successful seed, so t is the decisive step by definition” (§3.3). That is intentional synthetic-label design, not a hidden reduction of an independent claim to its own inputs. Model scores (Agent/Step/Error/Joint accuracies, macro-F1, cost-Pareto, protocol ablations) are then computed by exact match or F1 against these fixed labels using held-out LLMs that did not generate the labels; the numbers are therefore ordinary external measurements, not statistically forced by a fit. The 18-mode taxonomy is distilled from natural failures then used only to constrain injection prompts, which is ordinary author-defined categorization rather than a tautology that forces the ranking of models. The sole mild circularity-adjacent element is adoption of the decisive-error definition from Zhang et al. (2025c), whose author list overlaps the present paper; that citation supplies the evaluation target but is not a uniqueness theorem or ansatz that forbids alternatives or manufactures the performance numbers. Human review (100 traces) and fidelity checks are validation, not circular self-support. Consequently the central empirical claims remain independent of any self-definitional loop, fitted-input-as-prediction, or load-bearing self-citation chain. Score 1 reflects only the minor definitional self-reference; a pure non-finding of 0 is also defensible.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 3 invented entities

The central results rest on the decisive-error definition, the claim that warm-start plus single injection isolates that error, an 18-mode taxonomy from expert review, and standard LLM-as-judge evaluation protocols. No physical free parameters; free choices are design knobs of the benchmark (injection sampling, filters, taxonomy). Invented entities are methodological constructs (benchmark, taxonomy, pipeline), not new physical objects.

free parameters (3)
  • Injection-step sampler preferences by error family
    Position preferences (e.g., planning early, verification late, perception at image observations) are design choices that shape the label distribution and difficulty; not derived from a uniqueness theorem.
  • Human-review sample size (n=100 stratified traces)
    Acceptance rates and κ depend on this sample; the paper does not report full-set human labels.
  • Ablation subset size (1,444 stratified traces)
    Protocol and cost comparisons use a fixed stratified slice rather than the full 12,326; results could shift with different sampling.
axioms (4)
  • domain assumption Decisive step t* is the earliest index such that correcting a_t* turns a failed trajectory into a successful one (Zhang et al. 2025c).
    Adopted as the labeling target throughout §3; all golden labels are defined relative to this notion.
  • ad hoc to paper Replaying (a1..a_{t-1}) with tool caches and fidelity checks restores context and environment state sufficiently that only the injected action causes the subsequent failure.
    Core of the warm-start pipeline (§3.3, Appendix F); if violated, labels are approximate like re-rollout methods.
  • domain assumption An 18-mode taxonomy spanning perception, reasoning, planning, action, verification, and coordination covers the dominant failure patterns for evaluation.
    Built by Ph.D. review of natural failures (§3.2, Appendix C); limitations section notes non-exhaustiveness.
  • domain assumption Official benchmark evaluators correctly partition seed trajectories into success vs failure.
    Source trajectory collection (§3.1) relies on each benchmark's official evaluator when available.
invented entities (3)
  • Who&When Pro benchmark (12,326 labeled failure traces) independent evidence
    purpose: Provide scalable golden labels for agent, step, and error-mode attribution across modalities.
    Primary artifact; independent of any single evaluated model.
  • Warm-start controlled error-injection pipeline no independent evidence
    purpose: Generate high-fidelity decisive-error labels without pre-injection re-rollout drift.
    Methodological construct; fidelity depends on caches and abort-on-drift checks.
  • 18-mode failure taxonomy with per-(agent,benchmark) profiles no independent evidence
    purpose: Constrain which synthetic errors are injected and how modes are scored.
    Author-consolidated categories; human transfer matrix shows residual confusion with reasoning.

pith-pipeline@v1.1.0-grok45 · 39470 in / 3209 out tokens · 31589 ms · 2026-07-14T01:09:31.281765+00:00 · methodology

0 comments
read the original abstract

Automated failure attribution uses LLMs to identify where and why agentic systems fail. As agents become more capable, their failures become subtler, making automated attribution increasingly important. We introduce Who&When Pro, a large-scale benchmark for automated failure attribution in agentic systems. Using a strictly controlled pipeline that injects a failure only after exactly replaying a successful prefix, we construct 12,326 failed trajectories with golden labels across 3 modalities and 26 benchmarks covering various scenarios. Beyond benchmarking, we conduct extensive experiments and analyses, revealing systematic patterns in how models attribute failures across modalities, protocols, and model families, and providing empirical guidance for future automated failure attribution systems.

Figures

Figures reproduced from arXiv: 2607.09996 by Chi Wang, HuaJun Xi, Huazheng Wang, Jiale Liu, Jian Kang, Qingyun Wu, Shaokun Zhang, Tianwei Yue, Yifan Zeng.

Figure 1
Figure 1. Figure 1: Overview of WHO&WHEN PRO. The inner ring groups 26 source benchmarks into 9 task categories. The outer ring maps each benchmark to its modality (text, image, video). Four corner panels illustrate failure attribution examples across modalities and agent topologies. (Top left) a video agent misidentifies a frame. (Top right) a multi-agent coding pipeline adopts a suboptimal architecture. (Bottom left) an ima… view at source ↗
Figure 2
Figure 2. Figure 2: WHO&WHEN PRO failure-injection pipeline. Agent trajectories collected from benchmarks across text, image, and video modalities are split into successes and failures, with failures used to build an error taxonomy. For each successful seed, an error generator injects a taxonomy-conditioned mistake and a warm-start rollout collects the traces that turned to failure. (Bottom right) (a) Previous methods using r… view at source ↗
Figure 3
Figure 3. Figure 3: Dataset composition. Distributions of trace length, modality, and agent topology. Human approved (%) No clear Agreement κ Step Agent Error (%) 94.0 90.0 90.0 2.0 0.73 [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Analysis of attribution difficulty. (a) Step accuracy drops sharply on video traces while mode classification improves on multimodal traces. (b) Failure mode confusion matrix on multimodal traces, averaged over seven vision-capable models. (c) Step accuracy for four representative models, grouped by trace length in tokens. as reasoning. Longer traces are harder to diagnose. We then grouped the traces accor… view at source ↗
Figure 5
Figure 5. Figure 5: Distribution of prediction distance using different methods. Distances are calculated by subtracting the LLM￾predicted error step from the ground-truth step. Diamonds (♦) mark the mean. onto the first suspicious step and cannot revise after observ￾ing later context. Binary search is directionally unbiased but equally imprecise, as slicing can remove the relevant context needed for accurate localization. 4.… view at source ↗
Figure 6
Figure 6. Figure 6: Cost vs. joint accuracy across attribution protocols. Each point is one (model, protocol) pair on the ablation subset. Shapes denote protocols; colors denote models. The dashed line marks the Pareto frontier. All-at-once dominates the low-cost regime in all three modalities. Image QA MM-Deep Search Video 20 40 60 80 Error Accuracy (%) +0.8 -1.6 +9.7 -4.0 +9.1 +4.6 w/o w/ (perception) w/ (reasoning) (a) Imp… view at source ↗
Figure 7
Figure 7. Figure 7: Impact of ground truth on error mode classification. Per-trace accuracy for perception (blue) and reasoning (orange) errors, evaluated on the multimodal part of the ablation subset. Arrows indicate the change from w/o G to w/ G. closed-source alternatives at a fraction of the cost. Qwen3.5 achieves the highest text accuracy at seven times cheaper than GPT-5.4, and Gemma 4 leads on video. Failure at￾tributi… view at source ↗
Figure 8
Figure 8. Figure 8: Case study 1: ground truth shifts diagnosis from process to outcome verification. Without G, both judges trace the agent’s reasoning process and correctly identify a task misunderstanding: the agent answered a different question than what was asked. With G, both judges anchor on the fact that the agent “had the correct date” (matching G = 2024-10-30) and reclassify the error as a reasoning error (misapplie… view at source ↗
Figure 9
Figure 9. Figure 9: Case study 2: ground truth causes the judge to skip the root-cause error. Without G, both judges identify the fabricated “Ray-Ban Meta” claim at Step 2 as a hallucination: the decisive error that derails the trajectory. With G (the gold date 2024-05-01), both judges skip Step 2 entirely and focus on the downstream collection confusion at Step 3, reclassifying the error as a reasoning error. The gold answer… view at source ↗
Figure 10
Figure 10. Figure 10: Fine-grained mode-level confusion matrix. Row-normalized confusion on image and video traces, averaged over seven vision-capable models. Labels are colored by error category. error becomes plausible. Early injections (0–20%) and late injections (80–100%) are less common, as planning errors are concentrated near the start and verification errors near the end [PITH_FULL_IMAGE:figures/full_fig_p020_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Annotator correction transfer matrix. Rows: generated error families; columns: annotator-corrected families. Row￾normalized, c over three annotators on 100 held-out traces. Diagonal borders colored by category. GPT-5.4 Claude Sonnet 4.6 Gemini 3 Flash Grok 4.1 gpt-oss-120b GLM-5 DeepSeek-V4-Pro Gemma 4 Llama-4 Maverick Qwen3.5-122B 60 65 70 75 80 85 90 95 100 Step Accuracy (%) Text Step@0 Step@1 Step@2 [… view at source ↗
Figure 12
Figure 12. Figure 12: Step@k on text traces. Step localization accuracy under tolerance windows k ∈ {0, 1, 2} for all ten evaluated models. E.2. Self-Evolving Agentic Systems Classical building blocks in self-evolving agents combine verbal reflection (Shinn et al., 2023; Madaan et al., 2023; Gou et al., 2024), self-rewarding judgement (Yuan et al., 2024), refinement tuning (Fu et al., 2025; Yuan et al., 2025), and bootstrapped… view at source ↗
Figure 13
Figure 13. Figure 13: Step@k on image traces. Seven vision-capable models evaluated on image-modality traces. GPT-5.4 Claude Sonnet 4.6 Gemini 3 Flash Grok 4.1 Gemma 4 Llama-4 Maverick Qwen3.5-122B 45 50 55 60 65 70 75 80 85 90 95 100 Step Accuracy (%) Video Step@0 Step@1 Step@2 [PITH_FULL_IMAGE:figures/full_fig_p022_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Step@k on video traces. Seven vision-capable models evaluated on video-modality traces. The Step@0→Step@1 gain is largest here (+28.1 pp on average), reflecting dense step adjacency in video trajectories. sandboxes (Zhou et al., 2026), large-scale computer-use engines (Awadallah et al., 2025), unified data protocols (Song et al., 2026), and scalable web-environment synthesis (Zhang et al., 2026e). Closer … view at source ↗
Figure 15
Figure 15. Figure 15: Injection position distribution. Percentage of traces by relative injection position (injected step index divided by total trace length). 0 20 40 60 % within modality MetaGPT DyLAN MacNet ALFAgent Debate Magentic-One MathChat smolagents 1% 2% 4% 5% 7% 13% 15% 52% Text 0 20 40 % within modality Debate AgentOccam CoAct OpenAI CUA Gemini smolagents PixelCraft 2% 2% 5% 8% 10% 27% 47% Image 0 25 50 75 % within… view at source ↗
Figure 16
Figure 16. Figure 16: Framework distribution by modality. Percentage of traces per agent framework within each modality. F.2. Base Agent Model We use a mixture of frontier LLMs as the backbone models powering the agentic systems. The two primary models are GPT-4.1 (OpenAI, 2025b) and Gemini 3 Flash (Google DeepMind, 2025). GPT-4.1 is the strongest non-reasoning model at the time of data collection: its reasoning chains are int… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

23 extracted references · 11 linked inside Pith

  1. [1]

    URL https: //doi.org/10.1038/s41586-023-06792-0

    doi: 10.1038/s41586-023-06792-0. URL https: //doi.org/10.1038/s41586-023-06792-0. Bran, A. M., Cox, S., Schilter, O., Baldassari, C., White, A. D., and Schwaller, P. Chemcrow: Augmenting large- language models with chemistry tools.arXiv preprint arXiv:2304.05376, 2023. Cemri, M., Pan, M. Z., Yang, S., Agrawal, L. A., Chopra, B., Tiwari, R., Keutzer, K., P...

    Pith/arXiv arXiv doi:10.1038/s41586-023-06792-0 2023

  2. [2]

    Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H

    URL https://openreview.net/forum? id=fAjbYBmonr. Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y ., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021. Chen, Y ., Wang, Y ., Zhu, S., Yu, H., Feng, T., Zhang, M., Patwary, M., and You, J. Mult...

    Pith/arXiv arXiv doi:10.18653/v1/2025.emnlp-main 2021

  3. [3]

    emnlp-main.839/

    URL https://aclanthology.org/2025. emnlp-main.839/. 9 Who&When Pro: Can LLMs Really Attribute Failures in AI Agents? Ge, Y ., Xie, L., Li, Z., Pei, Y ., and Zhang, T. Who is intro- ducing the failure? automatically attributing failures of multi-agent systems via spectrum analysis.arXiv preprint arXiv:2509.13782, 2025b. Geng, X., Xia, P., Zhang, Z., Wang, ...

    arXiv 2025

  4. [4]

    com/deepmind-media/Model-Cards/ Gemini-3-Flash-Model-Card.pdf

    URL https://storage.googleapis. com/deepmind-media/Model-Cards/ Gemini-3-Flash-Model-Card.pdf . Offi- cial model card. Accessed: 2026-05-05. Google DeepMind. Gemma 4 Model Card, April

    2026

  5. [5]

    Gou, Z., Shao, Z., Gong, Y ., yelong shen, Yang, Y ., Duan, N., and Chen, W

    URL https://ai.google.dev/gemma/ docs/core/model_card_4. Gou, Z., Shao, Z., Gong, Y ., yelong shen, Yang, Y ., Duan, N., and Chen, W. CRITIC: Large language models can self-correct with tool-interactive critiquing. InThe Twelfth International Conference on Learning Represen- tations, 2024. URL https://openreview.net/ forum?id=Sx038qxjek. Grijalba, J. O., ...

    Pith/arXiv arXiv doi:10.18653/v1/2024.acl-long 2024

  6. [6]

    acl-long.371/

    URL https://aclanthology.org/2024. acl-long.371/. Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring math- ematical problem solving with the math dataset.NeurIPS, 2021. Hong, S., Zhuge, M., Chen, J., Zheng, X., Cheng, Y ., Wang, J., Zhang, C., Wang, Z., Yau, S. K. S., Lin, Z., et al. Metagpt: Me...

    Pith/arXiv arXiv 2024

  7. [7]

    Lee, Y ., Nair, R., Zhang, Q., Lee, K., Khattab, O., and Finn, C

    URL https://openreview.net/forum? id=fZfUdeCC5X. Lee, Y ., Nair, R., Zhang, Q., Lee, K., Khattab, O., and Finn, C. Meta-harness: End-to-end optimization of model harnesses, 2026. URL https://arxiv.org/abs/ 2603.28052. Li, Y ., Inan, H. A., Yue, X., Chen, W.-N., Wutschitz, L., Kulkarni, J., Poovendran, R., Sim, R., and Rajmohan, S. Simulating environments ...

    Pith/arXiv arXiv doi:10.18653/v1/2025.findings-emnlp 2026

  8. [8]

    findings-emnlp.1034/

    URL https://aclanthology.org/2025. findings-emnlp.1034/. Liu, J., Bursztyn, V ., Ai, L., Wang, H., Choudhary, S., Mi- tra, S., and Wu, Q. Teamfusion: Supporting open-ended teamwork with multi-agent systems. InProceedings of the 64th Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pp. 14435– 14456, 2026. Liu, Z., ...

    2025

  9. [9]

    Lou, X., Lázaro-Gredilla, M., Dedieu, A., Wendelken, C., Lehrach, W., and Murphy, K

    URL https://openreview.net/forum? id=XII0Wp1XA9. Lou, X., Lázaro-Gredilla, M., Dedieu, A., Wendelken, C., Lehrach, W., and Murphy, K. P. Autoharness: improving llm agents by automatically synthesizing a code harness. arXiv preprint arXiv:2603.03329, 2026. Luo, H., Dai, S., Ni, C., Li, X., Zhang, G., Wang, K., Liu, T., and Salam, H. Agentauditor: Human-lev...

    arXiv 2026

  10. [10]

    Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y ., Gupta, S., Majumder, B

    URL https://openreview.net/forum? id=mrEK16Jy6h. Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y ., Gupta, S., Majumder, B. P., Hermann, K., Welleck, S., Yazdanbakhsh, A., and Clark, P. Self-refine: Iter- ative refinement with self-feedback. InThirty-seventh Conference on Neural Informa...

  11. [11]

    Majgaonkar, O., Fei, Z., Li, X., Sarro, F., and Ye, H

    URL https://openreview.net/forum? id=S37hOerQLB. Majgaonkar, O., Fei, Z., Li, X., Sarro, F., and Ye, H. Understanding code agent behaviour: An empirical study of success and failure trajectories.arXiv preprint arXiv:2511.00197, 2025. Masry, A., Islam, M. S., Ahmed, M., Bajaj, A., Kabir, F., Kartha, A., Laskar, M. T. R., Rahman, M., Rahman, S., Shahmohamma...

    arXiv 2025

  12. [12]

    Official system card

    URL https://openai.com/index/ gpt-5-4-thinking-system-card/ . Official system card. Accessed: 2026-05-05. Pan, L., Zou, L., Guo, S., Ni, J., and Zheng, H.- T. Natural-language agent harnesses.arXiv preprint arXiv:2603.25723, 2026. Qian, C., Xie, Z., Wang, Y ., Liu, W., Zhu, K., Xia, H., Dang, Y ., Du, Z., Chen, W., Yang, C., Liu, Z., and Sun, M. Scaling l...

    Pith/arXiv arXiv 2026

  13. [13]

    Song, L., Liu, J., Zhang, J., Zhang, S., Luo, A., Wang, S., Wu, Q., and Wang, C

    URL https://openreview.net/forum? id=0IOX0YcCdTn. Song, L., Liu, J., Zhang, J., Zhang, S., Luo, A., Wang, S., Wu, Q., and Wang, C. Adaptive in-conversation team building for language model agents.arXiv preprint arXiv:2405.19425, 2024. Song, L., Dai, Y ., Prabhu, V ., Zhang, J., Shi, T., Li, L., Li, J., Savarese, S., Chen, Z., Zhao, J., Xu, R., and Xiong, ...

    Pith/arXiv arXiv 2024

  14. [14]

    Wu, X., Yang, J., Chai, L., Zhang, G., Liu, J., Du, X., Liang, D., Shu, D., Cheng, X., Sun, T., et al

    URL https://openreview.net/forum? id=BAakY1hNKS. Wu, X., Yang, J., Chai, L., Zhang, G., Liu, J., Du, X., Liang, D., Shu, D., Cheng, X., Sun, T., et al. Tablebench: A comprehensive and complex benchmark for table ques- tion answering. InProceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 25497–25506, 2025. Wu, Y ., Jia, F., Zhang,...

    Pith/arXiv arXiv 2025

  15. [15]

    Official model card

    URL https://data.x.ai/ 2025-11-17-grok-4-1-model-card.pdf . Official model card. Accessed: 2026-05-05. Xia, C. S., Deng, Y ., Dunn, S., and Zhang, L. Agentless: De- mystifying llm-based software engineering agents.arXiv preprint arXiv:2407.01489, 2024. Xia, C. S., Wang, Z., Yang, Y ., Wei, Y ., and Zhang, L. Live- swe-agent: Can software engineering agent...

    Pith/arXiv arXiv 2025

  16. [16]

    Yang, J., Jimenez, C

    URL https://openreview.net/forum? id=EEgYUccwsV. Yang, J., Jimenez, C. E., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., and Press, O. Swe-agent: Agent- computer interfaces enable automated software engineer- ing.Advances in Neural Information Processing Systems, 37:50528–50652, 2024. Yang, K., Liu, Y ., Chaudhary, S., Fakoor, R., Chaudhari, P., Karypi...

    Pith/arXiv arXiv 2024

  17. [17]

    ISBN 979-8-89176-251-0

    Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long

    doi:10.18653/v1/2025.acl-long 2025

  18. [18]

    structurally misunderstood

    URL https://aclanthology.org/2025. acl-long.736/. Zeng, A., Liu, M., Lu, R., Wang, B., Liu, X., Dong, Y ., and Tang, J. Agenttuning: Enabling generalized agent abilities for llms, 2023. Zhai, Y ., Tao, S., Chen, C., Zou, A., Chen, Z., Fu, Q., Mai, S., Yu, L., Deng, J., Cao, Z., Liu, Z., Ding, B., and Zhou, J. Agentevolver: Towards efficient self-evolving ...

    Pith/arXiv arXiv 2025

  19. [19]

    /tcg-pocket-october-2025-producer-letter) Obs:Official Release Date: October 30, 2024.Platform: iOS and Android

    2web_visit(pokemon.com/. . . /tcg-pocket-october-2025-producer-letter) Obs:Official Release Date: October 30, 2024.Platform: iOS and Android. . . The game is approaching itsfirst anniversary in late October 2025. . . 3† Reasoning:“. . . interpreting ‘arrive’ as when the game’s big new rollout / anniversary content arrives (rather than its original launch)...

    2025

  20. [20]

    Late October 2025

    ” final_answer(“Late October 2025.”) †Injected step. The agent reinterprets “arrive” as anniversary content rather than the original launch date. w/oGw/G Model Pred. Rationale (excerpt) Pred. Rationale (excerpt) GPT-5.4 Task Misund. “The agentstructurally misunderstoodthe user’s question by reinterpreting ‘arrive’ as ask- ing about anniversary content. . ...

    2025

  21. [21]

    Binary Search Protocol The binary search protocol bisects the transcript to narrow down the error location

    Reason: (your explanation) J.3. Binary Search Protocol The binary search protocol bisects the transcript to narrow down the error location. Each call presents a contiguous slice of steps and asks the model whether the error is more likely in the upper or lower half of that slice. The runner recursively halves the range until convergence. Binary Search Eva...

  22. [22]

    Half: upper or lower

  23. [23]

    {{ query }}

    Reason: (one sentence) K. Injection Prompt Templates As described in Section 3.3, we construct each corrupted action ˜at in two stages. First, a frontier model reads the seed trace’s context up to stept and generates anadaptive injection promptconditioned on the target failure mode. Second, this 32 Who&When Pro: Can LLMs Really Attribute Failures in AI Ag...

    2024