{"id":"663b7323-a64b-4094-b376-84e56ca0feda","arxiv_id":"2607.09996","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.5,"correctness_risk":"low","formal_verification":"none","parameter_count":3,"one_line_summary":"A warm-start error-injection pipeline yields 12,326 golden-labeled multimodal agent failures, and current LLMs remain weak at step-and-mode failure attribution.","lead":"Who&When Pro builds 12,326 labeled agent failure traces by injecting one controlled error after replaying a successful prefix, spanning text, image, and video. It shows frontier LLMs still struggle to name who failed, when, and why—especially error modes and long multimodal traces.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"Warm-start + post-injection failure does not guarantee the injected step is the earliest decisive error under Zhang et al.'s definition.","rationale":"The reader's weakest_assumption already isolates exactly this point (warm-start + post-injection failure ≠ true earliest decisive error; limited human audit; synthetic/natural gap). The paper's methodological advance over re-rollout drift is real and carefully engineered (cache + fidelity abort), and the scale/multimodal coverage are genuine contributions. The residual risk is definitional fidelity of the labels themselves, not circularity or fabricated numbers. That keeps the verdict CONDITIONAL rather than REJECT: the benchmark remains useful for relative ranking and protocol comparison, but absolute claims that models \"cannot attribute\" the decisive error inherit the same uncertainty until counterfactual earliest-step checks or larger natural-failure transfer studies are done. No stronger internal inconsistency appears; the concern is the one the reader already flagged.","tokens_in":35368,"tokens_out":561,"duration_ms":5891,"concrete_test":"On a stratified sample of ~200 retained traces, re-run the original agent from the successful seed while forcing the injected step to the original correct action, then also force each earlier step i < t* to a corrected alternative (or ablate it) and measure whether any earlier correction alone restores success. If >10–15% of traces have an earlier correctable step, or if post-injection rollouts frequently fail even after restoring ~at, the golden-label claim weakens and step metrics need re-labeling.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim rests on golden labels: after replaying a successful prefix and injecting one taxonomy-conditioned action, any resulting task failure is attributed exactly to that agent/step/mode because \"reverting ~at recovers the successful seed\" (§3.3, Fig. 2). That is necessary but not sufficient for the Zhang et al. decisive-error definition (earliest index such that correcting it turns failure into success). Two gaps remain: (1) an earlier latent flaw in the seed may already have made success fragile, so the injection is only the last straw rather than the earliest decisive step; (2) post-injection continuation can introduce new independent failures (especially in stateful browser/code environments where fidelity checks only abort on observed divergence, Appendix F), so the true earliest correctable step may lie after the injection. Human review on 100/12,326 traces (§3.5, Table 3) checks recognizability of the injected label, not counterfactual earliest-correctability. Thus reported step/joint accuracies may systematically overstate how well models recover the true decisive error.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper introduces WHO&WHENPRO, a large-scale benchmark for automated failure attribution in LLM agentic systems. Using a warm-start pipeline that replays a successful trajectory prefix, injects a single taxonomy-conditioned erroneous action, and continues execution, the authors construct 12,326 failed traces with labels for decisive agent, step, and failure mode across 26 source benchmarks, 9 task categories, and text/image/video modalities. They evaluate closed- and open-weight frontier models under all-at-once, step-by-step, and binary-search protocols, reporting that attribution remains difficult (e.g., best text step accuracy ~73.9% but error-mode F1 only ~22.2%), with systematic effects of modality, protocol, trace length, and ground-truth availability, plus cost–performance trade-offs favoring open-weight models under all-at-once evaluation.","tokens_in":35724,"tokens_out":1457,"duration_ms":24269,"significance":"If the label construction is accepted as high-fidelity, this is a substantial contribution: prior failure-attribution benchmarks are small and almost entirely text-only, while WHO&WHENPRO expands scale, modality coverage, and topology coverage in a way that matches modern agent deployments. The warm-start design is a clear methodological improvement over re-rollout injection that can drift before the injection point. The empirical findings—modality-dependent bottlenecks, symptom-vs-root-cause confusion, full-trajectory superiority, and open-weight cost efficiency—are actionable for both attribution systems and self-evolving agents. Strengths include a carefully engineered pipeline (tool caching, fidelity checks), stratified human review with substantial agreement, multi-protocol evaluation, and public release of traces/labels/code.","major_comments":[{"comment":"§3.3 and the decisive-error definition (opening of §3, following Zhang et al.): the construction shows that replacing the injected action ~a_t with the original a_t and restoring the successful continuation recovers success. That establishes the injection as a controlled cause of failure, but it is not identical to the formal earliest-decisive-step claim on the observed failed trajectory (a1..a_{t-1}, ~a_t, a'_{t+1}, …). Post-injection rollouts can introduce additional independent errors, especially in stateful browser/code settings (Appendix F), so correcting only ~a_t while keeping later observed actions need not yield success, and the earliest correctable index could differ. The paper should either (i) reframe labels as “first controlled deviation / injection locus under warm-start construction” rather than unqualified golden earliest-decisive labels, or (ii) provide counterfactual ch","section":"§3.3, Fig. 2, decisive-error definition"},{"comment":"§3.5 / Table 3: human validation covers 100 stratified traces (~0.8% of 12,326). Annotators largely accept the generated step/agent/mode labels (94%/90%/90%, Fleiss κ=0.73), which supports recognizability, but the protocol does not test counterfactual earliest-correctability or measure how often an earlier latent fragility or a later independent failure would change the label. Given that “golden labels” are the central selling point relative to prior automatic pipelines (Table 1), the manuscript should expand validation—at least on a larger stratified sample and with an explicit counterfactual or “would fixing this step alone recover success?” criterion—or clearly bound the claim to human-recognized injection loci rather than full decisive-error semantics.","section":"§3.5, Table 3"},{"comment":"§3.2–3.3 and experimental interpretation: failure modes are profiled from natural failures then re-injected into successful seeds. This is a reasonable scalability design, but the main results (Table 4, Fig. 4) are then used to conclude how models attribute failures in agentic systems generally. The paper should quantify or bound distribution shift between natural and injected failures (e.g., side-by-side human comparison of natural vs injected traces for the same modes, or attribution performance on a held-out natural-failure subset with expert labels). Without that, claims about “systematic patterns” risk being specific to synthetic injection signatures rather than organic agent failures.","section":"§3.2–3.3, Table 4, Fig. 4"}],"minor_comments":[{"comment":"Table 4 and §4.2: Joint is described as requiring Agent, Step, and Error all correct, but Agent is only defined on multi-agent traces. Clarify how Joint is computed on single-agent traces (76% of the data per Fig. 3)—e.g., whether Agent is treated as vacuously correct.","section":"§4.1 Metrics, Table 4"},{"comment":"Fig. 4b / Fig. 10: confusion matrices average over models; also report per-model or at least variance, since the text claims a general symptom-vs-root-cause pattern.","section":"Fig. 4b, Fig. 10"},{"comment":"§4.3: the ablation subset of 1,444 traces is stratified, but the main paper should state the sampling procedure and confirm that Table 6 trends match full-set all-at-once results for the overlapping setting.","section":"§4.3, Table 6"},{"comment":"Appendix F fidelity checks: report quantitative replay-abort rates by environment class (static tools vs browser vs code) so readers can assess residual selection bias in retained traces.","section":"Appendix F"},{"comment":"Minor polish: consistent naming (WHO&WHENPRO / Who&When Pro), and ensure all 18 mode codes in Table 5 align with the evaluation prompt taxonomy block without renumbering ambiguity.","section":"Table 5, Appendix J"}],"recommendation":"major_revision","confidential_remarks":"This is a strong systems/benchmark paper for a datasets-and-benchmarks or agent-evaluation venue. The main risk is overclaiming “golden” earliest-decisive labels; if the authors tighten the label semantics and add modest counterfactual/human validation, I would expect a clear accept. I do not see circularity in scoring judges against injection labels. Scope fit is good for AI/agent venues; less so for theory-heavy ML journals."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The thing worth knowing: this is a systems/benchmark paper that actually fixes a concrete label-fidelity problem in prior injection work. Instead of re-rolling from scratch (where the prefix can drift before the bug is planted), they replay a successful prefix, inject one taxonomy-conditioned action, warm-start the environment, and keep only trajectories that then fail. That gives them 12,326 labeled failures across 26 benchmarks, text/image/video, single- and multi-agent. That combination of method + scale + modality is new relative to Who&When, MAST, AEGIS, AgenTracer, etc.\n\nWhat they do well is the evaluation, not just the dump of data. All-at-once beats step-by-step and binary search; step accuracy collapses on long traces and video; mode F1 is weak (best text error F1 ~22%) while step can hit ~70%+; open-weight models sit on the cost Pareto; giving the gold answer helps perception modes and hurts reasoning modes. Those patterns are useful if you are building attribution or self-evolution loops. Human check on 100 stratified traces (94% step, 90% agent/mode, κ=0.73) is serious enough for a benchmark paper, and the taxonomy is grounded in natural failures before injection.\n\nSoft spots, in proportion. The stress-test point is fair but not fatal: “revert the injection and the seed succeeds” makes the injected step decisive under their operational definition; it does not fully prove it is the earliest latent correctable flaw, and post-injection continuations (especially stateful browser/code) can add new mess. They mitigate with caches, fidelity aborts, and filters, but the human audit is still ~0.8% of the set and checks recognizability more than counterfactual earliest-correctability. Synthetic-vs-natural gap remains; they own that in limitations. None of that makes the tables circular—judges are scored against construction labels, not self-labels.\n\nWho it’s for: people working on agent debugging, multi-agent failure analysis, or self-evolving agents who need a hard multimodal attribution yardstick and protocol/cost guidance. Not a theory paper and not a deployable fixer. I would bring it to reading group, cite it if I ship attribution or agent-eval work this year, and send it to peer review. The central construction is sound enough to referee; ask for more natural-failure transfer and a clearer statement of what “decisive” means under latent earlier fragility.","headline":"Warm-start injection at 12k multimodal scale is a real methodological step up; models still fail hard on failure-mode diagnosis, and the “golden earliest step” claim is a bit cleaner in the paper than under Zhang’s strict counterfactual reading.","tokens_in":36318,"tokens_out":635,"would_cite":true,"duration_ms":10291,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"A warm-start injection pipeline builds 12,326 golden-labeled agent failure traces across text, image, and video, and under that test frontier LLMs still cannot reliably name who failed, when, and why.","keywords":["failure attribution","LLM agents","multi-agent systems","error injection","multimodal agents","decisive error","self-evolving agents","agent debugging"],"falsifier":"A larger human audit or counterfactual “fix the injected action and re-run” study showing that a substantial fraction of retained traces already had an earlier decisive error, or that judges trained on Who&When Pro systematically fail on naturally occurring (non-injected) agent failures.","tokens_in":36287,"feed_emoji":"🔍","tokens_out":960,"duration_ms":11685,"temperature":0.7,"pith_summary":"As AI agents get stronger, their mistakes get subtler, so the field needs machines that can say which agent went wrong, at which step, and in what way. This paper’s answer is Who&When Pro: take a trajectory that already succeeded, restore the exact state just before a chosen step, force a single taxonomy-conditioned bad action, then let the real agent run on—and keep only the runs that now fail. Because the only controlled change was that one action, the decisive agent, step, and error mode become golden labels. The resulting 12,326 traces span 26 source benchmarks, nine task families, and three modalities. When frontier closed- and open-weight models try to attribute those failures, step localization peaks around 74% on text but error-mode diagnosis stays near 22% F1, joint accuracy is lower still, and behavior shifts systematically with modality, protocol, and model family. The authors treat the benchmark and those patterns as a foundation for building better automatic debuggers and self-improving agents.","feed_headline":"LLMs still can't say who failed or why in agents","feed_subtitle":"12,326 golden traces show step accuracy near 74% but mode F1 stuck around 22%","key_machinery":"Warm-start decisive-error injection: from a successful seed trajectory, restore agent and environment state through step t−1 (via cache or fidelity-checked replay), substitute one taxonomy-conditioned erroneous action, resume the original system, and retain only post-injection failures—so the injected step is, by construction, the earliest decisive error.","core_discovery":"Strict warm-start error injection after exact replay of a successful prefix yields large-scale golden labels for decisive agent, step, and failure mode; evaluated on the resulting 12,326 multimodal traces, current LLMs remain far from reliable automated failure attribution, with especially weak mode classification and clear difficulty on long and video traces.","pith_inferences":["If warm-start fidelity ever leaks pre-injection drift at scale, the “golden” labels become approximate and the entire ranking of models on the benchmark softens.","The surface-symptom bias (planning/verification/coordination misread as reasoning) suggests attribution judges may need explicit causal or counterfactual prompting, not only larger models.","A natural next stress test is transfer: train or prompt on injected traces and measure recovery of natural failures from the same agent frameworks.","Taxonomy extensibility matters: new embodied or long-horizon collaborative settings will likely require new modes before the same pipeline stays complete."],"forward_implications":["Full-trajectory (all-at-once) attribution should be preferred over step-by-step or binary-search protocols for both accuracy and cost.","Open-weight models can sit on the cost–accuracy Pareto frontier for practical attribution backbones.","Step localization is hardest on video and long traces; mode classification is relatively easier once visual or behavioral cues appear.","Providing the task’s ground-truth answer helps diagnose perception errors but can degrade diagnosis of reasoning errors by inviting outcome shortcuts.","Future self-evolving agents can use externally verified step-level labels of this form rather than coarse terminal or self-graded feedback."],"fun_headline_variants":["LLMs lag at naming who and why agents fail","12k golden traces: agent failure modes still hard for LLMs","Who&When Pro finds weak LLM mode attribution on agents","LLMs reach ~74% step hits but only ~22% mode F1 on agents","Long and video agent traces trip up LLM failure attribution"],"cache_read_input_tokens":32896,"weakest_assumption_plain":"That restoring state and forcing a single later failure is enough to guarantee the injected step was truly the earliest decisive error, and that these synthetic modes fairly represent the failures an attribution system must catch in the wild.","fun_headline_variants_meta":{"raw":{"variants":["LLMs lag at naming who and why agents fail","12k golden traces: agent failure modes still hard for LLMs","Who&When Pro finds weak LLM mode attribution on agents","LLMs reach ~74% step hits but only ~22% mode F1 on agents","Long and video agent traces trip up LLM failure attribution"]},"model":"grok-4.5","effort":"low","cost_usd":0.005338,"raw_usage":{"total_tokens":1355,"prompt_tokens":650,"num_sources_used":0,"completion_tokens":71,"cost_in_usd_ticks":53380000,"prompt_tokens_details":{"text_tokens":650,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":634,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":650,"tokens_out":71,"duration_ms":5126,"temperature":1.0,"reasoning_tokens":634,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-14T01:09:31.281765+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"A larger human audit or counterfactual “fix the injected action and re-run” study showing that a substantial fraction of retained traces already had an earlier decisive error, or that judges trained on Who&When Pro systematically fail on naturally occurring (non-injected) agent failures.","supporting_citations":[],"review_version":1}