{"id":"94ba121a-2183-4a2c-a969-b25b7b1a5192","arxiv_id":"2609.17930","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":8.0,"correctness_risk":"low","formal_verification":"none","parameter_count":5,"one_line_summary":"Long-horizon AI agents fail from a single early mistake that cascades, and a small trained verifier can locate that mistake better than frontier models.","lead":"This paper analyzes how long-horizon AI agents fail by labeling 2,518 real agent trajectories step by step. It finds that failure is usually caused by a single early mistake that goes undetected and uncorrected, and builds a small verifier that can locate such mistakes better than large frontier models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"The human-annotated first-mistake labels may contain systematic biases that Scout's fine-tuning encodes more faithfully than zero-shot frontier models, making the reported advantage partly an artifact of annotation style rather than objective superiority.","rationale":"The reader identified the human-annotation assumption as the weakest. I agree that it is load-bearing, but I push further: the paper does not test for systematic bias in the annotations, which is a concrete threat to validity, and the small SWE-bench improvement may not be statistically significant. However, the paper acknowledges the 23% diffuse-failure limitation, uses multiple judges and ranges for sensitive quantities, and releases the data and model. The central characterization of failure and the demonstration that a small model can be trained to locate mistakes are still valuable even if the exact ranking vs. frontier models shifts under re-annotation. Hence the verdict remains ACCEPT; no change needed.","tokens_in":29992,"tokens_out":5833,"duration_ms":56279,"concrete_test":"Commission an independent annotation team (different institution, fresh guidelines) to re-label the first mistake on a random subset of 200 Traverse trajectories (100 SWE-bench, 100 TerminalBench). Compute inter-annotator agreement with the original labels. Then evaluate Scout (SFT+RL) and the best frontier model (Gemini 3.1 Pro) on the new labels. If Scout's exact-match advantage over Gemini 3.1 Pro on the new labels is significantly smaller (e.g., more than halved) than on the original labels, then the reported advantage is sensitive to annotation bias. Additionally, compute bootstrapped 95% confidence intervals for all test-time selection success rates; if the 78.0% vs 77.4% difference is not significant (p>0.05), the SWE-bench claim should be tempered.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The entire pipeline—failure characterization, Traverse benchmark, Scout training, and evaluation—rests on gold first-mistake labels produced by human annotators. Inter-annotator agreement (κ=0.73 on where) is substantial but not perfect, and 23% of failures lack a localizable first mistake, indicating inherent ambiguity. The paper does not test whether the relative performance of Scout versus frontier models is stable under alternative annotation rubrics or a separate pool of annotators. If human labels systematically reflect a particular judgment style (e.g., flagging action-level errors earlier than causally prior reasoning errors), Scout's supervised training could internalize those biases, while zero-shot frontier models might make different (potentially more correct) judgments. The test-time selection improvement on SWE-Bench Verified is also small (78.0% vs. 77.4% for the next-best verifier, a 0.6 pp gap) and no confidence intervals are reported, so the gain may not be statistically significant. Together, these concerns weaken the central claim that Scout objectively locates failure 'far better' than frontier models and that the benefit for test-time selection is robust.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper studies 2,518 agent trajectories across software engineering, computer use, and science, and classifies 6,967 mistakes into 78 failure types. It finds a recurring failure signature: after a first mistake, agents rarely recover (30.5% recovery), rarely detect the error (38.5% never detect), and often keep acting, so the error cascades while the final state appears correct. The paper releases Traverse, a human-annotated benchmark for locating the first mistake in long-horizon trajectories. Using these annotations, the authors train Scout, a 4B-parameter verifier that locates the first mistake more accurately than six frontier judges (Gemini, Claude, GPT, DeepSeek, Qwen) and, when used for test-time selection, raises task success above agent self-performance (e.g., 90.2% vs. 81.8% on Terminal-Bench 2.0).","tokens_in":30390,"tokens_out":3074,"duration_ms":30929,"significance":"If the claims hold, this work is a substantial contribution to scalable oversight of long-horizon AI agents. The failure taxonomy is the first holistic step-level account across multiple deployment-relevant domains. Traverse fills a critical gap as the first benchmark combining long trajectories, naturally occurring failures, and human-verified first-mistake localization. The demonstration that a small specialized verifier can outperform much larger frontier models at failure localization, and that this translates into test-time gains, opens a practical path toward more reliable autonomous agents. The safety audit (irreversible actions even in solved runs) is a valuable addition beyond prior failure studies.","major_comments":[{"comment":"The claim that Scout 'locates failure far better' than frontier judges is not accompanied by confidence intervals for the key metrics (exact match, MAE, test-time selection gain). On SWE-Bench Verified, the test-time improvement over the next-best verifier is only 0.6 pp (78.0% vs. 77.4%); without bootstrapped CIs or a significance test, this difference may not be statistically significant. The authors should provide uncertainty estimates for all main comparisons to support the strength of the claims.","section":"Abstract, Section 3.3"},{"comment":"The entire pipeline—failure characterization, Traverse benchmark, Scout training—rests on gold first-mistake labels produced by a single annotation process (though two annotators and a third resolver). Inter-annotator agreement of κ=0.73 on 'where' is substantial but not perfect, and 23% of failed runs have no localizable first mistake. The paper does not test whether Scout's relative advantage over frontier models is stable under alternative annotation rubrics or a separate pool of annotators. A sensitivity analysis (e.g., evaluating Scout against the secondary annotator's labels) would increase confidence that the advantage is not an artifact of annotation style.","section":"Section 2.2, Limitations in Section 4"},{"comment":"Table 1 reports localization accuracy for frontier models but does not include Scout's performance on the same held-out test set (258 trajectories). Since the central claim is that Scout is superior, the main comparison table should include Scout's exact match, ±1, ±3, and MAE on SWE-bench and TerminalBench (and BixBench for transfer) to allow direct reading.","section":"Table 1, Section 3.2"}],"minor_comments":[{"comment":"The phrase 'far better' is imprecise; replace with specific effect sizes or 'substantially better' where supported by the data.","section":"Abstract, Section 3.3"},{"comment":"The reward weighting coefficients 0.7 and 0.3 appear without justification. A brief justification or a sensitivity analysis would help the reader understand their impact.","section":"Section 2.4, Eq. (1)"},{"comment":"Panel labels P1, P2, P4 could be more descriptive directly in the caption (e.g., P1: non-recovery, P2: non-detection, P4: persistence). Currently the caption uses 'P1, P2, P4' without defining them in the figure text.","section":"Figure 3"},{"comment":"The 23% of runs classified as 'diffuse failure' (no localizable first mistake) are mentioned but not further characterized. A brief description (e.g., gradual accumulation of small errors) would give a more complete picture of failure modes.","section":"Section 3.1"},{"comment":"The greyed-out categories (10 never observed) are clearly marked, but the table is long. Consider moving the full taxonomy to supplementary and keeping a concise summary in the main text.","section":"Appendix Table 3"}],"recommendation":"minor_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"Hey [Colleague],\n\nThis paper delivers the goods. It is the first holistic, step-level account of how long-horizon agents actually fail across multiple realistic domains (software engineering, computer use, science). The taxonomy (78 failure types) and the failure signature—after the first mistake, agents rarely recover, rarely self-detect, and keep going—are genuinely new and well-supported by the data. The Traverse benchmark with human-annotated first mistakes is a practical tool for the field. And Scout, a 4B verifier trained on these annotations, convincingly outperforms much larger frontier models at locating the first mistake and improves test-time task success, especially on Terminal-Bench.\n\nThe soft spots are real but not disqualifying. The entire pipeline depends on human annotations of the \"first mistake.\" Inter-annotator agreement is good (kappa 0.73 on where) but not perfect, and 23% of failures lack a localizable first mistake. The paper acknowledges this, but the stress-test concern—that Scout might internalize annotation biases that zero-shot frontier models do not share—is valid. The test-time gain on SWE-Bench Verified is only 0.6 percentage points (78.0% vs. 77.4%) with no confidence intervals reported, so that particular advantage may not be statistically significant. On Terminal-Bench, the gain is larger and more believable.\n\nStill, the central claims hold: long-horizon failure is characterizable, frontier models struggle to locate it, and a small model can be trained to do it better. The paper is transparent about its limitations, releases its data and model, and the code appears replicable. This is a solid contribution that moves the conversation from outcome-only evaluation to process-level oversight.\n\nWho this is for: anyone working on agent evaluation, safety, reward modeling, or test-time scaling. It should definitely be sent to peer review. I would cite it in my own work, and I'd bring it to a reading group focused on agent reliability.\n\nRecommendation: accept with minor revisions. Encourage the authors to add confidence intervals for the test-time selection results and to include an analysis of annotation bias (e.g., how performance changes under an alternative annotation protocol or a separate annotator pool). The rest is strong enough as is.","headline":"Thorough step-level failure analysis with a trained small verifier that outperforms frontier models, though the human annotation dependency and modest SWE gain warrant cautious interpretation.","tokens_in":30953,"tokens_out":2230,"duration_ms":23000},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"Long-horizon agent failure follows a predictable signature: one early mistake, unnoticed and unrecovered, yet a 4B verifier trained on human annotations can locate it and improve performance.","keywords":["long-horizon agents","failure analysis","process-level verification","AI agent safety","first mistake localization","reward model","test-time scaling","Scout verifier"],"falsifier":"If a replication study with a larger, more diverse set of long-horizon agent trajectories (e.g., from embodied or multi-agent settings) finds that agents frequently recover from their first mistake (e.g., >50% recovery) or that frontier models achieve >80% exact match on first-mistake localization, the paper's central claims would be weakened. Alternatively, if a second annotation study yields inter-annotator agreement on first-mistake location below $\\kappa=0.5$, the reliability of the ground truth would be undermined.","tokens_in":33251,"feed_emoji":"🔍","tokens_out":2040,"duration_ms":66232,"temperature":0.7,"pith_summary":"Long-horizon AI agents working autonomously across hundreds of steps fail in a recurring pattern: after committing their first mistake, they rarely recover (69.5% of the time), rarely detect the error (38.5%), and keep acting regardless (72.6%). This failure signature means a single early reasoning or planning error compounds into a cascade of downstream action mistakes, while the run ends looking correct. The paper shows that frontier models including GPT-5.5, Claude Opus 4.6, and Gemini 3.1 Pro cannot reliably identify this first mistake, succeeding on fewer than a third of long trajectories. Yet a 4B-parameter verifier, Scout, trained on human-annotated first mistakes, locates failure better than any frontier model and, when used to select among an agent's candidate trajectories, raises task success above the agent's own single-attempt performance (from 81.8% to 90.2% on Terminal-Bench). This establishes that failure localization is a learnable skill that a lightweight verifier can acquire, offering a practical path to more trustworthy long-horizon agents.","feed_headline":"4B verifier beats frontier models at spotting agent mistakes","feed_subtitle":"Scout locates the first error and boosts task success without retraining the agent.","key_machinery":"The central objects are the first mistake (the step where a run first goes wrong, human-annotated with Cohen's $\\kappa=0.73$) and the failure signature (non-recovery, non-detection, persistence) that characterizes long-horizon agent failure. The key mechanism is a trained verifier, Scout, a 4B-parameter model fine-tuned on step-level correctness labels using supervised learning and GRPO reinforcement learning. Its reward design uses the product of recall on correct and incorrect steps ($R_c \\cdot R_i$) to avoid the degenerate strategies of over-flagging or under-flagging that plague frontier judges.","core_discovery":"The paper's central discovery is that long-horizon agent failure is not a random accumulation of errors but follows a systematic and measurable signature: after the first mistake, agents recover in only 30.5% of runs, detect the error in only 38.5%, and persist in acting in 72.6%. Errors cascade, making a step 11\\u201314 times more likely to be wrong after an incorrect step. The first mistake is typically a reasoning or planning fault, while the more numerous action mistakes are downstream symptoms. Frontier models, even the strongest (GPT-5.5, Gemini 3.1 Pro), locate this first mistake on fewer than a third of long trajectories, and all are systematically biased toward over- or under-flagging. However, a 4B-parameter verifier (Scout) trained via supervised fine-tuning and reinforcement learning on human-annotated first mistakes outperforms every frontier model on localization, transfers to an unseen scientific domain, and improves agent task success through best-of-N selection without retraining the agent.","pith_inferences":["If failure localization can be learned by a 4B model, it could be distilled into the agent itself as a self-verification module, enabling agents to catch and correct their own mistakes mid-run without an external verifier.","The finding that 23% of failed runs have no localizable first mistake suggests a distinct failure mode (diffuse, gradual degradation) that may require a different oversight strategy, such as continuous monitoring of trajectory health rather than single-point detection.","The safety audit's identification of destructive actions in solved runs implies that deployment of long-horizon agents in high-stakes settings should mandate process-level auditing and not rely on outcome verification alone."],"forward_implications":["Agent training pipelines can use first-mistake localization as a dense reward signal for process-level reinforcement learning, replacing sparse outcome rewards.","Test-time selection with a small verifier can improve agent performance without retraining the agent, as shown by Scout's best-of-N gains on Terminal-Bench and SWE-Bench.","Safety audits of long-horizon agents should monitor the first mistake as the critical intervention point, since destructive actions often follow early errors.","Benchmarks for agent oversight should measure first-mistake localization accuracy, not just outcome pass rates, because frontier judges are biased at the process level.","The failure signature implies that building agents with built-in self-detection and recovery mechanisms could be more effective than reducing their raw error rate."],"supporting_citations":[{"why":"Provides SWE-bench, the software engineering benchmark from which many agent trajectories were collected.","marker":"(Jimenez et al., 2024)"},{"why":"Provides TerminalBench, the computer-use benchmark from which many agent trajectories were collected.","marker":"(Merrill et al., 2026)"},{"why":"Provides BixBench, the science benchmark used for the held-out transfer domain.","marker":"(Mitchener et al., 2025)"},{"why":"Prior process-verifier benchmark (ProcessBench) that Traverse extends to long-horizon agentic tasks.","marker":"(Zheng et al., 2025)"},{"why":"Provides the Group Relative Policy Optimization (GRPO) algorithm used to train Scout.","marker":"(Guo et al., 2025)"},{"why":"Source of Trail trace datasets used for software engineering and computer-use agent trajectories.","marker":"(Deshpande et al., 2025)"},{"why":"Provides the candidate trajectory sets used in the best-of-N test-time selection evaluation.","marker":"(Kwok et al., 2026)"}],"fun_headline_variants":["4B verifier Scout beats frontier models at spotting agent mistakes","Scout finds first agent error where GPT-5.5 and Gemini 3.1 fail","Small Scout verifier outperforms top AI models on locating failures","4B Scout detects hidden agent failures better than giant frontier models"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The paper assumes that most failed runs have a single decisive first mistake that can be reliably identified by human annotators; it explicitly notes that 23% of failed runs have no localizable first mistake (diffuse failure), so the first-mistake analysis covers only a subset of failures.","fun_headline_variants_meta":{"raw":{"variants":["4B verifier Scout beats frontier models at spotting agent mistakes","Scout finds first agent error where GPT-5.5 and Gemini 3.1 fail","Small Scout verifier outperforms top AI models on locating failures","4B Scout detects hidden agent failures better than giant frontier models"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":0.000317,"raw_usage":{"cost":0.000316568,"is_byok":false,"cost_details":{"upstream_inference_cost":0.000316568,"upstream_inference_prompt_cost":0.000120568,"upstream_inference_completions_cost":0.000196},"total_tokens":1766,"prompt_tokens":1066,"completion_tokens":700,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":256,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":621}},"tokens_in":1066,"tokens_out":700,"cost_source":"openrouter_usage","duration_ms":8056,"image_pages":[],"temperature":1.0,"reasoning_tokens":621,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-09-17T13:44:00.449498+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"If a replication study with a larger, more diverse set of long-horizon agent trajectories (e.g., from embodied or multi-agent settings) finds that agents frequently recover from their first mistake (e.g., >50% recovery) or that frontier models achieve >80% exact match on first-mistake localization, the paper's central claims would be weakened. Alternatively, if a second annotation study yields inter-annotator agreement on first-mistake location below $\\kappa=0.5$, the reliability of the ground truth would be undermined.","supporting_citations":[],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}