Artificial Intelligence · arXiv:2610.17633 v1 · submitted 13 Oct 2026 · 13 references
Executable Prefix Values for Locating Agent Failures
The paper says
“Work on debugging LLM agents treats the "first mistake" marked by a human or LLM judge as ground truth for where a trajectory failed.”
more of the abstract
We ask whether such labels point at causes or symptoms, and make the distinction executable: we restart the agent from reconstructed intermediate states of a failed trajectory and estimate its prefix value V(h_t), the probability of success from each step, with a certified point of no return (PNR) for a failure. On 150 SWE-bench Verified tasks with two frozen open-weight agents, we build such curves for 47 and 60 failed trajectories and probe every step labeled by four frontier judges under the Traverse rubric. We find that the labels do not mark what a controller needs. Judges of different lineages agree on the exact step in only 21/57 cases, and the newest judges flag mistakes in 8/10 and 10/10 successful runs. For 66% of the labels, the agent's success probability does not drop at the labeled step; only 7/188 answers name the decisive step. Restarting at the measured restart point raises expected success from 0.44 to 0.57 offline, and real restarts confirm the advantage, whereas restarting at a judge's label does not (-0.08 to -0.03). We release our code, curves, and labels.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 14 Oct 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. A solid, well-executed measurement paper that shows judge-assigned first-mistake labels don't align with where the agent's success probability actually drops; the main claims hold despite the acknowledged replayability limitation. This line is wrong
3 major objections · 6 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 13 references say.
The paper says, at §4, §6.1, Appendix L
The review did not quote this passage.
Pith says: Objection 1 of 3 · major
The replayability filter (exact replay required under 26-rule normalization) drops 27% of selected trajectories (73% replayable for the main policy; 72% for the second policy). The paper reports mismatch causes and strata rates but does not compare any property — judge label positions, trajectory length, end type, task — between the replayable set and the dropped non-replayable set.
more of objection 1
Since the V-curves and all comparisons to judge labels are built exclusively on the replayable subset, the generalizability of the central claim (that judge labels do not mark what a controller needs) depends on the untested assumption that the dropped trajectories do not have systematically different failure patterns. The paper acknowledges replay failures and bounds the rate, but does not quantify the risk. To close this gap, the authors should provide a comparison of at least the available metadata (e.g., judge label positions vs. t*, trajectory length, end type, task clustering) between the replayable and non-replayable trajectories, or explicitly bound how different the unobserved failure patterns would have to be to change the main conclusions.
The paper says, at §6.5, offline restart evaluation
The review did not quote this passage.
A classifier read this sentence of the paper as acknowledging the concern (probability 0.88):
“The restart evaluation is offline and assumes a value-based policy never restarts a run that would have succeeded.”
Pith says: Objection 2 of 3 · major
The offline restart evaluation assumes that value-based policies never restart a run that would have succeeded (i.e., the V-curve estimate identifies all and only the failing prefixes). The paper acknowledges this assumption and partially addresses it with an online check on 30 trajectories (240 fresh runs).
more of objection 2
However, the online check only validates the sign of the gain, not the magnitude, and the offline evaluation is used to compute the reported gains (+0.13 at tr, -0.08 to -0.03 for judges). The Limitation paragraph states this assumption, but the abstract and conclusions present the offline gains without repeating the caveat. I recommend adding a sentence in the abstract or conclusions that the offline gains are conditional on this assumption, and that the online check confirms the direction but the exact magnitudes may differ in practice.
The paper says, at §6.9, second policy budget control
The review did not quote this passage.
A classifier read this sentence of the paper as acknowledging the concern (probability 0.67):
“This policy fails the cap-hit check (§6.10): its curves are reported as a replication of the label results, and its restart evaluation is to be read with that caveat.”
Pith says: Objection 3 of 3 · major
For the second policy (Qwen3.5-9B), 58% of continuations exhaust the remaining budget (69% on capped base runs), and budget control moves t* later in all 10 trajectories tested. The paper acknowledges this (§6.9, Limitation) and states that for this policy 'lost under the remaining budget' and 'lost' nearly coincide.
more of objection 3
However, the label comparison results (76% post-cause or zero-drop) are still presented as a replication of the main pattern, and the restart evaluation is reported without adjusting for this confound. Since the budget effect is so large that it fundamentally changes what the V-curve measures for this policy, the label comparison should be clearly marked as conditional on the remaining budget, and the restart gains should not be presented as directly comparable to the main policy. The paper already separates these tables in Appendix X, but the main text (§6.9) could more crisply state that the second policy's restart evaluation is not a clean replication of the main policy's result.
6 minor objections
The paper says, at Abstract and §1
Pith says: minor objection 1 of 6
The abstract and introduction assert 'the labels do not mark what a controller needs' and 'judges of different lineages agree on the exact step in only 21/57 cases'. The 21/57 agreement number for GPT-5.5 vs. OpenAI o3 is correct but the base should be 57 trajectories (failed and decoys together, Table 6).
more of objection 1
For the main claim, the relevant count is the 47 failed trajectories; the decoys are important for over-flagging but including them in the agreement denominator mixes two different claims. I suggest reporting the agreement separately for failed trajectories (Table 16) and for decoys in the main text.
The paper says, at §6.3, Table 2
Pith says: minor objection 2 of 6
Table 2 reports 'symptom' share as 81% (Pooled) with a Wilson interval. The table caption and text should clarify that the 'symptom' category combines 'post' and 'post-PNR' and 'zero-drop' labels. Currently the last column is labelled 'symptom' but the footnotes say 'post-cause or zero-drop'. This is a minor clarity issue; a consistent label (e.g., 'post-cause or zero-drop') would be clearer.
show 4 more
The paper says, at §6.6 and Appendix R, Table 29
Pith says: minor objection 3 of 6
The cheaper-schedule analysis uses 20 replications per trajectory on the measured curves and reports that even the full schedule is exact only in 28% of replications. The text should explicitly state that these are parametric bootstrap results conditional on the observed curves, not a fresh data collection, to avoid implying that the schedule's actual real-world accuracy is that low. This is already clear from context but could be stated upfront in the paragraph.
The paper says, at §6.7, 'Interpretation and information use' paragraph
Pith says: minor objection 4 of 6
This paragraph is speculative and does not come with any measured evidence. It reads like a conjecture rather than a result. I recommend either removing it or moving it to the Discussion section with an explicit caveat that it is an interpretation, not a finding.
The paper says, at Appendix F, four trajectories
Pith says: minor objection 5 of 6
The close reading of matplotlib-25960, requests-6028, sympy-13877, and django-16938 is very helpful. I suggest adding one sentence per trajectory that states whether the judge labels fell before or after the decisive interval, to link the narrative directly to the quantitative results in Table 2.
The paper says, at General
Pith says: minor objection 6 of 6
The paper uses the notation 't_pnr' for point of no return in Tables 9–12 but 'tpnr' in the text and captions. Consistency would help readers. Also, the term 'decisive interval' is used interchangeably with 'the interval (a, b] containing t*' in several places; a single definition in §3 would reduce ambiguity.
What would settle it
Compare the distribution of judge label positions (fraction of trajectory length) between the replayable trajectories (n=47 for main policy) and the selected but non-replayable trajectories (n=17 for main policy, which had judge labels but could not be replayed). Use a two-sample Kolmogorov-Smirnov test and a logistic regression controlling for stratum (L0/L1/L2) and task. If the distributions are statistically indistinguishable (p>.05 and effect size <0.2 standard deviations), the replayability concern is mitigated. If they differ, the paper's claim must be scoped to the replayable subset, whose composition may be biased toward L2 (capped) runs (90% replay against 60-71% for L0/L1).
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 14 Oct 2026
- Reader holds up, with conditions
- Skeptic agrees
- Editor wrote the verdict above
- Referee major revision: 3 major objections, 6 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
This paper does something refreshingly concrete: it defines the prefix value V(ht) for an agent trajectory, measures it via exact replay and massive continuation sampling, and then simply places judge labels on the resulting V-curves to see where they fall. The core finding — that 81% of judge labels fall after the decisive value drop or at a step where V doesn't drop at all — is well supported by direct probes, Wilson intervals, task-cluster bootstrap, and synthetic validation. The paper is unusually transparent: frozen parameters, preregistered-style selection rules, controls for replay, budget, flakiness, and test-retest noise. It also replicates the finding on a second, weaker agent. This is real evidence, not just another benchmark built on judge agreement.
The main soft spot is the replayability requirement. The whole measurement depends on reconstructing intermediate states exactly, and only about 73% of selected trajectories pass. The paper reports the mismatch causes and acknowledges that non-replayable trajectories may differ systematically, but it does not compare any property — label positions, failure shapes, trajectory length — between the replayable set and the dropped set. That gap is real, but it's stated as a limitation, not hidden. It doesn't undermine the central measurement claim that labels don't mark value drops within the replayable set; it just leaves open whether the same pattern holds for the 27% of trajectories that can't be replayed. A follow-up analysis comparing at least judge-label distributions between the two sets would strengthen the paper.
The offline restart evaluation is a secondary claim; the judge-label restart comparison is never tested online, and the online check for the value-based restart shows slight optimism in the predicted R. The scope is also narrow: two agents from one family, one scaffold, one benchmark, judges mostly from one vendor. These are acknowledged limitations, not fatal flaws.
Who is this for? Anyone building agent failure localizers, restart controllers, or judge-label benchmarks. The paper makes a strong case that value captured should replace label agreement as the evaluation metric. I'd bring it to a reading group and would cite the empirical findings. It deserves a serious referee: accept with minor revisions, asking for the replayability characterization to be filled in and for the online judge-label comparison to be attempted (or acknowledged as future work).
The strongest objection
The central measurement (V-curves, label drop probes, restart gains) is built exclusively on trajectories that pass exact replay under the 26-rule normalization. The paper reports that 73% of selected main-policy trajectories (72% for the second policy) pass replay and that non-replayable trajectories print genuinely non-deterministic values (Appendix L). However, it does not compare any property — judge label positions, trajectory length, end type, failure shape — between the replayable set and the dropped non-replayable set. If non-replayable failures are systematically different (e.g., concentrated in tasks or commands that produce non-deterministic output, which may correlate with earlier or later decisions), then the paper's conclusion that 'labels do not mark what a controller needs' may hold only for the subset of failures that happen to be repeatable under Docker. The paper acknowledges this gap ('the paper measures the difference but cannot rule it out') but does not close it. Because the replay step is required for every continuation, the entire empirical core of the paper depends on the assumption that the replayable set is representative; without evidence, this is the weakest link.
The reader identified replayability as the weakest assumption, and I concur. The paper's entire measurement pipeline depends on replay, and the 27% of selected trajectories that fail replay are excluded without analysis. This leaves open the possibility that the conclusions about judges and restarts are artefacts of the replayable subset. The paper acknowledges the issue but does not quantify it; a simple comparison of label positions between replayable and non-replayable trajectories (using already-collected judge labels for both sets) would directly address this gap. If no difference is found, the core claim is robust to this assumption. If a difference is found, the paper's conclusions would need to be scoped to the replayable subset, which may be unrepresentative due to differential exclusion by stratum (L2 replays best). The concrete test is cheap, feasible with existing data, and would either confirm the assumption or force a qualification. The verdict remains UNCHANGED because the paper already states this limitation; the test would determine whether the condition can be relaxed.
The referee's report
If the central result holds, it challenges the standard practice of treating first-mistake labels as ground truth in LLM agent debugging and instead grounds localization in the agent's own behavioral statistics. The paper provides a concrete methodology (prefix value estimation with certified point of no return) and a clear failure typology (cliff vs. erosion) that could guide practical controller design and localizer benchmarking. The replication on a second agent and the inclusion of an offline and online restart evaluation strengthen the practical relevance. The paper also ships code, curves, and labels, making the empirical base fully transparent.
Pith's reading of the paper (a machine's summary, not the authors')
The paper challenges the common practice of treating the 'first mistake' identified by a human or LLM judge as the ground truth for where an agent's trajectory failed. Using a method that restarts the agent from reconstructed intermediate states of a failed run, the authors measure the prefix value V(h_t) — the agent's own probability of success from each step — and locate the decisive drop and a certified point of no return. On 150 SWE-bench tasks with two open-weight agents, they find that across four frontier judges, 81% of labels fall after the decisive drop or at a step where V does not drop at all; only 7/188 answers name the decisive step. Restarting at the measured restart point raises expected success from 0.44 to 0.57 offline (confirmed by 240 real restarts), whereas restarting at a judge's label either loses or leaves success unchanged (-0.08 to -0.03). The result suggests that current failure-localization benchmarks, which score by agreement with labels, reward the wrong signal.
What carries the argument. The key machinery is the prefix value V(h_t), the probability that the agent succeeds if it continues from a given history h_t with the remaining budget. It is estimated by replaying the trajectory to reconstruct the exact environment state at that step, then running many fresh continuations of the agent from that state, and measuring the success fraction. An adaptive schedule (screening, bisection, confirmation, and point-of-no-return certification) buids the V-curve efficiently. The curve is made monotone by isotonic fitting, and the decisive step t* is the argmax of the drop, while the point of no return is the earliest step where the one-sided Wilson upper bound of V is below 0.15.
Core claim. The central discovery is that first-mistake labels from judges do not correspond to the step where the agent's own success probability V(h_t) drops. Instead, the labels scatter around the decisive drop, and for 66% of labels V does not move at the labeled step at all. The paper defines the prefix value V(h_t) as the probability of success when the agent is resumed from that point, and shows that this quantity can be estimated by replaying the trajectory and taking fresh continuations. The V-curve reveals the decisive step (where V drops most) and a certified point of no return (earliest step at which V ≤ 0.15 with 95% confidence). A controller that restarts at the measured restart point gains +0.13 in expected success over not restarting, while a controller that researts at a judge's label loses (up to -0.08).
The premise it rests on. The method requires that every intermediate environment state of a trajectory can be reconstructed exactly so that the agent can be resumed from that state; if non-replayable trajectories differ systematically from replayable ones, the V-curves only describe the replayable subset of failures.
If the paper is right (Pith's inferences, not the authors')
Failure-localization benchmarks should score localizers by value captured (share of V drop within k steps) rather than by agreement with judge labels, since agreement rewards labeling like other judges, not finding where the task was lost.
Inference-time controllers can use the measured restart point or point of no return to decide when to abandon a failing run and start fresh, improving overall success probability by an average of 0.13 to 0.17 over affected tasks.
The method provides a reference signal against which future localizers (including improved judges) can be calibrated: a label is useful only if the agent's V drops at or before the labeled step.
The finding that 30/47 failures are erosions (value lost over many steps) suggests that a single 'decisive step' may not exist for many runs, and that rubrics forcing a single point onto a slope are ill-posed.
If the pattern generalizes to other agents and benchmarks (the paper tests only two agents from one family on SWE-bench), then much of the existing work on first-mistake annotation (Traverse, Who&When, LongRCA) may be measuring inter-judge agreement rather than pinpointing failure causes, and the resulting verifiers may be optimized for the wrong target.
The high rate of zero-drop at labeled steps (67%) suggests that judges are often marking steps where an error is committed but the run was already doomed — i.e., the label is a symptom of an earlier, invisible cause. This aligns with the observation that the decisive step is usually an innocuous read or probe before the first edit.
The cheap-schedule analysis (§6.6) implies that one can obtain a useful restart point for about 200 continuations without label probes, making the method potentially practical for production systems that can afford a small number of real restarts per run.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Work on debugging LLM agents treats the "first mistake" marked by a human or LLM judge as ground truth for where a trajectory failed. We ask whether such labels point at causes or symptoms, and make the distinction executable: we restart the agent from reconstructed intermediate states of a failed trajectory and estimate its prefix value V(h_t), the probability of success from each step, with a certified point of no return (PNR) for a failure. On 150 SWE-bench Verified tasks with two frozen open-weight agents, we build such curves for 47 and 60 failed trajectories and probe every step labeled by four frontier judges under the Traverse rubric. We find that the labels do not mark what a controller needs. Judges of different lineages agree on the exact step in only 21/57 cases, and the newest judges flag mistakes in 8/10 and 10/10 successful runs. For 66% of the labels, the agent's success probability does not drop at the labeled step; only 7/188 answers name the decisive step. Restarting at the measured restart point raises expected success from 0.44 to 0.57 offline, and real restarts confirm the advantage, whereas restarting at a judge's label does not (-0.08 to -0.03). We release our code, curves, and labels.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-10-14
The authors' main claim, as the classifier picked it out 0.99
“We find that the labels do not mark what a controller needs.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.07 · states limitation 0.37 · human participants 0.12 · controlled comparison 0.90 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.86 · mentions code or data availability 0.99
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Work on debugging LLM agents treats the "first mistake" marked by a human or LLM judge as ground truth for where a trajectory failed. | background 1.00 | 0.04 | 0.03 | asserted 1.00 |
| We ask whether such labels point at causes or symptoms, and make the distinction executable: we restart the agent from reconstructed intermediate states of a failed trajectory and estimate its prefix value V(h_t), the probability of success from each step, with a certified point of no return (PNR) for a failure. | contribution 0.61 | 0.94 | 0.09 | asserted 0.80 |
| On 150 SWE-bench Verified tasks with two frozen open-weight agents, we build such curves for 47 and 60 failed trajectories and probe every step labeled by four frontier judges under the Traverse rubric. | method 0.95 | 0.96 | 0.03 | asserted 0.98 |
| We find that the labels do not mark what a controller needs. | interpretation 0.55 | 0.85 | 0.70 | asserted 1.00 |
| Judges of different lineages agree on the exact step in only 21/57 cases, and the newest judges flag mistakes in 8/10 and 10/10 successful runs. | result 1.00 | 0.89 | 0.07 | asserted 1.00 |
| For 66% of the labels, the agent's success probability does not drop at the labeled step; only 7/188 answers name the decisive step. | result 1.00 | 0.95 | 0.35 | asserted 1.00 |
| Restarting at the measured restart point raises expected success from 0.44 to 0.57 offline, and real restarts confirm the advantage, whereas restarting at a judge's label does not (-0.08 to -0.03). | result 1.00 | 0.96 | 0.27 | asserted 1.00 |
| We release our code, curves, and labels. | contribution 0.99 | 0.81 | 0.02 | asserted 0.78 |
What kind of paper the classifier reports this is: empirical study 0.63 · evidence from observation or experiment 0.98
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-14: 1 request, 6,385 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (12)
Figures are the authors', reproduced from the paper.
References (13)
- [1] Locating Hidden Failures Makes Long-Horizon Agents More Reliable , author =. arXiv preprint arXiv:2609.17930 , year = on PithPith's reviewarXiv
- [2] Wang, Chenyu and Lyu, Yunbo and He, Junda and Yang, Zhou and Zhong, Chenxing and Harel, Yaniv and Lo, David , journal =. Fail-Fast, Restart-Smart: Early Failure Prediction and Restart for. 2026 , doi = on Pith
- [3] Zhu, Tianshu and Zhang, Wenyu and Zuo, Xiaoying and Tian, Lun and Zhao, Haotian and Zeng, Yucheng and Gu, Jingnan and Dong, Daxiang and Wu, Jianmin and Yin, Dawei and Shen, Dou , journal =. Rollout Pass-Rate Control: Steering Binary-Reward. 2026 , doi = on Pith
- [4] Zhu, Kunlun and Liu, Zijia and Li, Bingxuan and Tian, Muxin and Yang, Yingxuan and Zhang, Jiaxun and Han, Pengrui and Xie, Qipeng and Cui, Fuyang and Zhang, Weijia and Ma, Xiaoteng and Yu, Xiaodong and Ramesh, Gowtham and Wu, Jialian and Liu, Zicheng and Lu, Pan and Zou, James... on Pith
- [5] Zhang, Shaokun and Yin, Ming and Zhang, Jieyu and Liu, Jiale and Han, Zhiguang and Zhang, Jingyang and Li, Beibin and Wang, Chi and Wang, Huazheng and Chen, Yiran and Wu, Qingyun , journal =. Which Agent Causes Task Failures and When? On Automated Failure Attribution of. 2025 , doi = on Pith
- [6] Ma, Ming and Zhang, Jue and Yang, Fangkai and Kang, Yu and Lin, Qingwei and Rajmohan, Saravan and Zhang, Dongmei , journal =. 2025 , doi = on Pith
- [7] Zhang, Yunfei and Feng, Boyu and Pei, Changhua and Wang, Zexin and Peng, Zhihuang and Liu, Xinlong and Jiang, Hengyue and Ma, Difeng and Zhang, Jiayi and Yao, Yongzhou and Zhao, Yanan and Sun, Fei and Huo, Yintong and Liu, Zhaoyang and Li, Jingjing and Xie, Gaogang and Pei, Da... on Pith
- [8] Jimenez, Carlos E. and Yang, John and Wettig, Alexander and Yao, Shunyu and Pei, Kexin and Press, Ofir and Narasimhan, Karthik , booktitle =. 2024 , doi = on Pith
- [9] Yang, John and Jimenez, Carlos E. and Wettig, Alexander and Lieret, Kilian and Yao, Shunyu and Narasimhan, Karthik and Press, Ofir , journal =. 2024 , doi = on Pith
- [10] mini-swe-agent: The 100 line on Pith
- [11] Kwon, Woosuk and Li, Zhuohan and Zhuang, Siyuan and Sheng, Ying and Zheng, Lianmin and Yu, Cody Hao and Gonzalez, Joseph E. and Zhang, Hao and Stoica, Ion , booktitle =. Efficient Memory Management for Large Language Model Serving with. 2023 , doi = on Pith
- [12] Probable inference, the law of succession, and statistical inference , author =. Journal of the American Statistical Association , volume =. 1927 , doi = on Pith
- [13] Thinking Before Thinking: Scaling Agentic Inference Through Meta-Reasoning , author =. arXiv preprint arXiv:2609.38147 , year = on PithPith's reviewarXiv
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment