Pith. sign in

REVIEW 2 major objections 4 minor 53 references

Fail-Fast, Restart-Smart: Early Failure Prediction and Restart for SWE Agentic Tasks

T0 review · 2 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A lightweight monitor can predict when a software-engineering agent will fail from its observable prefix, and a same-policy restart that reuses the aborted run's repository edits recovers more failures and saves tokens than stopping alone o

desk verdict Solid, well-run empirical paper with a genuinely new restart mechanism; the overlay's replay-determinism premise is unvalidated, which is the main thing to fix before trusting the mechanism. read the letter →

arxiv 2608.03222 v1 pith:TLY5TB4X submitted 2026-08-04 cs.SE cs.AI

Chenyu Wang , Yunbo Lyu , Junda He , Zhou Yang , Chenxing Zhong , Yaniv Harel , David Lo This is my paper
classification cs.SEcs.AI
keywords earlyterminationfailurepredictionsoftwareengineeringagentsrestartpolicyeditoverlaySWE-benchVerifiedtokensavingssame-policyrecovery
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether a doomed software-engineering agent run can be stopped early and restarted cheaply rather than paid for to completion. It proposes FailFast–RestartSmart: a 0.6B monitor that reads only the issue statement and recent steps and alarms under a fixed false-positive budget, plus a restart that launches a fresh same-policy run with the aborted trajectory's repository edits offered as an optional overlay. On SWE-bench Verified, a monitor trained only on one policy's trajectories transfers to four policies, including a closed-API model, and saves 14.6%–20.4% of execution tokens at a 5% false-positive budget. At a 25% budget the overlay restart raises Qwen3.6-27B resolution from 66.6% to 71.8%, while cold restart under the same alarms reaches 66.8%. The authors restrict the claim to SWE-bench Verified under one agent scaffold and flag generalization to other tasks and frameworks as untested.

What carries the argument

The load-bearing machinery is a two-stage controller. FailFast is a frozen 0.6B language model with LoRA and three linear heads over the last hidden state: a value head, a fail-to-pass (F2P) head, and a pass-to-pass head (later dropped); each prefix is serialized as [ISSUE] plus a recent eight-step [WINDOW] with a pinned patch step. Dense F2P targets come from replaying the prefix's bash commands in a fresh environment and scoring intermediate patches with the official evaluator, and Bradley–Terry preference pairs match successful and failed prefixes within step-fraction deciles. Deployment uses Platt scaling plus a logistic meta-score over the calibrated value, the F2P estimate, and the ste

What would settle it

Take the trajectories that triggered restarts; replay each aborted trajectory's bash commands in a fresh instance and compare the resulting repository state to the state the original agent actually had at the abort point. Any divergence from nondeterministic commands, missing dependencies, or environment differences would break the overlay premise. As a second probe, give the restart the same diff as plain text in the prompt instead of as an optional tool: if the resolution gain disappears or reverses, the non-anchoring overlay presentation, not the edit content, is carrying the result.

Watch

Extended reading notes

Core claim

The paper claims that failure in an SWE agent's trajectory can be read from the observable prefix, and that an interrupted failing run contains recoverable value. FailFast is a frozen 0.6B language model adapted with LoRA and trained on terminal labels plus dense fail-to-pass progress targets obtained by replaying the agent's commands; at deployment a calibrated meta-score gates an alarm under a false-positive-rate budget. When the alarm fires, RestartSmart replays the aborted run's bash commands in a fresh instance, captures successive git diff HEAD snapshots, waits for the intended edit to settle, and offers the resulting diff to a fresh same-policy rollout as an optional overlay that star

Load-bearing premise

The restart gains rest on two unproven premises: replaying the aborted run's commands in a fresh environment reproduces its exact repository state, and a fresh agent will verify the offered diff rather than be anchored by it.

Editorial extensions

If this is right

  • At a 5% false-positive budget, one monitor trained on a single policy's trajectories can reclaim 15–20% of agent execution tokens across open and closed policies.
  • At a 25% budget, aborting and restarting with the edit overlay improves Qwen3.6-27B resolution by 5.2 points, versus 0.2 points for cold restart, so the salvaged diff is the source of recovery.
  • Offering the diff as an optional overlay instead of text avoids the anchoring effect, letting a strong agent verify edits without inheriting the failed trajectory's reasoning; a prompt-intervention baseline can even reduce resolution on a strong policy.
  • The false-positive budget is a usable dial: 5% is a pure compute-saving mode, while 10–25% trades up to roughly 44% net token overhead for resolution gains, still below the cost of a second full rollout.
  • Because the monitor needs only observable text, early termination remains deployable on closed-API agents where token-log-probability and hidden-state supervisors are impossible.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the replay determinism assumption holds beyond this benchmark, the same overlay extraction could be combined with other test-time scaling strategies: aborted runs can bank stable diffs that warm-start the next attempt or help rank candidate patches instead of being discarded.
  • The cross-policy transfer of a single 0.6B monitor hints that failure-prefix patterns are largely policy-agnostic; a plausible untested consequence is that one shared stop-early service could monitor a fleet of agents without per-policy training.
  • A direct test of the anchoring story would be to give the restart the same diff as plain text in the prompt or as an already-applied patch; the paper's design predicts the optional overlay will beat both.
  • The paper's energy and carbon savings are estimates derived from FLOP-based scaling, not direct measurements; the token-reclaim numbers themselves are measured and hardware-agnostic, but the environmental figures depend on hardware and data-center conditions.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper proposes FailFast–RestartSmart, a two-stage controller for a single SWE-agent trajectory. FailFast is a 0.6B Qwen-based monitor, trained with LoRA on terminal labels plus dense F2P/P2P targets, that predicts failure from observable prefix text (issue statement plus a window of recent steps) without policy logits or hidden states. On alarm, RestartSmart aborts the trajectory and launches a fresh same-policy rollout, offering the interrupted repository diff as an optional, git-apply-backed overlay. On SWE-bench Verified, a monitor trained solely on Qwen3.6-27B trajectories is reported to transfer to three other policies (including Gemini3-Flash), saving 14.6%–20.4% of execution tokens at a 5% false-positive-rate budget. At 10% and 25% FPR budgets, RestartSmart is reported to improve resolution over cold restart, with Qwen3.6-27B resolution rising from 66.6% to 71.8% at 25% FPR. Appendix A supplies exact McNemar tests and bootstrap confidence intervals for the main comparisons.

Significance. If the reported numbers are correct, the paper makes a useful and timely contribution: a lightweight, policy-agnostic failure monitor that transfers across model families and a restart mechanism that salvages partial repository edits without anchoring the retry in the failed trajectory's reasoning. The evaluation has real strengths: it uses the external SWE-bench Verified benchmark with held-out test instances, pairs comparisons at the trajectory level, applies exact McNemar tests and nonparametric bootstrap CIs, and includes ablations of the observation window, progress targets, training seed count, and monitor backbone. These strengths make the central cost-saving and resolution-gain claims statistically well supported, conditional on the validity of the replay-based overlay extraction and training-target construction. The paper's main weakness is that this replay determinism assumption is load-bearing but not directly validated, and the relation between the out-of-fold training and the held-out test split is stated imprecisely for claims that depend on avoiding test-label leakage.

major comments (2)
  1. [§3.2 and §3.1] The overlay and dense supervision both depend on an unvalidated replay-determinism assumption. Section 3.2 extracts the overlay by replaying the aborted trajectory's bash commands in a fresh instance and recording git diff HEAD after each command; Section 3.1 uses the same replay to compute F2P/P2P targets and Bradley–Terry pairs. Table 3 attributes the +5.2 point resolution gain at 25% FPR on Qwen3.6-27B to the overlay, with FP lost 8.8% vs 27.5% for cold restart; if replay diverges (nondeterministic commands, environment drift, missing dependencies), the overlay can be empty, partial, or wrong, and RestartSmart collapses toward cold restart. No fidelity check is reported. Please add a direct validation: on a sample of trajectories, compare replayed per-command diffs against the original agent's actual working-tree state, report exact-match and hunk-level precision/recall, and quantify
  2. [§3.1, §4.1, Appendix A] The relation between the out-of-fold training, the 350/50/100 instance split, and the statistical tests is not stated precisely enough for the central quantitative claims. The text says 'the monitor is trained entirely out-of-fold; only calibration and threshold selection are fitted on evaluation-fold predictions' and that a 'nested instance-level cross-fit' yields nearly identical operating points. It must be explicit that the 100 test instances (and their 5 seeded trajectories) used in Tables 1–3 and Appendix A were never used to select Platt parameters, meta-score weights, or abort-rule thresholds; otherwise the reported FPR-budgeted recall/savings and resolution gains are optimistic. Please describe the exact procedure and report the selected operating-point parameters (M, K, phi_min, thresholds) per FPR budget.
minor comments (4)
  1. [Table 3 caption] FP lost and TP rec. are conditional percentages of aborted would-pass and aborted would-fail runs, respectively; as written the caption reads like absolute percentages of all runs. Add the denominators to the caption; Appendix A clarifies but the main table is easily misread.
  2. [§4.2] The 'Saved' values in Table 1 are gross token savings; the monitor's inference cost is subtracted only in the energy/carbon paragraph. State this explicitly in the table caption or main text to avoid overstating net compute savings.
  3. [§3.2 / Appendix C] The claim that offering the overlay as a tool rather than as text prevents anchoring is not directly tested. The only comparison is SWE-PRM, which uses a different, stronger model and injects prompts rather than a text summary of the same edits. An ablation with a text-summary version of the overlay would strengthen the design rationale.
  4. [General] Some reported numbers use policy-specific monitors (Table 12) while the abstract emphasizes the transferred monitor (Table 1). Make clear in the abstract and table captions which monitor configuration underlies each headline claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: monitor training and restart evaluation are externally grounded on held-out SWE-bench Verified instances; the replay-determinism caveat is a correctness risk, not a circular reduction.

full rationale

The derivation chain is self-contained against the external benchmark. FailFast's terminal labels and dense F2P targets are training supervisions only: Section 3.1 computes F2P by replaying prefix commands and scoring intermediate patches with the official evaluator, and Appendix E states that "gold F2P/P2P coverage and the final verdict are label fields and never enter the text." At deployment the meta-score uses the model's predicted F2P estimate, not the ground-truth score, so the prediction is not equivalent to its input. Calibration and threshold selection are restricted to validation folds with a nested instance-level cross-fit check, and the reported transfer/savings are on 100 held-out test instances. RestartSmart's overlay is a deterministic git-diff extraction from the aborted trajectory (Section 3.2), not a fitted parameter; the comparison to cold restart on the same aborted runs is a controlled experiment isolating the overlay's effect. The only self-citations (Lyu et al. 2026a, 2026b) are incidental mentions of framework usage and practitioner studies, not load-bearing. The paper does not directly validate the replay-determinism assumption behind F2P label generation and overlay extraction (Sections 3.1-3.2); if replayed commands diverge, the labels and overlays could be corrupted. That is an empirical robustness threat, not a circularity: the reported numbers would be wrong, but they are not forced by construction from the method's inputs.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The paper is an empirical ML systems paper; it introduces no new physical or mathematical entities. The load-bearing ledger entries are the data-generation assumptions (replay fidelity, representative benchmark) and the fitted calibration/abort parameters. The headline numbers, 20.4% token saving and +5.2pt resolution, are experimental measurements on a held-out set, not derived quantities, so their support rests on the trajectory corpus and the calibration procedure.

free parameters (6)
  • Meta-score logistic regression weights (w, b) = not reported
    Fused calibrated value v, F2P estimate, and step fraction into the per-step alarm score; fitted on validation predictions in Section 3.1.
  • Abort rule parameters (step-fraction floor phi_min=0.20, count-M, sustained-K) = grid-searched, exact values not reported
    Grid search over floors 0.20-0.50, M in [1,12], K in [1,8], maximizing recall at target FPR on validation (Section 3.1, Appendix D).
  • Patience window g=5 edit-free steps = 5
    Chosen via the Wait vs. Immediate comparison in Table 6; the overlay cut waits until five steps pass without edits.
  • Observation window k=8 = 8
    Chosen via ablation in Table 9; k=8 matches k=16 within noise and halves input cost.
  • Training seed count N=11 = 11
    Chosen via ablation in Table 11; N=11 consistently improves recall over N=3 or N=6.
  • Loss weights lambda_f=0.5, lambda_r=0.25, lambda_p=0 = 0.5/0.25/0
    Selected on validation by recall at budgeted FPRs; P2P signal dropped for no gain (Appendix B, Section 4.3).
assumptions (5)
  • domain assumption SWE-bench Verified and mini-swe-agent are representative proxies for SWE agent tasks
    The conclusion explicitly limits scope: 'Our evaluation is limited to SWE-bench Verified with mini-swe-agent, so the findings may not generalize to other software engineering tasks or agent frameworks.'
  • domain assumption Replaying bash commands in a fresh instance reproduces the original repository state
    Section 3.2 extracts the overlay from replayed commands and git diff snapshots; replay determinism is not validated.
  • domain assumption Inference energy scales approximately as 2N FLOPs/token with parameter count N
    Section 4.2 converts token savings into energy and carbon savings using Kaplan et al. (2020) scaling.
  • domain assumption Dense F2P/P2P targets computed by scoring intermediate patches with the official evaluator are faithful progress signals
    Section 3.1 uses these replayed targets as auxiliary supervision; noisy replay would bias the progress heads.
  • standard math Platt scaling and Bradley-Terry pairing are valid calibration and ranking objectives
    Used in Section 3.1 for probability calibration and for constructing instance-invariant preference pairs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Fail-Fast, Restart-Smart: Early Failure Prediction and Restart for SWE Agentic Tasks." pith.science (2026). https://pith.science/paper/TLY5TB4X

@misc{pith2026260803222,
  author       = {Pith},
  title        = {Pith review of: Fail-Fast, Restart-Smart: Early Failure Prediction and Restart for SWE Agentic Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TLY5TB4X}},
  note         = {Machine review of arXiv:2608.03222}
}
read the original abstract

Software engineering (SWE) agents resolve repository-level issues through long trajectories that grow increasingly expensive as context accumulates. Failed runs tend to be longer and exhibit redundant exploration or looping, suggesting that some failures may be detectable before completion. Early termination, however, risks interrupting trajectories that would otherwise succeed; conversely, an unsuccessful trajectory may still contain useful repository edits. We present FailFast-RestartSmart, a two-stage controller for a single active trajectory. FailFast is a lightweight 0.6B monitor trained with terminal and dense fail-to-pass supervision to predict failure from observable prefixes without policy logits or hidden states. Upon an alarm, RestartSmart launches a fresh same-policy rollout without prior prompt history and offers the interrupted repository diff as an optional overlay that the agent may inspect, apply, or discard. On SWE-bench Verified, a monitor trained solely on Qwen3.6-27B trajectories transfers to three other policies, including a closed-API model, and saves 14.6%-20.4% of execution tokens at a target 5% false-positive rate; on Qwen3.6-27B, its 20.4% saving exceeds the 12.5% achieved by our per-step AgentStop adaptation. At a target 25% false-positive rate, RestartSmart raises Qwen3.6-27B resolution from 66.6% to 71.8%, whereas cold restart reaches only 66.8%. Together, these results support early termination with sequential same-policy recovery.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

53 extracted references · 22 canonical work pages

  1. [1]

    Aggarwal, P.; Madaan, A.; Yang, Y.; and Mausam. 2023. Let's Sample Step by Step: Adaptive-Consistency for Efficient Reasoning and Coding with LLMs . In Conference on Empirical Methods in Natural Language Processing, 12375--12396. Association for Computational Linguistics

    work page 2023

  2. [2]

    Aggarwal, V.; Kamal, O.; Japesh, A.; Jin, Z.; and Schölkopf, B. 2025. DARS : Dynamic Action Re-Sampling to Enhance Coding Agent Performance by Adaptive Tree Traversal. In Annual Meeting of the Association for Computational Linguistics, 19808--19855. Association for Computational Linguistics

    work page 2025

  3. [3]

    Antoniades, A.; Örwall, A.; Zhang, K.; Xie, Y.; Goyal, A.; and Wang, W. 2024. Swe-search: Enhancing software agents with monte carlo tree search and iterative refinement. In International Conference on Learning Representations, volume 2025, 64485--64515

    work page 2024

  4. [4]

    Baidya, A.; Liang, X.; Guo, R.; Gao, X.; and Das, K. 2026. When Evidence is Sparse: Weakly Supervised Early Failure Alerting in Dialogs and LLM-Agent Trajectories. arXiv preprint arXiv:2606.05414

    arXiv 2026

  5. [5]

    Barke, S.; Goyal, A.; Khare, A.; Singh, A.; Nath, S.; and Bansal, C. 2026. AgentRx : Diagnosing AI Agent Failures from Execution Trajectories. arXiv:2602.02475

    arXiv 2026

  6. [6]

    Bouzenia, I.; and Pradel, M. 2025. Understanding Software Engineering Agents: A Study of Thought-Action-Result Trajectories. In 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE), 2846--2857. IEEE

    work page 2025

  7. [7]

    A.; and Terry, M

    Bradley, R. A.; and Terry, M. E. 1952. Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. Biometrika, 39(3/4): 324

    work page 1952

  8. [8]

    Ceka, I.; Pujar, S.; Ramji, S.; Buratti, L.; Kaiser, G.; and Ray, B. 2025. Understanding Automated Program Repair Agents Through the Lens of Traceability: An Empirical Study. arXiv:2506.08311

    arXiv 2025

Show all 53 references
  1. [9]

    Z.; Yang, S.; Agrawal, L

    Cemri, M.; Pan, M. Z.; Yang, S.; Agrawal, L. A.; Chopra, B.; Tiwari, R.; Keutzer, K.; Parameswaran, A.; Klein, D.; Ramchandran, K.; et al. 2025. Why Do Multi-Agent LLM Systems Fail? arXiv:2503.13657

    2025 arXiv

  2. [10]

    Chen, W.; He, W.; Xi, Z.; Guo, H.; Hong, B.; Zhang, J.; Zheng, R.; Li, N.; Gui, T.; Li, Y.; et al. 2025 a . Better Process Supervision with Bi-directional Rewarding Signals. In Annual Meeting of the Association for Computational Linguistics, 14471--14485. Association for Compu...

    2025

  3. [11]

    Chen, X.; Li, G.; Wang, Z.; Jin, B.; Qian, C.; Wang, Y.; Wang, H.; Zhang, Y.; Zhang, D.; Zhang, T.; et al. 2025 b . Rm-r1: Reward modeling as reasoning. arXiv preprint arXiv:2505.02387

    2025

  4. [12]

    Chen, Z.; Ma, W.; and Jiang, L. 2025. Beyond Final Code: A Process-Oriented Error Analysis of Software Development Agents in Real-World GitHub Scenarios. arXiv:2503.12374

    2025 arXiv

  5. [13]

    J.; Jaffe, O.; Sherburn, D.; Starace, G.; Mays, E.; Dias, R.; Aljubeh, M.; Glaese, M.; Jimenez, C

    Chowdhury, N.; Aung, J.; Shern, C. J.; Jaffe, O.; Sherburn, D.; Starace, G.; Mays, E.; Dias, R.; Aljubeh, M.; Glaese, M.; Jimenez, C. E.; Yang, J.; Ho, L.; Patwardhan, T.; Liu, K.; and Madry, A. 2024. Introducing SWE -bench Verified

    2024

  6. [14]

    Ding, Y.; and Zhang, L. 2026. SWE-Replay : Efficient Test-Time Scaling for Software Engineering Agents. arXiv:2601.22129

    2026

  7. [15]

    Fan, S.; Ye, X.; Huo, Y.; Chen, Z.-Y.; Guo, Y.; Yang, S.; Yang, W.; Ye, S.; Chen, J.; Chen, H.; et al. 2026. AgentProcessBench : Diagnosing Step-Level Process Quality in Tool-Using Agents. arXiv:2603.14465

    2026 arXiv

  8. [16]

    M.; He, P.; and Hassan, A

    Fan, Z.; Vasilevski, K.; Lin, D.; Chen, B.; Chen, Y.; Zhong, Z.; Zhang, J. M.; He, P.; and Hassan, A. E. 2025. Swe-effi: Re-evaluating software ai agent system effectiveness under resource constraints. arXiv preprint arXiv:2509.09853

    2025

  9. [17]

    Gandhi, S.; Tsay, J.; Ganhotra, J.; Kate, K.; and Rizk, Y. 2025. When Agents go Astray: Course-Correcting SWE Agents with PRMs. arXiv:2509.02360

    2025

  10. [18]

    Gao, P.; Tian, Z.; Meng, X.; Wang, X.; Hu, R.; Xiao, Y.; Liu, Y.; Zhang, Z.; Chen, J.; Gao, C.; et al. 2025. Trae Agent: An LLM -based Agent for Software Engineering with Test-time Scaling. arXiv:2507.23370

    2025 arXiv

  11. [19]

    M.; Harman, M.; Lou, Y.; Liu, Y.; and Chen, Z

    Guo, Y.; Xiao, Y.; Zhang, J. M.; Harman, M.; Lou, Y.; Liu, Y.; and Chen, Z. 2026. EET : Experience-Driven Early Termination for Cost-Efficient Software Engineering Agents. arXiv:2601.05777

    2026 arXiv

  12. [20]

    Han, H.; Xie, J.; Ma, X.; Zhu, W.; Zhang, Z.; Long, Z.; Chen, H.; and Ye, Q. 2026. SWE-TRACE: Optimizing Long-Horizon SWE Agents Through Rubric Process Reward Models and Heuristic Test-Time Scaling. arXiv:2604.14820

    2026 arXiv

  13. [21]

    J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W

    Hu, E. J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W. 2021. LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685

    2021 arXiv

  14. [22]

    Huang, Y.; Bie, B.; NA, Z.; Ruan, W.; Lei, S.; Yue, Y.; and He, X. 2026. Understanding the Anchoring Effect of LLM with Synthetic Data: Existence, Mechanism, and Potential Mitigations. In ICLR 2026 Workshop - From Human Cognition to AI Reasoning: Models, Methods, and Applications

    2026

  15. [23]

    Jain, N.; Singh, J.; Shetty, M.; Zheng, L.; Sen, K.; and Stoica, I. 2025. R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agents. arXiv:2504.07164

    2025 arXiv

  16. [24]

    E.; Yang, J.; Wettig, A.; Yao, S.; Pei, K.; Press, O.; and Narasimhan, K

    Jimenez, C. E.; Yang, J.; Wettig, A.; Yao, S.; Pei, K.; Press, O.; and Narasimhan, K. 2024. Swe-bench: Can language models resolve real-world github issues? In International Conference on Learning Representations, volume 2024, 54107--54157

    2024

  17. [25]

    B.; Chess, B.; Child, R.; Gray, S.; Radford, A.; Wu, J.; and Amodei, D

    Kaplan, J.; McCandlish, S.; Henighan, T.; Brown, T. B.; Chess, B.; Child, R.; Gray, S.; Radford, A.; Wu, J.; and Amodei, D. 2020. Scaling Laws for Neural Language Models. arXiv:2001.08361

    2020 arXiv

  18. [26]

    Kim, N.; and Yoo, S. 2026. Atropos: Improving Cost-Benefit Trade-off of LLM -based Agents under Self-Consistency with Early Termination and Model Hotswap. arXiv:2604.15075

    2026 arXiv

  19. [27]

    Li, X.; Chen, K.; Long, Y.; Bai, X.; Xu, Y.; and Zhang, M. 2025. Generator-Assistant Stepwise Rollback Framework for Large Language Model Agent. arXiv:2503.02519

    2025

  20. [28]

    Li, Y.; Yuan, P.; Feng, S.; Pan, B.; Wang, X.; Sun, B.; Wang, H.; and Li, K. 2024. Escape Sky-high Cost: Early-stopping Self-Consistency for Multi-step Reasoning. In International Conference on Learning Representations

    2024

  21. [29]

    Lin, J.; Guo, Y.; Han, Y.; Hu, S.; Ni, Z.; Wang, L.; Chen, M.; Liu, H.; Chen, R.; He, Y.; et al. 2025 a . SE-Agent : Self-Evolution Trajectory Optimization in Multi-Step Reasoning with LLM -Based Agents. arXiv:2508.02085

    2025

  22. [30]

    Lin, Y.; Wang, Z.; Liu, M.; Shan, Y.; Bai, L.; Zhang, J.; Jin, X.; Chen, B.; Su, J.; Wang, X.; et al. 2026. BAGEN : Are LLM Agents Budget-Aware? arXiv:2606.00198

    2026 arXiv

  23. [31]

    Lin, Z.; Tang, Y.; Yao, X.; Yin, D.; Hu, Z.; Sun, Y.; and Chang, K.-W. 2025 b . QLASS : Boosting Language Agent Inference via Q-Guided Stepwise Search. In International Conference on Machine Learning

    2025

  24. [32]

    Liu, S.; Chen, Y.; Krishna, R.; Sinha, S.; Ganhotra, J.; and Jabbarvand, R. 2025. Process-Centric Analysis of Agentic Software Systems. Proceedings of the ACM on Programming Languages, 10(OOPSLA1)

    2025

  25. [33]

    J.; Widyasari, R.; He, J.; Niu, Y.; Yang, C.; Chen, J.; Yang, Z.; Lawall, J.; et al

    Lyu, Y.; Shi, J.; Kang, H. J.; Widyasari, R.; He, J.; Niu, Y.; Yang, C.; Chen, J.; Yang, Z.; Lawall, J.; et al. 2026 a . AgentSZZ: Teaching the LLM Agent to Play Detective with Bug-Inducing Commits. arXiv preprint arXiv:2604.02665

    2026 arXiv

  26. [34]

    Lyu, Y.; Williams, D.; Shi, J.; Sun, Z.; Peng, C.; Yang, Z.; Sarro, F.; and Lo, D. 2026 b . How Do Practitioners Build SE Agents? Insights from a Mixed-Methods Study. arXiv preprint arXiv:2607.10856

    2026 arXiv

  27. [35]

    Majgaonkar, O.; Fei, Z.; Li, X.; Sarro, F.; and Ye, H. 2025. Understanding Code Agent Behaviour: An Empirical Study of Success and Failure Trajectories. arXiv:2511.00197

    2025

  28. [36]

    Mehtiyev, T.; and Assun c \ a o, W. 2026. Beyond Resolution Rates: Behavioral Drivers of Coding Agent Success and Failure. arXiv:2604.02547

    2026 arXiv

  29. [37]

    M.; Paltenghi, M.; and Chandra, S

    Nanda, R.; Maddila, C.; Jha, S.; Khan, E. M.; Paltenghi, M.; and Chandra, S. 2026. Wink: Recovering from Misbehaviors in Coding Agents. arXiv:2602.17037

    2026

  30. [38]

    OpenLM AI . 2026. SWE-bench +

    2026

  31. [39]

    Pan, J.; Wang, X.; Neubig, G.; Jaitly, N.; Ji, H.; Suhr, A.; and Zhang, Y. 2024. Training Software Engineering Agents and Verifiers with SWE -Gym. In Singh, A.; Fazel, M.; Hsu, D.; Lacoste-Julien, S.; Berkenkamp, F.; Maharaj, T.; Wagstaff, K.; and Zhu, J., eds., International ...

    2024

  32. [40]

    Park, Y.-J.; Greenewald, K.; Alim, K.; Wang, H.; and Azizan, N. 2025. Know What You Don't Know: Uncertainty Calibration of Process Reward Models. arXiv:2506.09338

    2025

  33. [41]

    S.; and Haddadi, H

    Pham, D.; Katevas, K.; Shamsabadi, A. S.; and Haddadi, H. 2026. AgentStop : Terminating Local AI Agents Early to Save Energy in Consumer Devices. arXiv:2605.15206

    2026 arXiv

  34. [42]

    Platt, J. 1999. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 10(3): 61--74

    1999

  35. [43]

    Raghavendra, M.; Gunjal, A.; Liu, B.; and He, Y. 2026. Agentic Rubrics as Contextual Verifiers for SWE Agents. arXiv:2601.04171

    2026

  36. [44]

    Ruan, K.; Huang, Z.; Zhou, Z.; Wei, Q.; Wang, X.; and Sun, H. 2026. Doomed from the Start: Early Abort of LLM Agent Episodes via a Recall-Controlled Probe Cascade. arXiv:2607.06503

    2026 arXiv

  37. [45]

    Sahoo, P.; Mittal, G.; Li, X.; Ma, S.; Steenhoek, B.; Lin, P.; and Hu, Y. 2026. AgentLens: Revealing The Lucky Pass Problem in SWE-Agent Evaluation. arXiv preprint arXiv:2605.12925

    2026 arXiv

  38. [46]

    Shum, K.; Hui, B.; Chen, J.; Zhang, L.; x., W.; Yang, J.; Huang, Y.; Lin, J.; and He, J. 2025. SWE-RM : Execution-free Feedback For Software Engineering Agents. arXiv:2512.21919

    2025

  39. [47]

    Sun, H.; Haider, M.; Zhang, R.; Yang, H.; Qiu, J.; Yin, M.; Wang, M.; Bartlett, P.; and Zanette, A. 2024. Fast Best-of-N Decoding via Speculative Rejection. In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  40. [48]

    Vasudev, R.; Russak, M.; Bikel, D.; and Alshikh, W. 2026. The Intervention Paradox: Accurate Failure Prediction in Agents Does Not Imply Effective Failure Prevention. arXiv:2602.03338

    2026

  41. [49]

    Wang, X.; Chen, V.; Ji, H.; and Neubig, G. 2026. A Rubric-Supervised Critic from Sparse Real-World Outcomes. arXiv:2603.03800

    2026

  42. [50]

    Xi, Z.; Liao, C.; Li, G.; Yang, Y.; Chen, W.; Zhang, Z.; Wang, B.; Jin, S.; Zhou, Y.; Guan, J.; et al. 2025. AgentPRM : Process Reward Models for LLM Agents via Step-Wise Promise and Progress. arXiv:2511.08325

    2025

  43. [51]

    E.; Wettig, A.; Lieret, K.; Yao, S.; Narasimhan, K

    Yang, J.; Jimenez, C. E.; Wettig, A.; Lieret, K.; Yao, S.; Narasimhan, K. R.; and Press, O. 2024. SWE -agent: Agent-Computer Interfaces Enable Automated Software Engineering. In Neural Information Processing Systems, 50528--50652. Neural Information Processing Systems Foundati...

    2024

  44. [52]

    Zhang, B.; Zhu, J.; Shi, Z.; Liu, D.; and Tang, R. 2026. AgentForesight : Online Auditing for Early Failure Prediction in Multi-Agent Systems. arXiv:2605.08715

    2026 arXiv

  45. [53]

    Zhang, S.; Yin, M.; Zhang, J.; Liu, J.; Han, Z.; Zhang, J.; Li, B.; Wang, C.; Wang, H.; Chen, Y.; et al. 2025. Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems. In International Conference on Machine Learning

    2025

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.