Pith. sign in

REVIEW 3 major objections 5 minor 42 references

Online Monitoring and Corrective Steering of Programming Agents

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LivePlan, a deterministic monitor paired with a selectively consulted advisor LLM, lifts programming-agent issue-resolution rates by up to 15.2 percentage points at an added cost of about $0.08 per task.

desk verdict A credible, unusually thorough evaluation of a genuinely new judge-advisor decoupling; the θ_p threshold needs a sensitivity sweep and the success tables need confidence intervals before the headline claim is taken at face value. read the letter →

arxiv 2608.06701 v1 pith:JVAQ4LKP submitted 2026-08-07 cs.SE cs.AIcs.CLcs.LG

Shuyang Liu , Saman Dehghan , Ji Young Kim , Jatin Ganhotra , Martin Hirzel , Reyhaneh Jabbarvand This is my paper · ORCID
classification cs.SEcs.AIcs.CLcs.LG
keywords LivePlanonlinemonitoringcorrectivesteeringbehavioraldriftprogrammingagentsprocess-centrictrajectoryanalysisrule-basedmonitorLLMadvisor
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tackles a practical failure mode of long-horizon coding agents: they drift off their plan, repeat useless actions, or terminate without a patch. Its claim is that this drift can be caught cheaply and reliably by making judgment deterministic and rule-based, then calling an LLM advisor only after a problem has already been flagged. On SWE-bench Verified and SWE-bench Pro, LivePlan raises SWE-agent's resolved-instance rate by up to 15.2 percentage points (average 9.9%) for an extra cost of about $0.08 per instance, with gains concentrated on medium and hard issues and few regressions on runs that the vanilla agent already solved. The design matters because it separates judging from advising, avoiding both hallucinated interventions of replanning systems and the cost and false alarms of periodic LLM judges.

What carries the argument

The load-bearing objects are GraphEctory and Langutory, two process-centric trajectory representations adapted from prior work and extended here: GraphEctory turns a trajectory into a directed graph whose nodes are distinct actions plus separate thought nodes, and Langutory abstracts the execution into a phase alphabet covering navigation, reproduction, patching, validation, and general work. Deterministic rules over these representations fire on ten signals: repeated actions appear as back-edges, thought or action oscillation as self-loops or multi-node cycles, long stagnation as more than $\theta_p = 7$ consecutive same-phase steps, and plan violation as missing phases. The monitor's verdict determines whether to block the offending action or let it proceed with appended advice, and a cooling period of $\theta_c = 5$ steps limits how often the advisor LLM is consulted. This mechanism carries the entire contribution, since it removes the LLM from the judging path and keeps advice targeted and timely.

What would settle it

Run LivePlan on a fixed model-benchmark pair with the stagnation threshold set to, say, 3, 5, 7, 10, and 14: if the reported gains collapse or reverse outside a narrow window around 7, then the threshold, not the monitoring principle, is carrying the result. A complementary check is to have independent annotators label trajectory segments as harmful drift or legitimate work and measure how often the deterministic monitor agrees with those labels.

Watch

Extended reading notes

Core claim

The paper's central claim is that prior steering techniques conflate two jobs that should be separated: judging whether a trajectory is in trouble, and advising how to fix it. An LLM asked to do both is biased to find problems, so it can derail healthy runs, and periodic checks either cost too much or arrive too late. LivePlan detects behavioral drift with deterministic rules over two trajectory representations, and only when a rule fires does it invoke an advisor LLM to produce a single high-level next-step correction. The paper argues this division of labor is what produces consistent resolution-rate gains of up to 15.2% (average 9.9%) across three executor models and two benchmarks, at modest cost, while preserving most vanilla successes.

Load-bearing premise

The fixed threshold of seven consecutive steps in one phase, together with the hand-defined deterministic signal set, correctly separates harmful drift from legitimate long-horizon work on every executor and benchmark tested, even though the threshold was calibrated from vanilla trajectories and never re-calibrated for LivePlan trajectories.

Editorial extensions

If this is right

  • If deterministic monitoring plus on-demand advice is the right division of labor, other long-horizon agent tasks can adopt the pattern without retraining the executor.
  • The reported gains concentrate on medium and hard instances, so the method is most useful exactly where coding agents struggle.
  • LivePlan preserves most vanilla successes: regressions are low, so enabling the intervention carries little downside on runs that already work.
  • Monitoring costs only a few milliseconds and advisor calls cost $0.01 to $0.06 per instance, making the technique cheap enough for routine use.
  • LivePlan trajectories show higher plan compliance, especially in executing required phases in order, while still allowing productive out-of-plan steps such as extra regression testing.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not test whether the fixed stagnation threshold $\theta_p=7$, derived from vanilla statistics, remains optimal when advisor-extended trajectories change phase-length distributions; an easy extension would be to re-calibrate the threshold per executor model or adapt it online.
  • The judging/advising separation is not repository-specific: the signals are action-level and phase-level, so the same monitor could plausibly steer long-horizon agents in research, data analysis, or robotics, provided a phase taxonomy exists.
  • The manual analysis attributes most LivePlan regressions to executors failing to follow correct advice, not to wrong advice; this points to intervention-conditioned post-training as the next lever, a direction the paper itself names.
  • A cheap but decisive follow-up would be to vary $\theta_p$ across a grid and check whether gains persist; if they collapse outside a narrow window, the threshold rather than the monitoring principle is doing the work.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper introduces LivePlan, an online monitoring and corrective-steering layer for programming agents. LivePlan separates a deterministic rule-based monitor from an LLM-based advisor: the monitor analyzes two trajectory representations, GRAPHECTORY and LANGUTORY, for behavioral drifts such as plan violations, oscillation, stagnation, and repeated actions, and only when a drift is detected does the advisor issue predefined or custom next-step advice. The system is implemented on top of SWE-agent and evaluated on SWE-bench Verified and SWE-bench Pro with three executor LLMs and two advisor LLMs. Compared with vanilla SWE-agent, LivePlan reports consistent resolution-rate gains of up to 15.2 percentage points (average 9.9), fewer regressions, and small additional cost, supported by a deterministic-subset analysis and by process-centric trajectory and manual root-cause analyses.

Significance. If the reported results hold, LivePlan is a valuable, pluggable steering mechanism for long-horizon agentic software engineering. The core design idea—decoupling judging from advising and replacing LLM judging with deterministic rules—directly addresses documented failure modes of prior work, including hallucinated or sycophantic LLM judgments and costly periodic interventions. The paper's strengths include a large empirical study (7,752 trajectories), publicly available artifacts, a deterministic-subset robustness check that preserves the method ranking, plan-compliance and trajectory-similarity analyses, and a careful manual investigation of improvement and regression root causes. The main reservation is that the headline success-rate claims rest on single-run percentages and on monitor thresholds calibrated from vanilla statistics without a sensitivity analysis, so the empirical evidence is suggestive rather than conclusive.

major comments (3)
  1. [§III-A, Tables II–III] The central claim of consistent gains (up to 15.2%, average 9.9%) is supported only by single-run success rates, with no confidence intervals, significance tests, or repeated paired rollouts for LivePlan outcomes. The RQ5 deterministic-subset analysis (§III-E) controls for vanilla run-to-run variation, but it does not estimate the variance of LivePlan's own outcomes, and the paper's own §III-D guidance states that 'the evaluation should use repeated paired rollouts to estimate how an intervention changes the probability of resolution, rather than drawing conclusions from a single trajectory.' Because the headline is a reliability claim, please report repeated LivePlan runs for at least one benchmark/executor pair, or provide a paired statistical comparison (e.g., McNemar's test) for Tables II–III.
  2. [§III, hyperparameters paragraph] The long-stagnation threshold θ_p = 7 is calibrated from vanilla SWE-agent phase-length statistics (mean maximum consecutive phase lengths of 5.64 for resolved and 7.61 for unresolved instances) and is never re-estimated on LivePlan trajectories, and no sensitivity sweep is reported for θ_p, θ_c, or θ_i. This is load-bearing because Table VI shows that Long Stagnation fires in 26.1–48.7% of LivePlan trajectories depending on the setting; changing θ_p changes how often advice is injected and could therefore explain a substantial part of the reported gains. The RQ5 deterministic-subset analysis controls for baseline volatility, not threshold dependency. Please add a sweep over θ_p (and ideally θ_c) with all other components fixed, for at least one executor/benchmark pair, and report the resulting success rates.
  3. [§I and §III-B, Table V] The abstract's cost claim ('only an additional cost of $0.08 per instance') is a mean over settings that hides substantial variation: using the Table II cost columns, the additional total cost ranges from about +$0.26 for Gemini-2.5-Flash on SWE-bench Pro to −$0.05 for MiniMax-M2.5 on SWE-bench Verified. Please report the per-pair cost distribution with confidence intervals and state explicitly that the $0.08 figure is an average across the six model-benchmark pairs. The cost accounting in Table V should also be clarified, since the DeepSeek-V3 Pro row (executor 0.12 plus advisor 0.04) does not obviously reconcile with the 0.15 average in Table II.
minor comments (5)
  1. [Abstract and §I] The text contains several missing spaces around the method name (e.g., 'LIVEPLANto', 'LIVEPLANand'); please proofread the rendering of LivePlan throughout.
  2. [Figure 8] The stacked bar chart in Figure 8 is difficult to read because the per-pair labels and legend are compressed; consider a faceted plot or a table with larger fonts.
  3. [Table V] The column headings 'Cost ($) Exec. Adv. Mon.' are ambiguous: it is unclear whether 'Cost' is total cost or executor-only cost, and the DeepSeek-V3 Pro row should be reconciled with the average cost in Table II.
  4. [§III-D] The manual root-cause analysis reports independent coding followed by consensus, but no inter-annotator agreement statistic is reported; Cohen's kappa or a similar measure would strengthen the qualitative claims.
  5. [§III-A] The comparison against SAGE, Periodic Advisor, and Predefined Advice is reported only for SWE-bench Pro; the abstract's claim that LivePlan 'consistently outperforms alternative approaches' should be qualified to the benchmark on which those baselines were run.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central resolution-rate claim is externally benchmarked against SWE-bench gold patches, and the self-cited representations and plan-compliance metrics are reused tools rather than the source of the headline result.

full rationale

The paper's main claim is that LivePlan improves issue resolution rates relative to vanilla SWE-agent and other baselines; this is evaluated against the external SWE-bench Verified and SWE-bench Pro gold-patch labels, so the target result is not defined in terms of the method's own signals. The hyperparameter θ_p = 7 is estimated from vanilla SWE-agent phase-length statistics, including resolved/unresolved splits, but the paper does not derive the reported improvement from that estimate: there is no equation in which the predicted gain equals the fitted threshold, and the evaluation uses independent runs with different intervention mechanisms. The self-citations to GRAPHECTORY/LANGUTORY [17] and the plan-compliance metrics [18] are reuse of published representations and metrics by overlapping authors; they are load-bearing only as implementation infrastructure, not as the justification for the resolution-rate improvement, which is externally falsifiable against gold patches and multiple baselines. The absence of a θ_p sensitivity sweep is a legitimate robustness concern, but it is not circularity: the threshold is not fitted to LivePlan success labels, and the central claim does not reduce by construction to any fitted parameter or cited prior result.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

No new physical or domain entities are introduced. The method's constructs (GRAPHECTORY thought nodes, Navigation and Reproduction phase splits) are representational extensions of prior work [17] with evidence only through the reported evaluations. The numeric hyperparameters theta_p, theta_c, and theta_i are hand-set or fitted to vanilla baseline statistics, and the stronger-advisor pairing is an explicit assumption that affects interpretation of the gains.

free parameters (3)
  • theta_p (long stagnation threshold) = 7 consecutive steps in the same phase
    Set from average maximum consecutive phase lengths of vanilla runs (5.64 resolved, 7.61 unresolved) before any intervention; a hand-chosen boundary that separates normal progress from likely stagnation, not re-calibrated for LivePlan trajectories.
  • theta_c (advisor cooldown) = 5 trajectory steps
    Set to match the SWE-PRM periodic evaluation interval, making the Periodic Advisor ablation fair; the choice is otherwise arbitrary and no sensitivity analysis is provided.
  • theta_i (max consecutive blocking interventions) = 5
    Hand-set to prevent repeated blocking from stalling the executor indefinitely; the paper does not study the impact of this limit on resolution rate.
assumptions (5)
  • domain assumption GRAPHECTORY and LANGUTORY trajectory abstractions from prior work [17] correctly encode phases, back-edges, oscillations, and skipped phases.
    These same-author representations are the core inputs to every drift signal in Section II-A; the paper does not independently validate the accuracy of the phase classifier on these benchmarks.
  • domain assumption The phase classification and action parsing are accurate for all repositories, executors, and problem types in SWE-bench Verified and SWE-bench Pro-Python.
    Misclassified actions would create false drift detections and spurious interventions; the paper reports no parser accuracy or misclassification analysis.
  • domain assumption The rule-based signals are general enough to transfer across executors and benchmarks without per-repository tuning.
    The authors claim generality in Section II-A, but the fixed thresholds are computed from vanilla runs on these specific benchmarks and are not tested on other repositories or languages.
  • domain assumption Passing hidden tests in SWE-bench is a valid proxy for correctly resolving a GitHub issue.
    This is the standard benchmark assumption, but test-pass does not guarantee that the patch is semantically correct or matches the maintainer's intent.
  • domain assumption Using a strictly stronger advisor LLM is an acceptable design, and the advisor's capability is not the true source of the reported gains.
    Every LivePlan configuration pairs the executor with a stronger model; the Predefined Advice and Periodic Advisor controls share that pairing, but the vanilla baseline does not, so part of the gain could come from the advisor's knowledge rather than from the monitoring mechanism.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Online Monitoring and Corrective Steering of Programming Agents." pith.science (2026). https://pith.science/paper/JVAQ4LKP

@misc{pith2026260806701,
  author       = {Pith},
  title        = {Pith review of: Online Monitoring and Corrective Steering of Programming Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JVAQ4LKP}},
  note         = {Machine review of arXiv:2608.06701}
}
abstract

Fixing GitHub issues in large-scale projects is a long-horizon task, especially when a fix requires changes across multiple locations or the issue description lacks the information needed to localize and repair it. As a result, agents traverse long trajectories that are prone to inefficiency and error: they drift away from their intended plan, repeat failed actions, or terminate without a working patch. This paper proposes LivePlan to monitor, detect, and correct such behavioral inefficiencies and drifts in real time. LivePlan decouples judging from advising: a deterministic, rule-based monitor examines general signals over the trajectory to detect issues without invoking an LLM, and only when an issue is detected does it consult an advisor LLM for a high-level, next-step correction. This design avoids the misleading re-planning and costly interventions of prior approaches. We implement LivePlan on top of SWE-agent and evaluate it using five LLMs (three as executor agents and two as advisors) across SWE-bench Verified and SWE-bench Pro. Compared to vanilla SWE-agent, LivePlan notably improves issue resolution rates, achieving consistent gains of up to 15.2% (average: 9.9%), while incurring only an additional cost of $0.08 per instance. The additional solutions concentrate on medium and hard instances. LivePlan consistently outperforms alternative approaches in resolution rate, with minimal regression on already successful runs and new successes on problems that no baseline solves.

Figures

Figures reproduced from arXiv: 2608.06701 by the authors.

Figure 1
Figure 1. An example of SAGE [11] incorrect advice (a new [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. An advisor periodically checking the trajectory fails to [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 4
Figure 4. Examples of behavioral drifts captured by G [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figures from the paper (7 more)
Figure 5
Figure 5. Figure 5: Examples of predefined advice in LIVEPLAN. When invoked, the LLM-based Advisor receives four inputs: the issue description, the committed trajectory slice since the last advice (to minimize context), the latest advice if available, and the predefined advice for the det…
Figure 7
Figure 7. Figure 7: Overlapping resolved instances among techniques. [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 9
Figure 9. Figure 9: Average plan compliance score for Resolved (R) and [PITH_FULL_IMAGE:figures/full_fig_p007_9.png]
Figure 8
Figure 8. Figure 8: Behavioral drifts in final trajectories. [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 10
Figure 10. Figure 10: Trajectory similarity between Vanilla and L [PITH_FULL_IMAGE:figures/full_fig_p008_10.png]
Figure 11
Figure 11. Figure 11: Case study of a U→R transition. Intervention redirects the executor from ineffective validation to successful repair. unproductive exploration and back toward the core problem. (3) Progressive online correction. 182/244 (74.6%) of U→R cases receive multiple interventi…
Figure 12
Figure 12. Figure 12: Case study of a R→U transition under LIVEPLAN. TABLE VIII. Effectiveness in resolving instances of SWE￾bench Pro that deterministically pass or fail in Vanilla runs. Model #Instances Method Success Rate (%) Advisor Executor - DeepSeek (V3) 222 Vanilla 19.82 GPT-5.2 SA…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

42 extracted references · 23 canonical work pages

  1. [1]

    Introducing claude opus 4.7,

    Anthropic, “Introducing claude opus 4.7,” https://www. anthropic.com/news/claude-opus-4-7, 2026

    work page 2026

  2. [2]

    Agentrx: Diagnosing ai agent failures from execution trajectories,

    S. Barke, A. Goyal, A. Khare, A. Singh, S. Nath, and C. Bansal, “Agentrx: Diagnosing ai agent failures from execution trajectories,”arXiv preprint arXiv:2602.02475, 2026

    arXiv 2026

  3. [3]

    Unlocking Model Potentials Through Adaptive Multi-Agent Scaffolding for Efficient Issue Resolution

    Y . Chen, A. Ahmad, Y . Zhou, and R. Jabbarvand, “Un- locking model potentials through adaptive multi-agent scaffolding for efficient issue resolution,”arXiv preprint arXiv:2606.25514, 2026

    work page Pith review arXiv 2026

  4. [4]

    Beyond final code: A process-oriented error analysis of software development agents in real-world github scenarios,

    Z. Chen, W. Ma, and L. Jiang, “Beyond final code: A process-oriented error analysis of software development agents in real-world github scenarios,”arXiv preprint arXiv:2503.12374, 2025

    arXiv 2025

  5. [5]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities,

    G. Comanici, E. Bieber, M. Schaekermann, I. Pasupat, N. Sachdeva, I. Dhillon, M. Blistein, O. Ram, D. Zhang, E. Rosenet al., “Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities,”arXiv preprint arXiv:2507.06261, 2025

    arXiv 2025

  6. [6]

    Can llm be a good path planner based on prompt engineering? mitigat- ing the hallucination for path planning,

    H. Deng, H. Zhang, J. Ou, and C. Feng, “Can llm be a good path planner based on prompt engineering? mitigat- ing the hallucination for path planning,” inInternational Conference on Intelligent Computing. Springer, 2025, pp. 3–15

    work page 2025

  7. [7]

    SWE-Bench Pro: Can AI agents solve long-horizon software engineering tasks?

    X. Deng, J. Da, E. Pan, Y . Y . He, C. Ide, K. Garg, N. Lauffer, A. Park, N. Pasari, C. Rane, K. Sampath, M. Krishnan, S. Kundurthy, S. Hendryx, Z. Wang, C. B. C. Zhang, N. Jacobson, B. Liu, and B. Kenstler, “SWE-Bench Pro: Can AI agents solve long-horizon software engineering tasks?” Sep. 2025. [Online]. Available: https://arxiv.org/abs/2509.16941

    arXiv 2025

  8. [8]

    Trail: Trace reasoning and agen- tic issue localization,

    D. Deshpande, V . Gangal, H. Mehta, J. Krishnan, A. Kan- nappan, and R. Qian, “Trail: Trace reasoning and agen- tic issue localization,”arXiv preprint arXiv:2505.08638, 2025

    arXiv 2025

Show all 42 references
  1. [9]

    Plan-and-act: Improving planning of agents for long- horizon tasks,

    L. E. Erdogan, N. Lee, S. Kim, S. Moon, H. Furuta, G. Anumanchipalli, K. Keutzer, and A. Gholami, “Plan-and-act: Improving planning of agents for long- horizon tasks,” inInternational Conference on Machine Learning (ICML), Jul. 2025. [Online]. Available: https://openreview.net...

    2025

  2. [10]

    When agents go astray: Course-correcting SWE agents with PRMs,

    S. Gandhi, J. Tsay, J. Ganhotra, K. Kate, and Y . Rizk, “When agents go astray: Course-correcting SWE agents with PRMs,” inWorkshop on Scaling Environments for Agents (SEA@NeurIPS), Dec. 2025. [Online]. Available: https://openreview.net/forum?id=wyrcoDNaGO

    2025

  3. [11]

    Self- abstraction from grounded experience for plan-guided policy refinement,

    H. Hayashi, B. Pang, W. Zhao, Y . Liu, A. Gokul, S. Bansal, C. Xiong, S. Yavuz, and Y . Zhou, “Self- abstraction from grounded experience for plan-guided policy refinement,”arXiv preprint arXiv:2511.05931, 2025

    2025

  4. [12]

    SWE-bench: Can language models resolve real-world github issues?

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan, “SWE-bench: Can language models resolve real-world github issues?” inThe Twelfth International Conference on Learning Representations, 2024. [Online]. Available: https:// openreview.net/forum?id...

    2024

  5. [13]

    Traje- val: Decomposing code agent trajectories for fine-grained diagnosis,

    M. Kim, D. Wang, S. Cui, F. Farmahinifarahani, S. Garg, B. Ray, T. Y . Zhuo, R. Mukherjee, and V . Kumar, “Traje- val: Decomposing code agent trajectories for fine-grained diagnosis,”arXiv preprint arXiv:2603.24631, 2026

    2026 arXiv

  6. [14]

    Llm-based agents suf- fer from hallucinations: A survey of taxonomy, methods, and directions,

    X. Lin, Y . Ning, J. Zhang, Y . Dong, Y . Liu, Y . Wu, X. Qi, N. Sun, Y . Shang, K. Wanget al., “Llm-based agents suf- fer from hallucinations: A survey of taxonomy, methods, and directions,”arXiv preprint arXiv:2509.18970, 2025

    2025

  7. [15]

    Deepseek- v3 technical report,

    A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruanet al., “Deepseek- v3 technical report,”arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  8. [16]

    Lost in the middle: How language models use long contexts,

    N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang, “Lost in the middle: How language models use long contexts,”Transactions of the Association for Computational Linguistics, vol. 12, pp. 157–173, 02 2024. [Online]. Available: https://doi.org/1...

    2024 doi

  9. [17]

    Process-centric analysis of agentic software systems,

    S. Liu, Y . Chen, R. Krishna, S. Sinha, J. Ganhotra, and R. Jabbarvand, “Process-centric analysis of agentic software systems,”Proc. ACM Program. Lang., vol. 10, no. OOPSLA1, Apr. 2026. [Online]. Available: https: //doi.org/10.1145/3798271

    2026 doi

  10. [18]

    Evaluating plan compliance in autonomous programming agents,

    S. Liu, S. Dehghan, J. Ganhotra, M. Hirzel, and R. Jab- barvand, “Evaluating plan compliance in autonomous programming agents,”arXiv preprint arXiv:2604.12147, 2026

    2026 arXiv

  11. [19]

    An empirical study on failures in automated issue solving,

    S. Liu, F. Liu, L. Li, X. Tan, Y . Zhu, X. Lian, and L. Zhang, “An empirical study on failures in automated issue solving,”arXiv preprint arXiv:2509.13941, 2025

    2025

  12. [20]

    Agen- thallu: Benchmarking automated hallucination attribution of llm-based agents,

    X. Liu, X. Yang, Z. Li, P. Li, and R. He, “Agen- thallu: Benchmarking automated hallucination attribution of llm-based agents,”arXiv preprint arXiv:2601.06818, 2026

    2026

  13. [21]

    Artifact repository,

    Liu, Shuyang and Dehghan, Saman and Kim, Jiyoung and Ganhotra, Jatin and Hirzel, Martin and Jabbar- 11 vand, Reyhaneh, “Artifact repository,” https://github.com/ Intelligent-CAT-Lab/Agent-Planner.git, 2026

    2026

  14. [22]

    Self-refine: Iterative refinement with self-feedback,

    A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y . Yang, S. Gupta, B. P. Majumder, K. Hermann, S. Welleck, A. Yazdanbakhsh, and P. Clark, “Self-refine: Iterative refinement with self-feedback,” in Conference on Neural Infor...

    2023

  15. [23]

    Minimax m2.5,

    MiniMax, “Minimax m2.5,” https://www.minimax.io/ models/text, 2025

    2025

  16. [24]

    Minimax m3,

    ——, “Minimax m3,” https://www.minimax.io/blog/ minimax-m3, 2026

    2026

  17. [25]

    Wink: Recovering from misbehaviors in coding agents,

    R. Nanda, C. Maddila, S. Jha, E. M. Khan, M. Paltenghi, and S. Chandra, “Wink: Recovering from misbehaviors in coding agents,”arXiv preprint arXiv:2602.17037, 2026

    2026

  18. [26]

    Introducing gpt-5.2-codex,

    OpenAI, “Introducing gpt-5.2-codex,” https://openai. com/index/introducing-gpt-5-2-codex/, 2025

    2025

  19. [27]

    Why do multiagent systems fail?

    M. Z. Pan, M. Cemri, L. A. Agrawal, S. Yang, B. Chopra, R. Tiwari, K. Keutzer, A. Parameswaran, K. Ramchandran, D. Klein, J. E. Gonzalez, M. Zaharia, and I. Stoica, “Why do multiagent systems fail?” in ICLR 2025 Workshop on Building Trust in Language Models and Applications, 2...

    2025

  20. [28]

    Falat: Tracing failures in llm agent tra- jectories via dependency-guided search,

    M. N. Rafi, M. Ahasanuzzaman, D. J. Kim, Z. Wang, and T.-H. Chen, “Falat: Tracing failures in llm agent tra- jectories via dependency-guided search,”arXiv preprint arXiv:2606.00765, 2026

    2026 arXiv

  21. [29]

    Autorubric: Uni- fying rubric-based llm evaluation,

    D. Rao and C. Callison-Burch, “Autorubric: Uni- fying rubric-based llm evaluation,”arXiv preprint arXiv:2603.00077, 2026

    2026 arXiv

  22. [30]

    Towards understanding sycophancy in language models,

    M. Sharma, M. Tong, T. Korbak, D. Duvenaud, A. Askell, S. Bowman, E. Durmus, Z. Hatfield-Dodds, S. Johnston, S. Kravecet al., “Towards understanding sycophancy in language models,” inInternational Con- ference on Learning Representations, vol. 2024, 2024, pp. 110–144

    2024

  23. [31]

    Reflexion: Language agents with verbal reinforcement learning,

    N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao, “Reflexion: Language agents with verbal reinforcement learning,”Advances in neural information processing systems, vol. 36, pp. 8634–8652, 2023

    2023

  24. [32]

    A goal without a plan is just a wish: Efficient and effective global planner training for long-horizon agent tasks,

    S. Si, H. Zhao, K. Luo, G. Chen, F. Qi, M. Zhang, B. Chang, and M. Sun, “A goal without a plan is just a wish: Efficient and effective global planner training for long-horizon agent tasks,” Oct. 2025. [Online]. Available: https://arxiv.org/abs/2510.05608

    2025 arXiv

  25. [33]

    Swe-agent documentation,

    SWE-agent, “Swe-agent documentation,” https://swe-agent.com/latest/usage/batch mode/, 2025

    2025

  26. [34]

    Context rot: Why ai gets worse the longer you chat,

    Teresa Torres, “Context rot: Why ai gets worse the longer you chat,” https://www.producttalk.org/ context-rot/, 2026

    2026

  27. [35]

    The long-horizon task mirage? diagnosing where and why agentic systems break,

    X. J. Wang, H. Bai, Y . Sun, H. Wang, S. Zhang, W. Hu, M. Schroder, B. Mutlu, D. Song, and R. D. Nowak, “The long-horizon task mirage? diagnosing where and why agentic systems break,”arXiv preprint arXiv:2604.11978, 2026

    2026 arXiv

  28. [36]

    Individual comparisons by ranking methods,

    F. Wilcoxon, “Individual comparisons by ranking methods,”Biometrics Bulletin, vol. 1, no. 6, pp. 80–83,

  29. [37]

    Swe-agent: agent- computer interfaces enable automated software engineer- ing,

    J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press, “Swe-agent: agent- computer interfaces enable automated software engineer- ing,” inProceedings of the 38th International Conference on Neural Information Processing Systems, ser. NIPS ’24. Red H...

    2024

  30. [38]

    ReAct: Synergizing reasoning and acting in language models,

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. R. Narasimhan, and Y . Cao, “ReAct: Synergizing reasoning and acting in language models,” inInternational Conference on Learning Representations (ICLR), May

  31. [39]

    ReCAP: Recursive context-aware reasoning and planning for large language model agents,

    Z. Zhang, T. Chen, W. Xu, A. Pentland, and J. Pei, “ReCAP: Recursive context-aware reasoning and planning for large language model agents,” inConference on Neural Information Processing Systems (NeurIPS), Dec. 2025. [Online]. Available: https://openreview.net/forum?id=r2ykUnzuGt

    2025

  32. [40]

    Language agent tree search unifies reasoning, acting, and planning in language models,

    A. Zhou, K. Yan, M. Shlapentokh-Rothman, H. Wang, and Y .-X. Wang, “Language agent tree search unifies reasoning, acting, and planning in language models,” in International Conference on Machine Learning (ICML), Jul. 2024, pp. 62 138–62 160. [Online]. Available: https://procee...

    2024

  33. [1945]

    Available: http://www.jstor.org/stable/ 3001968

    [Online]. Available: http://www.jstor.org/stable/ 3001968

  34. [2023]

    Available: https://openreview.net/forum? id=WE vluYUL-X

    [Online]. Available: https://openreview.net/forum? id=WE vluYUL-X

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.