{"id":"e574f93f-feec-4f7e-9a63-e27234de8c05","arxiv_id":"2608.06701","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A deterministic rule-based monitor that detects coding-agent drift, paired with an LLM advisor called only when needed, raises SWE-agent issue resolution rates by up to 15.2 percentage points at about $0.08 extra per task.","lead":"This paper introduces LivePlan, a system that watches a coding agent's actions with fixed rules and only asks a second, stronger AI for advice when the rules detect that the agent is stuck or drifting. In tests on two GitHub-issue benchmarks, it improved the share of issues fixed by up to 15.2 percentage points at roughly $0.08 extra per task.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The stagnation threshold θ_p = 7 is fitted to vanilla phase-length statistics and never recalibrated for LivePlan trajectories; absent a sensitivity sweep, the reported consistent gains may be an artifact of this single hyperparameter.","rationale":"The paper's central claim is that a deterministic monitor with rule-based signals and a fixed threshold produces consistent gains. The monitor's trigger condition is the key mechanism, and θ_p is the numeric hyperparameter governing Long Stagnation, one of the most frequently fired signals. The paper reports no variation over θ_p, no per-model recalibration, and only averaged vanilla statistics that hide distribution overlap. I also considered whether the stronger-advisor pairing or missing uncertainty quantification is the dominant issue, but the advisor pairing is symmetric across baselines and RQ5 mitigates ordinary run-to-run nondeterminism. The threshold question directly attacks whether the headline gain is attributable to the proposed mechanism rather than to a favorable calibration point. The clean check is a small threshold sweep: if it shows stability, the CONDITIONAL verdict can later be upgraded; if not, the claim needs qualification. Since the reader already identified this threshold-calibration issue as the weakest assumption, my read does not change the verdict.","tokens_in":19525,"tokens_out":5606,"duration_ms":52315,"concrete_test":"On the deterministic SWE-bench Pro subset from §III-E (DeepSeek-V3, 222 instances), run LivePlan with θ_p ∈ {5, 7, 9} and all other hyperparameters fixed, then compare success rate, R→U/U→R transitions relative to Vanilla, and advisor trigger frequency. If the success-rate gain over Vanilla varies by more than about 3 percentage points across the sweep, or if the method ranking changes, the central claim is threshold-dependent and must be qualified; if gains remain approximately flat, this concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The weakest point is the calibration of the monitor's Long Stagnation detector in the hyperparameters paragraph of §III. θ_p is set to seven consecutive same-phase steps from vanilla SWE-agent statistics: average maximum consecutive phase lengths of 5.64 for resolved instances and 7.61 for unresolved instances, across all models. The paper states that this threshold separates normal progress from likely stagnation and is not tuned to any intervention method, but it is never re-estimated on LivePlan trajectories, and no sensitivity analysis is reported. This matters because §II-A defines Long Stagnation as a same-phase sequence exceeding θ_p, and Table VI shows this signal fires in a large fraction of runs; for example, Long Stagnation Navigation is observed in 26.1–48.7% of LivePlan trajectories depending on the setting. If advisor-extended trajectories change phase-length distributions, θ_p = 7 could over-trigger and impose unnecessary advice, or under-trigger and miss genuine drift. Either way, the headline gain of +9.9% average would not be a stable property of the method. The RQ5 deterministic-subset analysis in §III-E controls for vanilla run-to-run variation, not for threshold dependency, so it does not resolve this concern. The missing experiment is a threshold sweep that keeps all other components fixed.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces LivePlan, an online monitoring and corrective-steering layer for programming agents. LivePlan separates a deterministic rule-based monitor from an LLM-based advisor: the monitor analyzes two trajectory representations, GRAPHECTORY and LANGUTORY, for behavioral drifts such as plan violations, oscillation, stagnation, and repeated actions, and only when a drift is detected does the advisor issue predefined or custom next-step advice. The system is implemented on top of SWE-agent and evaluated on SWE-bench Verified and SWE-bench Pro with three executor LLMs and two advisor LLMs. Compared with vanilla SWE-agent, LivePlan reports consistent resolution-rate gains of up to 15.2 percentage points (average 9.9), fewer regressions, and small additional cost, supported by a deterministic-subset analysis and by process-centric trajectory and manual root-cause analyses.","tokens_in":19743,"tokens_out":11307,"duration_ms":87432,"significance":"If the reported results hold, LivePlan is a valuable, pluggable steering mechanism for long-horizon agentic software engineering. The core design idea—decoupling judging from advising and replacing LLM judging with deterministic rules—directly addresses documented failure modes of prior work, including hallucinated or sycophantic LLM judgments and costly periodic interventions. The paper's strengths include a large empirical study (7,752 trajectories), publicly available artifacts, a deterministic-subset robustness check that preserves the method ranking, plan-compliance and trajectory-similarity analyses, and a careful manual investigation of improvement and regression root causes. The main reservation is that the headline success-rate claims rest on single-run percentages and on monitor thresholds calibrated from vanilla statistics without a sensitivity analysis, so the empirical evidence is suggestive rather than conclusive.","major_comments":[{"comment":"The central claim of consistent gains (up to 15.2%, average 9.9%) is supported only by single-run success rates, with no confidence intervals, significance tests, or repeated paired rollouts for LivePlan outcomes. The RQ5 deterministic-subset analysis (§III-E) controls for vanilla run-to-run variation, but it does not estimate the variance of LivePlan's own outcomes, and the paper's own §III-D guidance states that 'the evaluation should use repeated paired rollouts to estimate how an intervention changes the probability of resolution, rather than drawing conclusions from a single trajectory.' Because the headline is a reliability claim, please report repeated LivePlan runs for at least one benchmark/executor pair, or provide a paired statistical comparison (e.g., McNemar's test) for Tables II–III.","section":"§III-A, Tables II–III"},{"comment":"The long-stagnation threshold θ_p = 7 is calibrated from vanilla SWE-agent phase-length statistics (mean maximum consecutive phase lengths of 5.64 for resolved and 7.61 for unresolved instances) and is never re-estimated on LivePlan trajectories, and no sensitivity sweep is reported for θ_p, θ_c, or θ_i. This is load-bearing because Table VI shows that Long Stagnation fires in 26.1–48.7% of LivePlan trajectories depending on the setting; changing θ_p changes how often advice is injected and could therefore explain a substantial part of the reported gains. The RQ5 deterministic-subset analysis controls for baseline volatility, not threshold dependency. Please add a sweep over θ_p (and ideally θ_c) with all other components fixed, for at least one executor/benchmark pair, and report the resulting success rates.","section":"§III, hyperparameters paragraph"},{"comment":"The abstract's cost claim ('only an additional cost of $0.08 per instance') is a mean over settings that hides substantial variation: using the Table II cost columns, the additional total cost ranges from about +$0.26 for Gemini-2.5-Flash on SWE-bench Pro to −$0.05 for MiniMax-M2.5 on SWE-bench Verified. Please report the per-pair cost distribution with confidence intervals and state explicitly that the $0.08 figure is an average across the six model-benchmark pairs. The cost accounting in Table V should also be clarified, since the DeepSeek-V3 Pro row (executor 0.12 plus advisor 0.04) does not obviously reconcile with the 0.15 average in Table II.","section":"§I and §III-B, Table V"}],"minor_comments":[{"comment":"The text contains several missing spaces around the method name (e.g., 'LIVEPLANto', 'LIVEPLANand'); please proofread the rendering of LivePlan throughout.","section":"Abstract and §I"},{"comment":"The stacked bar chart in Figure 8 is difficult to read because the per-pair labels and legend are compressed; consider a faceted plot or a table with larger fonts.","section":"Figure 8"},{"comment":"The column headings 'Cost ($) Exec. Adv. Mon.' are ambiguous: it is unclear whether 'Cost' is total cost or executor-only cost, and the DeepSeek-V3 Pro row should be reconciled with the average cost in Table II.","section":"Table V"},{"comment":"The manual root-cause analysis reports independent coding followed by consensus, but no inter-annotator agreement statistic is reported; Cohen's kappa or a similar measure would strengthen the qualitative claims.","section":"§III-D"},{"comment":"The comparison against SAGE, Periodic Advisor, and Predefined Advice is reported only for SWE-bench Pro; the abstract's claim that LivePlan 'consistently outperforms alternative approaches' should be qualified to the benchmark on which those baselines were run.","section":"§III-A"}],"recommendation":"major_revision","confidential_remarks":"This is a well-executed empirical systems paper with a sensible design and a robustness check in RQ5; the main gap is statistical. I would support acceptance after the authors add repeated paired rollouts (or a paired significance test) and a threshold-sensitivity analysis, and after the cost claims are reported with their spread. No concerns about authorship or scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"LivePlan is a believable, well-evaluated intervention: a deterministic trajectory monitor paired with an on-demand LLM advisor, which cleanly decouples judging from advising. The empirical package is unusually complete—three executors, two advisors, two benchmarks, 7,752 trajectories, comparisons against SAGE and two ablations, a deterministic-subset rerun, transition matrices, and manual root-cause labeling. On the evidence here, the headline claim (up to +15.2, average +9.9 points, roughly $0.08/instance) is credible, and the paper should not be desk-rejected. What is genuinely new is the decoupling itself: the monitor is rule-based over GRAPHECTORY/LANGUTORY, so it cannot hallucinate a problem, and the LLM advisor is consulted only when the monitor fires. The paper also positions itself clearly against SAGE's re-planning and periodic advisors like SWE-PRM and Wink. Reusing the authors' prior representations is not a flaw, but the novelty delta over that prior work should be stated more explicitly.\n\nSoft spots, in order of seriousness. (1) The stagnation threshold θ_p is the real weak point. It is set to 7 using vanilla phase-length statistics (averages of 5.64 resolved, 7.61 unresolved) and never tested for sensitivity on LivePlan trajectories. The stress-test concern is legitimate: if advisor-extended trajectories lengthen same-phase stretches, the monitor could over- or under-trigger. I do not think it sinks the paper—7 sits between the two vanilla averages, and RQ5 preserves the ranking—but a θ_p sweep is a required addition, and the paper should report how often Long Stagnation fires as a function of θ_p. (2) The main success-rate tables report single-run percentages without confidence intervals or significance tests. The deterministic-subset analysis partially mitigates, but only for one dimension of uncertainty; paired repeated rollouts would be stronger, and the paper's own guidelines in §III-D make that same point. (3) Always pairing the executor with a stronger advisor is a reasonable design choice, but it is a stated prerequisite rather than a tested variable; some readers will wonder about the advisor-equals-executor setting. (4) The artifact link lacks a commit hash and a one-command reproduction path. That is minor and easy to fix.\n\nThe citation pattern looks honest. I did not notice missing related work that would change the picture. The R→U analysis is one of the most useful parts—the distinction between wrong advice and executor noncompliance is often skipped, and the authors treat it carefully. Who benefits: anyone building or evaluating steering mechanisms for coding agents, and researchers running head-to-head comparisons on SWE-bench Pro. This paper deserves a serious referee; with the threshold sweep and uncertainty quantification added, I would be comfortable seeing it accepted.","headline":"A credible, unusually thorough evaluation of a genuinely new judge-advisor decoupling; the θ_p threshold needs a sensitivity sweep and the success tables need confidence intervals before the headline claim is taken at face value.","tokens_in":20331,"tokens_out":2251,"would_cite":true,"duration_ms":20723,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LivePlan, a deterministic monitor paired with a selectively consulted advisor LLM, lifts programming-agent issue-resolution rates by up to 15.2 percentage points at an added cost of about $0.08 per task.","keywords":["LivePlan","online monitoring","corrective steering","behavioral drift","programming agents","process-centric trajectory analysis","rule-based monitor","LLM advisor"],"falsifier":"Run LivePlan on a fixed model-benchmark pair with the stagnation threshold set to, say, 3, 5, 7, 10, and 14: if the reported gains collapse or reverse outside a narrow window around 7, then the threshold, not the monitoring principle, is carrying the result. A complementary check is to have independent annotators label trajectory segments as harmful drift or legitimate work and measure how often the deterministic monitor agrees with those labels.","tokens_in":19307,"feed_emoji":"🛠️","tokens_out":5194,"duration_ms":44370,"temperature":0.7,"pith_summary":"The paper tackles a practical failure mode of long-horizon coding agents: they drift off their plan, repeat useless actions, or terminate without a patch. Its claim is that this drift can be caught cheaply and reliably by making judgment deterministic and rule-based, then calling an LLM advisor only after a problem has already been flagged. On SWE-bench Verified and SWE-bench Pro, LivePlan raises SWE-agent's resolved-instance rate by up to 15.2 percentage points (average 9.9%) for an extra cost of about $0.08 per instance, with gains concentrated on medium and hard issues and few regressions on runs that the vanilla agent already solved. The design matters because it separates judging from advising, avoiding both hallucinated interventions of replanning systems and the cost and false alarms of periodic LLM judges.","feed_headline":"Rule-based monitor steers coding agents, lifting fixes by 15.2%","feed_subtitle":"LivePlan checks deterministic drift signals before calling an LLM advisor, adding about 8 cents per task.","key_machinery":"The load-bearing objects are GraphEctory and Langutory, two process-centric trajectory representations adapted from prior work and extended here: GraphEctory turns a trajectory into a directed graph whose nodes are distinct actions plus separate thought nodes, and Langutory abstracts the execution into a phase alphabet covering navigation, reproduction, patching, validation, and general work. Deterministic rules over these representations fire on ten signals: repeated actions appear as back-edges, thought or action oscillation as self-loops or multi-node cycles, long stagnation as more than $\\theta_p = 7$ consecutive same-phase steps, and plan violation as missing phases. The monitor's verdict determines whether to block the offending action or let it proceed with appended advice, and a cooling period of $\\theta_c = 5$ steps limits how often the advisor LLM is consulted. This mechanism carries the entire contribution, since it removes the LLM from the judging path and keeps advice targeted and timely.","core_discovery":"The paper's central claim is that prior steering techniques conflate two jobs that should be separated: judging whether a trajectory is in trouble, and advising how to fix it. An LLM asked to do both is biased to find problems, so it can derail healthy runs, and periodic checks either cost too much or arrive too late. LivePlan detects behavioral drift with deterministic rules over two trajectory representations, and only when a rule fires does it invoke an advisor LLM to produce a single high-level next-step correction. The paper argues this division of labor is what produces consistent resolution-rate gains of up to 15.2% (average 9.9%) across three executor models and two benchmarks, at modest cost, while preserving most vanilla successes.","pith_inferences":["The paper does not test whether the fixed stagnation threshold $\\theta_p=7$, derived from vanilla statistics, remains optimal when advisor-extended trajectories change phase-length distributions; an easy extension would be to re-calibrate the threshold per executor model or adapt it online.","The judging/advising separation is not repository-specific: the signals are action-level and phase-level, so the same monitor could plausibly steer long-horizon agents in research, data analysis, or robotics, provided a phase taxonomy exists.","The manual analysis attributes most LivePlan regressions to executors failing to follow correct advice, not to wrong advice; this points to intervention-conditioned post-training as the next lever, a direction the paper itself names.","A cheap but decisive follow-up would be to vary $\\theta_p$ across a grid and check whether gains persist; if they collapse outside a narrow window, the threshold rather than the monitoring principle is doing the work."],"forward_implications":["If deterministic monitoring plus on-demand advice is the right division of labor, other long-horizon agent tasks can adopt the pattern without retraining the executor.","The reported gains concentrate on medium and hard instances, so the method is most useful exactly where coding agents struggle.","LivePlan preserves most vanilla successes: regressions are low, so enabling the intervention carries little downside on runs that already work.","Monitoring costs only a few milliseconds and advisor calls cost $0.01 to $0.06 per instance, making the technique cheap enough for routine use.","LivePlan trajectories show higher plan compliance, especially in executing required phases in order, while still allowing productive out-of-plan steps such as extra regression testing."],"supporting_citations":[{"why":"Supplies the SWE-agent executor that LivePlan wraps and the vanilla baseline it is compared against.","marker":"[37]"},{"why":"Provides the SWE-bench Verified dataset, one of the two evaluation benchmarks.","marker":"[12]"},{"why":"Provides the SWE-bench Pro dataset, the harder and contamination-resistant evaluation benchmark.","marker":"[7]"},{"why":"Defines SAGE, the replanning baseline that LivePlan consistently outperforms at lower cost.","marker":"[11]"},{"why":"Defines the periodic-advisor strategy that LivePlan's Periodic Advisor ablation approximates.","marker":"[10]"},{"why":"Supplies the original GraphEctory and Langutory trajectory representations that LivePlan extends with thought nodes and finer phases.","marker":"[17]"},{"why":"Provides the plan-compliance metrics used to show that LivePlan trajectories follow the intended repair workflow more closely.","marker":"[18]"}],"fun_headline_variants":["Split judge and advisor to boost coding agent fixes by 15%","Rule-based drift detection before LLM advice lifts fixes by 15%","Separating judge and advisor in agent steering yields 15% more fixes","Don't let an LLM judge and advise; split roles for 15% gain","Monitor rules first, then call LLM advisor: fix rate up 15%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The fixed threshold of seven consecutive steps in one phase, together with the hand-defined deterministic signal set, correctly separates harmful drift from legitimate long-horizon work on every executor and benchmark tested, even though the threshold was calibrated from vanilla trajectories and never re-calibrated for LivePlan trajectories.","fun_headline_variants_meta":{"raw":{"variants":["Split judge and advisor to boost coding agent fixes by 15%","Rule-based drift detection before LLM advice lifts fixes by 15%","Separating judge and advisor in agent steering yields 15% more fixes","Don't let an LLM judge and advise; split roles for 15% gain","Monitor rules first, then call LLM advisor: fix rate up 15%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000574,"raw_usage":{"total_tokens":2716,"prompt_tokens":953,"completion_tokens":1763,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":569,"completion_tokens_details":{"reasoning_tokens":1677}},"tokens_in":569,"tokens_out":1763,"duration_ms":11020,"temperature":1.0,"reasoning_tokens":1677,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:30:15.812266+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run LivePlan on a fixed model-benchmark pair with the stagnation threshold set to, say, 3, 5, 7, 10, and 14: if the reported gains collapse or reverse outside a narrow window around 7, then the threshold, not the monitoring principle, is carrying the result. A complementary check is to have independent annotators label trajectory segments as harmful drift or legitimate work and measure how often the deterministic monitor agrees with those labels.","supporting_citations":[{"cited_title":"Swe-agent: agent- computer interfaces enable automated software engineer- ing,","cited_arxiv_id":null,"evidence_quote":"Supplies the SWE-agent executor that LivePlan wraps and the vanilla baseline it is compared against."},{"cited_title":"SWE-bench: Can language models resolve real-world github issues?","cited_arxiv_id":null,"evidence_quote":"Provides the SWE-bench Verified dataset, one of the two evaluation benchmarks."},{"cited_title":"When agents go astray: Course-correcting SWE agents with PRMs,","cited_arxiv_id":null,"evidence_quote":"Defines the periodic-advisor strategy that LivePlan's Periodic Advisor ablation approximates."}],"review_version":1}