{"id":"d664d1c4-431a-40d1-9690-e81a1c96bbbd","arxiv_id":"2607.17641","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"In noisy verify-repair loops, repair only while the estimated expected gain (1-b)α - bβ stays positive; stop when the belief crosses b* = α/(α+β), and fall back to keep-best when calibration is unreliable.","lead":"LLM agents that check and repair their plans can accidentally damage correct plans when both the checker and the fixer make mistakes. This paper introduces a stopping rule based on the estimated marginal repair gain, plus a conservative fallback, and shows it prevents catastrophic quality collapse on a stressed math benchmark.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"VRR-Stop's posterior is computed under a binomial/stationality model that its own data show to be violated, and the deployed rule has no online check for this violation; the headline results may be sensitive to this misspecification.","rationale":"The reader identified the same weakest assumption: conditional independence of verifier votes and local stationarity of the noise/repair parameters. I agree that this is the most load-bearing concern because the entire stopping decision reduces to the sign of G_k, which is computed from a posterior that is only valid under those assumptions. The paper is unusually honest: it states the assumptions as local approximations, provides beta-binomial diagnostics and shift experiments, and reports the interior-peak failure in H.2. However, the headline empirical claims (e.g., 0.722 vs. 0.666 under a different observation model) are sensitive to the exact form of the model, and the deployed rule has no online safeguard for assumption violations—the separation test for VRR-Guard is offline, and the B_k coverage of Proposition 1 is not proven for the deployed rule. These are addressable, non-fatal weaknesses that justify a CONDITIONAL verdict, which the reader already issued. My proposed check directly quantifies the sensitivity and would settle whether the concern lands. I credit the paper for releasing trajectories, reporting paired tests, and explicitly marking its limitations; the concern is about the robustness of the central claim under its own acknowledged violations, not about any internal inconsistency or dishonesty.","tokens_in":27728,"tokens_out":10993,"duration_ms":106410,"concrete_test":"On the released frozen trajectories for the GSM8K PM-stress (N=500) and Llama-stress settings, recompute the VRR-Stop posterior b_k under (i) a beta-binomial observation model with per-class overdispersion fit from the calibration probe, and (ii) round-dependent transition probabilities α_k, β_k estimated from the same trajectories. Re-run the stopping rule with these alternative posteriors while keeping all other components fixed. If the final true validity or mean repair rounds change by more than 3 percentage points or 0.2 rounds relative to the reported binomial/stationary results, or if the stop/repair decision reverses on more than 5% of instances, the reported results are not robust to the model's stated assumptions and the central claim must be qualified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central mechanism of VRR-Stop is the posterior b_k from the belief recursion (Eq. 5) and the predict step (Eq. 6), which require (i) the M verifier votes to be conditionally independent given y_k and (ii) ρ0, ρ1, α, β to be locally constant. The paper's own evidence shows both assumptions are violated on its own evaluation data. Figure 9(a) shows per-plan acceptance rates spanning roughly 0.25–1.0 within the valid class and 0.0–0.75 within the invalid class, i.e., pronounced within-class heterogeneity, which makes the binomial likelihood (fixed ρ0, ρ1) misspecified. Appendix H.2 reports the repair success rate α dropping from 0.415 in round 1 to 0.032 in round 5 in the favorable setting, directly contradicting the stationarity of α across rounds. The deployed rule ignores these violations: it uses plain binomial EM for verification noise and round-1 transition estimates for α, β. Appendix I shows the observation-model choice is consequential—replacing binomial EM with beta-binomial EM changes the headline GSM8K stress validity from 0.722 to 0.666, a 5.6 pp shift. Moreover, the experiments act directly on the sign of ˆG_k rather than via the certified interval of Proposition 1/Eq. (8), whose error radius B_k is asserted to cover but whose coverage is not proven for the deployed rule. Thus a misspecified posterior can flip the stop/repair decision exactly in the low-J, low-Δ regimes the method is designed to safeguard. The paper labels these as local approximations and provides valuable diagnostics, but it gives no online criterion to detect when the approximation is failing before the commit decision.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies stopping rules for LLM verify-repair loops when both the verifier and the repairer are noisy. It introduces a four-parameter model (verifier false-acceptance ρ0, false-rejection ρ1, repair success α, repair damage β), a belief-filtering recursion that estimates the probability that the current plan is truly valid, and a myopic stopping rule based on the sign of the one-step marginal gain G_k = (1−b_k)α − b_kβ, with boundary b* = α/(α+β). When calibrated stopping is unreliable, the paper proposes VRR-Guard, a keep-best fallback that replaces the incumbent candidate only if the new plan receives at least δ more verifier votes. Experiments on GSM8K, MATH-500, MBPP, and BFCL claim large gains over fixed-budget repair, with a headline 60.6-point improvement on a GSM8K prompt-mismatch stress setting, and show that VRR-Guard restores validity when calibrated stopping collapses. The paper is transparent about local-stationarity assumptions and the limitation of binary validity, and it releases replay artifacts.","tokens_in":28128,"tokens_out":2711,"duration_ms":30251,"significance":"If the framework holds up, it addresses a genuinely important problem: verify-repair loops are ubiquitous in LLM agents, and existing stopping heuristics can commit corrupted plans. The paper's strengths include a clean myopic-gain decomposition, a recognizably correct derivation of the belief recursion and stopping boundary, five-fold cross-fitting for calibration, paired McNemar/bootstrap tests for headline claims, and an unusually candid discussion of failure modes and the boundaries of the method. The empirical map of decision-flip risk as a function of verifier discrimination J and decision margin Δ is a useful conceptual contribution, and the guarded keep-best fallback is a simple, plausible safety mechanism. However, as detailed below, the central guarantee of the method is conditional on assumptions that the paper's own data show to be violated in the deployed regime, and the deployed rule does not use the certified interval of Proposition 1.","major_comments":[{"comment":"The identifiability-collapse analysis on Llama is presented as the mechanism behind VRR-Stop's failure, but the evidence is partly circular. The claim that calibrated stopping recovers only for ρ̂1 ≳ 0.30–0.35 is obtained from a parameter sweep under the deployed rule, which itself assumes the binomial model. Since the true failure of the deployed rule may be due to model misspecification rather than solely to EM estimation error, the proposed separation between 'estimation error flips the sign' and 'model misspecification flips the sign' is not established. A more direct test would be to evaluate the stopping rule with true parameters on the actual Llama trajectories, which would isolate the misspecification component.","section":"§5.4, Table 8"}],"minor_comments":[{"comment":"Algorithm 1 switches mode to Guard when Eq. (8) fails, but the main text says experiments use VRR-Stop acting directly on the sign of Ĝ_k without the interval. Please clarify whether Algorithm 1 describes the intended deployment procedure or the experimental protocol, and reconcile the discrepancy.","section":"Algorithm 1 vs. §4.2"},{"comment":"Lemma 1's bound is stated for a single round and then multiplied by K_max using a union bound. It would be helpful to note that this union bound is conservative, especially when erroneous replacements are positively correlated across rounds, as they may be under within-class difficulty heterogeneity.","section":"§4.3, Lemma 1"},{"comment":"The colorbar label 'Pr[sign flip]' is cut off in the figure. Please ensure the axis label is legible in the final version.","section":"§5.4, Fig. 5"},{"comment":"The discussion of α+β ≤ 1 is useful, but it is stated as a 'strict condition' without explaining why it should hold for a well-behaved repair operator. Consider adding a brief derivation or reference.","section":"Appendix H.6"}],"recommendation":"major_revision","confidential_remarks":"The paper is substantive and unusually honest about its limitations, but the central robustness claim is currently supported more by empirical breadth than by the formal machinery. The main gap is that the deployed rule bypasses the certified interval of Proposition 1 and relies on a binomial/stationarity model that the paper's own diagnostics show to be violated. This is fixable within the manuscript's scope by adding online misspecification checks, using the interval-based decision, or sharply redefining the claim as 'sign criterion under local approximations' rather than 'robust stopping.' I would encourage the editor to send this to a statistical learning referee in addition to an LLM-agent referee, because the key issues are about posterior calibration and coverage rather than about the benchmark results."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, this one is worth your time if you care about agent reliability. The core idea is clean: separate verifier noise (rho0, rho1) from repairer behavior (alpha, beta), filter the belief, and stop based on the sign of expected marginal gain. That separation is genuinely new relative to prior self-correction models with a single rate, and the empirical section documents an important failure mode: verification acceptance can rise while true validity collapses, with raw-label statistics and an instructive trace.\n\nThe evaluation is stronger than most in this subfield: five-fold cross-fitting means decisions for an instance don't use its own labels, and the paper reports paired tests, confidence intervals, and honest boundary cases. VRR-Stop beating fixed five-round repair by 60.6 pp on the stress setting is credible.\n\nThe soft spots are proportionate. The belief recursion assumes conditional independence of verifier votes and local stationarity of the four parameters. The paper's own data show both are violated: alpha drops from ~0.42 in round 1 to ~0.03 in round 5 in the favorable setting, and per-plan acceptance rates vary widely within class. The paper labels these as local approximations, which is honest, but the deployed rule has no online check for when the approximation is failing. The experiments act on the sign of the estimated gain rather than the certified interval of Proposition 1, whose error radius coverage is not proven for the deployed rule. The appendix shows the observation-model choice is consequential: binomial vs beta-binomial EM shifts the headline GSM8K stress validity by 5.6 pp. And VRR-Guard's margin delta is selected by replay on all seven evaluation settings—a mild tuning step. The paper also correctly notes in H.6 that the strict alpha+beta<=1 condition fails on its own GSM8K stress setting, so the global monotonicity claim in the main text is overstated.\n\nThese are fixable issues, not fatal ones. The framework's value—stopping before repair damages correct plans—is supported by the evidence. The paper would benefit from an online stationarity check and a clearer separation of proven guarantees from heuristics.\n\nBottom line: a solid, careful paper on a practical problem. It deserves a serious referee. I'd bring it to a reading group and would cite it in work on agent stopping or self-correction.","headline":"A serious, well-evidenced paper on when to stop verify-repair loops, with a sound core framework and clearly flagged approximation assumptions that can flip the decision in low-discrimination regimes.","tokens_in":28648,"tokens_out":2707,"would_cite":true,"duration_ms":24031,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A noisy verify-repair loop should stop or continue according to the sign of the one-step expected gain, and a rule built on this lifted final true validity by 60.6 percentage points.","keywords":["verify-repair loops","LLM agents","stopping rule","belief filtering","marginal gain","verifier noise","verifier discrimination","keep-best fallback"],"falsifier":"Run VRR-Stop on a trajectory set where verifier votes are made dependent (e.g., a single judge that sees its own previous answers, or self-consistency voting with shared drafting) and compare its accept/repair decisions against ground-truth validity labels: if the sign-flip rate systematically exceeds the rate predicted by the conditional-independence model, the core premise is falsified. A complementary check: label per-round repair outcomes and test whether the sign of (1-b_t)α_t - b_tβ_t computed from round-lagged estimates disagrees with the round-1-parameter decision on trajectories where","tokens_in":27597,"feed_emoji":"🛑","tokens_out":10923,"duration_ms":99820,"temperature":0.7,"pith_summary":"The paper is trying to establish that deciding when to stop repairing a plan in an LLM agent loop is an identifiable sign decision, not a heuristic. It models verifier false acceptance, verifier false rejection, repair success, and repair damage with four parameters, filters repeated verification votes into a belief of true validity, and continues repair only while the expected one-step gain G = (1-b)α - bβ is positive, stopping at boundary b* = α/(α+β). A sympathetic reader would care because fixed repair budgets and pass-rate thresholds can silently destroy correct plans — the paper shows a stress setting where reported acceptance rises while true validity collapses, and the new rule lifts final true validity by 60.6 percentage points over fixed five-round repair. The paper is explicit about its boundaries: the model assumes local stationarity and conditional independence (Section 3.2), does not locate interior peaks under abrupt repair changes (Section H.2), and uses a binary validity state (Section H.6). When verifier discrimination approaches zero, calibrated stopping can fail, and a margin-based keep-best fallback restores near-no-repair safety.","feed_headline":"Repair only while it helps: stopping rule gains 60.6 points","feed_subtitle":"A sign-based stopping rule keeps language-model agents from repairing correct plans, at 0.72 repair rounds on average.","key_machinery":"The four-parameter noise model: ρ0 = false acceptance, ρ1 = false rejection, α = repair success rate, β = damage rate, with verifier discrimination J = 1 - ρ0 - ρ1. Carrying the argument is the belief recursion: after M conditionally independent votes, the posterior b_k updates by Bayes' rule, and the prediction step is b^-_{k+1} = (1-β)b_k + α(1-b_k). These combine into the marginal gain G_k = b^-_{k+1} - b_k = (1-b_k)α - b_kβ, whose zero crossing defines the stopping boundary b* = α/(α+β). The sign of G_k is the decision rule. VRR-Guard is the secondary mechanism: a keep-best rule that replaces the incumbent only when the new plan's vote count exceeds the champion's by margin δ, with a con","core_discovery":"On the paper's own terms, the discovery is that the optimal myopic stop/continue decision in a noisy verify-repair loop is determined solely by the sign of G_k = (1 - b_k)α - b_kβ, where b_k is the posterior probability that the current plan is valid. The threshold b* = α/(α+β) is the fixed point of the valid-validity dynamics and does not depend on verifier noise; verifier noise enters only through how reliably the belief can be located. The paper proves a stopping-sign consistency bound (Proposition 1), gives an error-radius criterion for when the sign is identifiable, and shows empirically that reliability is governed by the verifier discrimination J and the decision margin Δ, not by the","pith_inferences":["Editorial inference: because the stopping boundary b* varies from 0.954 in the favorable setting to 0.289 in the stress setting, any universal confidence threshold is unsafe; this implies that per-task or even per-instance estimates of α and β, keyed by difficulty, would make the boundary adaptive.","Editorial inference: VRR-Guard's concentration-bound argument suggests a model-agnostic 'keep the champion, replace only on a vote margin' protocol that could be layered onto any iterative refinement loop where calibrated parameters are unavailable; testing it as a standalone wrapper is a direct, low-cost extension.","Editorial inference: given the paper's Section H.2 admission that it cannot detect abrupt repair-mechanism changes, a natural next step is online monitoring of vote-stream residuals or per-round transition rates to switch from calibrated stopping to the guarded fallback mid-trajectory.","Editorial inference: the binary-validity state (Section H.6) is the main structural limit; extending the belief to a structured state of partial correctness would replace the scalar boundary b* with a set of state-dependent thresholds, and long-horizon tool-use plans are a plausible testbed."],"forward_implications":["If the sign-criterion is correct, fixed repair budgets are an unsafe default: in six of the eight evaluated settings unconditional repair drives true validity down monotonically, with measured damage probabilities up to 0.94.","Pass-rate thresholds and majority voting can actively mislead, since reported acceptance can rise while true validity falls; stopping rules should be built on the estimated belief of true validity, not raw acceptance.","Stopping can be calibrated from weak supervision: repeated verification votes plus at most 300 labeled repair transitions suffice to locate the sign of the marginal gain when verifier discrimination J and the decision margin are not too small.","Near-zero verifier discrimination is a distinct failure mode where more calibration data can make estimates worse; the margin-based keep-best fallback recovers near-no-repair validity instead of collapsing.","The cost profile makes the rule practical: one-off calibration, M verifier calls per round, and an average of 0.72 repair rounds on the stress setting — better validity than no-repair and fixed repair at a fraction of the compute."],"fun_headline_variants":["Stopping rule boosts LLM plan validity by 60.6 points","Know when to stop: 60.6-point gain with 0.72 repairs","Sign-based stop rule doubles validity in noisy loops","VRR-Stop: halt repairs only when they truly help","One rule to stop repair: sign of marginal gain"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that repeated verifier votes are conditionally independent given true validity and that the four noise and repair parameters are stable within a local decision window — the paper itself calls this a local approximation (Sections 3.2, 4.1, and H.2) — so if votes are correlated or repair dynamics drift mid-loop, the estimated sign of the marginal gain can flip and the stopping decision is no longer reliable.","fun_headline_variants_meta":{"raw":{"variants":["Stopping rule boosts LLM plan validity by 60.6 points","Know when to stop: 60.6-point gain with 0.72 repairs","Sign-based stop rule doubles validity in noisy loops","VRR-Stop: halt repairs only when they truly help","One rule to stop repair: sign of marginal gain"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00017,"raw_usage":{"total_tokens":1140,"prompt_tokens":813,"completion_tokens":327,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":557,"completion_tokens_details":{"reasoning_tokens":239}},"tokens_in":557,"tokens_out":327,"duration_ms":3541,"temperature":1.0,"reasoning_tokens":239,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T17:24:24.051894+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run VRR-Stop on a trajectory set where verifier votes are made dependent (e.g., a single judge that sees its own previous answers, or self-consistency voting with shared drafting) and compare its accept/repair decisions against ground-truth validity labels: if the sign-flip rate systematically exceeds the rate predicted by the conditional-independence model, the core premise is falsified. A complementary check: label per-round repair outcomes and test whether the sign of (1-b_t)α_t - b_tβ_t computed from round-lagged estimates disagrees with the round-1-parameter decision on trajectories where","supporting_citations":[],"review_version":1}