{"id":"bb7cef65-77a4-4d5f-b3e9-8ed41783749a","arxiv_id":"2601.21522","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"ReD, which resets after each attempt and discards solved problems, provably increases coverage@cost (unique solved problems per fixed budget) over solve-to-completion for LLMs with power-law pass@k, and can be used to estimate the power-law exponent.","lead":"This paper proposes Reset-and-Discard (ReD), a rule for spending a fixed LLM inference budget across many verifiable problems: try each problem a few times, discard solved ones, and rotate to unsolved ones. The authors prove ReD improves the expected number of distinct solved problems compared with the usual 'solve one problem before moving on' policy, and test it on coding benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Finite-pool counterexample falsifies the unqualified 'any budget' claim: with N=2, p=(0.6,0.4), τ=1, budget 4, solve-to-completion beats ReD (1.585 vs 1.48).","rationale":"I read the paper in good faith and find the core renewal-theoretic idea interesting and largely sound for the infinite-pool model. The empirical demonstrations on HumanEval are consistent with ReD improving average coverage on those models and budgets. However, the paper's most prominent claim—that ReD improves coverage@cost at every budget for any difficulty distribution—is the load-bearing assertion of the abstract and §1. The provided proof only establishes a mean-completion-time ordering (Thm 4.1, App. B), which is insufficient for pointwise coverage dominance. More decisively, an explicit finite-pool counterexample (N=2, p=(0.6,0.4), τ=1, t=4) shows the unqualified claim is false: solve-to-completion gives expected coverage 1.5848, while ReD gives 1.48. This is not a matter of imperfect verifiers or correlated attempts—it fails under the paper's own perfect-verifier, i.i.d. assumptions. The finite-size section (§4.3) only offers an approximation, not a theorem, so it cannot be used to support the 'any budget' phrasing. The paper should be revised to state the theoretical guarantee as asymptotic/large-pool and should either prove or remove the pointwise claim. This does not warrant rejection because the main empirical effect and the large-pool theory remain valuable, and a corrected claim is likely salvageable. I therefore keep the reader's CONDITIONAL verdict, with the reason sharpened from a proof gap to a demonstrated counterexample.","tokens_in":18538,"tokens_out":22286,"duration_ms":231079,"concrete_test":"Perform an exact enumeration (or 10^5 Monte Carlo realizations) of the N=2, τ=1, t=4 case with success probabilities p1=0.6 and p2=0.4, comparing solve-to-completion (random initial order) with ReD. If the enumeration reproduces expected coverages 1.5848 (solve-to-completion) and 1.48 (ReD), the paper's unqualified 'any budget / any difficulty distribution' claim is falsified in the finite-pool setting. Additionally, for the infinite-pool renewal model, verify directly from Eq. (10) whether Fτ(t)≥F(t) for all t; if this stochastic-dominance check passes, the paper should add that proof to support the infinite-pool pointwise claim, but it would not rescue the finite-pool statement.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim, stated in the abstract and §1 and repeated in §4.2, is that ReD 'increases coverage@cost for a given budget' and 'strictly improves coverage@cost for any budget and any underlying difficulty distribution.' The proof given actually concerns the infinite-pool renewal model: Theorem 4.1 (App. B) shows only that E[Tτ] is minimized at τ=1, which is a statement about the asymptotic slope of coverage@cost, not about pointwise dominance at every finite budget. Moreover, the unqualified claim is false for finite question pools, which is the setting of the paper's own experiments (HumanEval, N=164). Concrete counterexample: N=2, p1=0.6, p2=0.4, τ=1 (the optimal reset interval claimed in Thm 4.1), and budget t=4. Under solve-to-completion with random question order, if q1 is attempted first the expected coverage is P(T1≤4)+P(T1+T2≤4) = (1−0.4^4) + [0.6·(1−0.6^3)+0.24·(1−0.6^2)+0.096·0.4] = 0.9744+0.6624 = 1.6368; if q2 is first it is 0.8704+0.6624 = 1.5328; averaging over the two equally likely orders gives 1.5848. ReD with τ=1 gives each question exactly two attempts in four total attempts, so expected coverage is (1−0.4^2)+(1−0.6^2)=0.84+0.64=1.48. Thus solve-to-completion strictly outperforms ReD at this budget. The paper's finite-size equations (§4.3, Eqs. 13–17) are an approximation that averages over fluctuations in Rn and tn and provide no such guarantee. Therefore the headline 'any budget' theorem is not merely unproven; as stated for finite pools it is false.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Reset-and-Discard (ReD), a query-allocation policy for LLM inference under a fixed budget. ReD cycles through questions, giving each at most τ attempts, discarding solved questions, and resetting unsolved ones to the back of the queue. The authors connect pass@k to coverage@cost using renewal theory, claim that resetting strictly improves coverage@cost for any budget and any difficulty distribution, and that τ=1 is optimal. They derive asymptotic growth laws, give an approximate finite-pool prediction, and validate the method on HumanEval with three LLMs, also proposing a method to infer the pass@k power-law exponent from ReD runs. The central theoretical result is that resetting minimizes the mean number of attempts per solved question; the empirical results show ReD improves coverage for the tested models and budgets.","tokens_in":19084,"tokens_out":13208,"duration_ms":134421,"significance":"If the claims were fully established, the paper would provide a simple, practical scheduling rule for LLM inference with quantifiable savings, plus a method to estimate inference scaling exponents from cheaper ReD runs. The renewal-theoretic setup is standard but well applied, and the connection between pass@k and coverage@cost is a useful framing. The paper responsibly states its perfect-verifier assumption and acknowledges the finite-pool approximation. However, the headline universal claim—ReD improves coverage@cost for any budget and any difficulty distribution—is false as stated for finite pools, which is the setting of the paper's own experiments. The proof in App. B only establishes a mean-completion-time inequality, not pointwise dominance of coverage@cost at every finite budget. The empirical demonstration on HumanEval is suggestive but does not rescue the unqualified theoretical claim. The paper's contribution is therefore real but needs substantial revision in its claims and scope.","major_comments":[{"comment":"The abstract and §1 claim ReD 'increases coverage@cost for a given budget' and 'strictly improves coverage@cost for any budget and any underlying difficulty distribution.' This is false for finite question pools. Counterexample: N=2, p1=0.6, p2=0.4, τ=1, budget t=4. For solve-to-completion with random order, expected coverage is (1−0.4^4)+[0.6(1−0.6^3)+0.24(1−0.6^2)+0.096·0.4] = 1.6368 if q1 is first, and 1.5328 if q2 is first; averaging over orders gives 1.5848. ReD with τ=1 gives each question exactly two attempts, so expected coverage is (1−0.4^2)+(1−0.6^2)=1.48. Thus solve-to-completion beats ReD at this budget. App. B proves only E[Tτ] ≤ E[T] (mean completion time), which is an asymptotic-slope statement, not pointwise dominance of coverage@cost. The unqualified finite-budget claim must be removed or replaced by a precisely stated condition.","section":"§1, §4.2, Theorem 4.1, App. B"},{"comment":"The finite-pool prediction is an approximation that replaces the random total attempt count t(n) by its mean and neglects fluctuations in Rn and tn. The paper acknowledges this, but the counterexample above shows the discrepancy is structural, not merely a fluctuation: the finite-pool process changes the pool composition after discarding, so the renewal model with i.i.d. questions does not apply. The paper should state explicitly that the 'any budget' improvement is an infinite-pool/large-budget asymptotic result, and that the finite-pool version is only an approximation validated empirically on specific datasets, not a theorem.","section":"§4.3, Eqs. (13)–(17)"},{"comment":"The abstract (as provided) claims experiments on 'coding (HumanEval), math (GSM8K), and reasoning (MMLU-Pro)', but Section 6 describes experiments only on HumanEval. No GSM8K or MMLU-Pro results appear in the main text or appendices. This discrepancy must be resolved—either the abstract is overclaiming, or the experiments are missing from the manuscript.","section":"Abstract vs. §6"}],"minor_comments":[{"comment":"The notation n=floor(t/τ) and u=t−nτ is used without defining the range of u; specify that 0≤u<τ and that t is an integer number of attempts. Also clarify that Eq. (10) relies on the infinite-pool assumption where each reset draws an independent question.","section":"§4.1, Eq. (10)"},{"comment":"The inference method assumes P0(p)=Beta(α,β) exactly, not merely the small-p power-law form used elsewhere. This is an additional modeling assumption and should be stated more prominently; the empirical validation on one model (llama-3.1-8b-instant) is limited.","section":"§5.3, Eq. (22)"},{"comment":"The fit is truncated at the 15th round because of noise, but the choice of cutoff is not justified quantitatively. Please describe the criterion (e.g., Rn threshold) and show sensitivity of the inferred α to the cutoff.","section":"Fig. 3"},{"comment":"The reference 'Tang et al. (2024)' is incomplete; full author list and venue are needed.","section":"§2, References"},{"comment":"The paper states 'All the code used to generate the results of this paper will be posted on GitHub' but no link is provided. For reproducibility, include a URL or an anonymized repository link.","section":"§6"}],"recommendation":"major_revision","confidential_remarks":"The central theoretical result (E[Tτ] minimization) is a known restart result, and the novelty is the application to LLM coverage and the finite-pool approximation. The paper's abstract and §1 significantly overstate the strength of the result; the finite-pool counterexample is decisive against the universal 'any budget' claim. This is fixable by re-scoping the claims, but the current version should not be accepted. Also, the abstract/experiments mismatch on benchmarks needs attention. I would be open to a revised version that carefully delineates asymptotic vs. finite-pool statements and provides the missing empirical comparisons or removes the claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: Reset-and-Discard is a simple, practical allocation policy with a genuine theoretical backbone, but the flagship claim — resetting strictly improves coverage@cost at any budget for any difficulty distribution — is not what the proofs show, and it is false for finite question pools. The paper deserves a serious referee, but it needs major revision before it can be accepted.\n\nWhat is actually new: they map pass@k to coverage@cost via renewal theory, which is a legitimate and useful connection. The renewal model gives a clean explanation of why solve-to-completion has diminishing returns when the per-question success distribution has a power-law tail with alpha < 1. ReD, especially with tau = 1, makes the mean renewal time finite and converts sublinear coverage growth to linear growth asymptotically. That is a real result. The extension to inferring the power-law exponent from ReD runs is useful, and the HumanEval experiments show the advantage holds on average for three models.\n\nNow the soft spots. The stress-test counterexample is on target: with two questions, p = (0.6, 0.4), and a budget of four attempts, solve-to-completion gives expected coverage about 1.58 while ReD with tau = 1 gives 1.48. So the 'any budget' pointwise claim is simply false in the finite-pool setting, which is the setting of the paper's own experiments. Theorem 4.1 only proves E[T_tau] is minimized at tau = 1 — that is a statement about the asymptotic slope of coverage@cost, not about dominance at every finite budget. The finite-size equations (13)--(17) are approximations that average over fluctuations, and the paper acknowledges this in Sec. 4.3 but continues to use the unqualified language. This needs to be fixed, not softened in a footnote.\n\nAlso, the abstract as submitted claims experiments on GSM8K and MMLU-Pro, but the paper only reports HumanEval. That is a straightforward inconsistency. If those results exist, report them; if not, remove them from the abstract.\n\nThe review should also push for code and data release. The paper says the code will be posted on GitHub, but for an empirical paper of this kind, the reviewers should see the actual pass@k matrices and the generation protocol.\n\nOverall: the renewal mapping and the ReD policy are worth building on. The theory as stated is overbroad, and the empirical base is a single benchmark. With a corrected theorem statement, an honest finite-size discussion, and the missing benchmarks either removed or added, this could be a solid contribution to the inference-scaling literature.\n\nRecommendation: send it to peer review, but with a clear request from the editor to address the finite-pool counterexample and the abstract mismatch. It is not a desk reject, and it is not an accept as is.","headline":"Good idea, real theory, overbroad claim: ReD's 'any budget' guarantee fails on a simple finite-pool counterexample, but the renewal mapping is worth publishing after a careful revision.","tokens_in":19471,"tokens_out":4020,"would_cite":true,"duration_ms":41007,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["60K05","60K10"],"pacs":[],"model":"deepseek-v4-flash","headline":"A simple query policy—reset after each failed attempt, discard solved questions—provably increases the number of distinct tasks an LLM solves for any fixed inference budget, regardless of the shape of pass@k.","keywords":["coverage@cost","pass@k","reset and discard","inference budget allocation","renewal theory","power-law scaling","LLM inference","verifiable tasks"],"falsifier":"Run ReD (τ=1) versus solve-to-completion on a task where the model retains memory of previous attempts or uses self-correction, so attempts are correlated; if coverage@cost no longer improves, the i.i.d.-attempt assumption is the load-bearing premise. Alternatively, inject controlled verifier error rates into a code benchmark and check where the predicted attempt savings stop matching the measured curve.","tokens_in":18474,"feed_emoji":"⏱️","tokens_out":4756,"duration_ms":46422,"temperature":0.7,"pith_summary":"The paper tries to establish that a simple query policy—after a fixed number of failed attempts at a question, move on to the next question, and drop questions as soon as they are solved—strictly increases the number of distinct LLM-solved problems that a fixed inference budget buys, for any difficulty mix and regardless of the shape of pass@k. It proves that resetting every attempt is the optimal reset interval, derives the exact mapping from pass@k to coverage@cost using renewal theory, and shows that the empirically common power-law decay of 1−pass@k turns solve-to-completion coverage growth sublinear, while ReD restores linear growth. It also shows how to predict the attempt, token, and USD savings from pass@k alone and how to estimate the power-law exponent from ReD runs without expensive large-k evaluation. If correct, this gives practitioners a provable, deployable allocation rule that converts more of an inference budget into solved problems.","feed_headline":"Reset after each failed try solves more LLM tasks per budget","feed_subtitle":"ReD turns diminishing returns into linear coverage growth, and the gain is predictable from pass@k alone.","key_machinery":"A renewal-process representation of coverage@cost: solving questions is a renewal process whose inter-arrival distribution is T, the attempts needed for one question, whose CDF equals pass@k. The paper leverages the Z-transform and Tauberian theorems to extract asymptotic coverage growth, and the resetting CDF F_τ(t) = 1−(1−F(τ))^n (1−F(u)) to re-derive coverage under deterministic resetting every τ attempts. The proof of optimality of τ=1 reduces to a covariance inequality: E[T_{τ+1}]−E[T_τ] is non-negative because p(1−p)^τ/(1−(1−p)^τ) is strictly decreasing in p. For exponent inference, the surviving-difficulty distribution after n ReD rounds converges to Beta(α, β+n), giving the ratio ide","core_discovery":"The central claim is that Reset-and-Discard (ReD) improves coverage@cost—the expected number of unique problems solved after a fixed total number of attempts—for any budget and any underlying difficulty distribution, and that the optimal resetting protocol is to reset after every attempt (τ=1). Under the empirically observed power-law behavior 1−pass@k ∝ k^{-α} with 0<α<1, the standard solve-to-completion policy has divergent mean time per question and yields sublinear coverage growth; ReD makes the mean time finite and coverage grow linearly. Given pass@k, the paper provides an exact closure (Eq. 6 plus Eq. 10) predicting ReD's coverage@cost and the resulting attempt savings; without pass@k","pith_inferences":["Editorial inference: if attempts on the same question are correlated (self-correction, backtracking, memory), the independence-based renewal analysis may break; a natural extension is to test whether some reset interval other than 1 becomes optimal under such memory.","Editorial inference: ReD's difficulty-flattening dynamics double as an active-learning filter for self-improvement pipelines; it could preferentially harvest solvable trajectories, but this would need verification beyond the paper's coverage experiments.","Editorial inference: the paper's finite-dataset approximation (replacing t(n) by its average) predicts ReD's gains will hold for larger pools; this is testable on bigger benchmarks than HumanEval.","Editorial inference: the reported USD advantage conflates model price and verbosity; a practitioner could combine ReD with output-length control to amplify the savings further, but that is outside the paper's controlled setup."],"forward_implications":["If 0<α<1, standard solve-to-completion delivers sublinear coverage growth; ReD converts it to linear growth, a qualitative improvement at any finite budget.","Given an empirical pass@k curve, the paper's formulas predict exact attempt/token/USD savings from ReD before running it, enabling budget planning.","Resetting every attempt (τ=1) is strictly optimal for any difficulty distribution, so practitioners need not tune the reset interval.","ReD runs provide a statistically efficient estimator of the inference power-law exponent α, avoiding costly direct measurement of pass@k at large k.","Empirically, on coding benchmarks, ReD reaches a target coverage with fewer attempts, tokens, and dollars than solve-to-completion across all tested models."],"fun_headline_variants":["Resetting after each failure boosts LLM task coverage per budget","ReD resets failed attempts to linearize LLM coverage growth","Reset-and-Discard turns diminishing returns into linear LLM gains","Reset after each wrong answer to solve more LLM tasks per cost","Predictable savings: ReD resets LLM queries to beat diminishing returns"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The proofs assume every question has a fixed, independent success probability and a perfect verifier; if repeated attempts on a question are correlated, or if verification can be wrong, the predicted coverage gains and the optimality of resetting every attempt do not follow.","fun_headline_variants_meta":{"raw":{"variants":["Resetting after each failure boosts LLM task coverage per budget","ReD resets failed attempts to linearize LLM coverage growth","Reset-and-Discard turns diminishing returns into linear LLM gains","Reset after each wrong answer to solve more LLM tasks per cost","Predictable savings: ReD resets LLM queries to beat diminishing returns"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000164,"raw_usage":{"total_tokens":1109,"prompt_tokens":794,"completion_tokens":315,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":538,"completion_tokens_details":{"reasoning_tokens":223}},"tokens_in":538,"tokens_out":315,"duration_ms":3824,"temperature":1.0,"reasoning_tokens":223,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T06:55:33.899759+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run ReD (τ=1) versus solve-to-completion on a task where the model retains memory of previous attempts or uses self-correction, so attempts are correlated; if coverage@cost no longer improves, the i.i.d.-attempt assumption is the load-bearing premise. Alternatively, inject controlled verifier error rates into a code benchmark and check where the predicted attempt savings stop matching the measured curve.","supporting_citations":[],"review_version":1}