{"id":"1a73c1f1-d2c5-47b3-9e72-572de5279552","arxiv_id":"2510.23486","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Discounted correctness rewards in GRPO post-training shorten LLM reasoning traces by roughly 13-40% while matching pass@1 accuracy, supported by Blackwell-optimality analysis for restricted policy classes.","lead":"This paper proposes a simple training change: discount the correctness reward based on how many reasoning tokens a model uses, while leaving formatting rewards undiscounted. The result is a theoretical and empirical case that reasoning models can produce shorter chains of thought without losing accuracy.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 3.10's equality is unsupported: Lemma A.13's first-order expansion cannot resolve ties among policies with equal p and L, so a shortest-path policy need not be Blackwell optimal; the proof only supports inclusion.","rationale":"The reader's weakest_assumption identifies the same issue: the proof only supports inclusion. I agree. This is the most load-bearing theoretical flaw because Theorem 3.10 is explicitly called the main result and stated as equality. However, the inclusion direction is sufficient for the paper's core existence claim (gamma-optimal near 1 policies are Blackwell optimal and hence shortest-path), so the flaw is correctable and does not invalidate the empirical findings. The practical algorithm's additional gaps (reasoning-token discounting vs. full-path discounting, GRPO convergence to γ-optimal in Σ) are worth noting but are secondary; the equality overclaim is the clearest, most directly testable soft spot. Verdict remains CONDITIONAL: the paper should weaken the theorem and add the counterexample/qualification.","tokens_in":16688,"tokens_out":14215,"duration_ms":157526,"concrete_test":"Run the finite counterexample above: define a deterministic MDP with horizon H=4 and Π={π_A,π_B} with p=1, L=2.5, and success-time distributions as specified; compute exact Jγ for several γ∈(0.9,1) (e.g., γ=0.99,0.999) and verify π_B ∉ Π*_γ while π_B ∈ argmin_{π∈Πmaxp} L(π). If confirmed, Theorem 3.10's equality is false; re-derive the theorem with '⊆' and check the main narrative still holds.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central theoretical result Theorem 3.10 asserts Π*_bw = argmin_{π∈Πmaxp} L(π). The proof (Lemma A.13 + Theorem A.14) establishes only the '⊆' direction: any Blackwell-optimal policy maximizes p and, among p-maximizers, minimizes L. The converse is not established and is generally false. Lemma A.13 expands Jγ(π)=p(π)[1−ε(L(π)−1)] + Rπ(ε) with |Rπ(ε)|≤C_H ε². For two policies with identical p and L, the first-order terms cancel and the sign of Jγ(π)−Jγ(π′) for all γ∈(γ_bw,1) is determined by the O(ε²) remainder, which depends on the full distribution of success times, not just L. Hence a policy can be shortest-path (same p, minimal L) yet have strictly lower discounted return than another shortest-path policy for every γ sufficiently close to 1, so it is not Blackwell optimal. Example: H=4, Π={π_A,π_B}, both p=1 and L=2.5, with success-time distributions P_A(τ=2)=P_A(τ=3)=0.5 and P_B(τ=1)=0.25, P_B(τ=4)=0.75. Exact returns give Jγ(π_A)−Jγ(π_B)=0.75ε−1.75ε²+O(ε³)>0 for small ε>0, so π_B is in argmin L but not Blackwell optimal. This falsifies the equality in Theorem 3.10 (and Corollary A.15). The inclusion direction, which is what the practical 'accuracy then length' claim needs, remains intact; the theorem should be restated as Π*_bw ⊆ argmin... or 'every Blackwell optimal policy is shortest-path'.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes discounted reinforcement learning as a principled way to shorten LLM reasoning traces without harming accuracy. It models verifier-based reasoning as a finite-horizon deterministic MDP with binary terminal reward and studies Blackwell optimality in possibly restricted policy classes. The main theoretical results are: (i) for finite policy classes, gamma-discounted optimal policies stabilize to a Blackwell-optimal set near gamma=1; (ii) an explicit, though extremely small, upper bound on the Blackwell discount factor for greedy deployment policies; and (iii) a characterization (Theorem 3.10) that Blackwell-optimal policies are exactly the shortest-path policies among accuracy maximizers. The authors instantiate this in GRPO by discounting only reasoning-token environment rewards, keeping KL regularization, and matching token budgets, and report experiments on GSM8K, MATH, AMC, AIME, MINERVA, and OLYMPIAD showing accuracy preservation with shorter responses.","tokens_in":17081,"tokens_out":16635,"duration_ms":148863,"significance":"If the inclusion direction of Theorem 3.10 is correct, the paper makes a useful conceptual point: in the near-undiscounted regime, an optimal policy must first maximize success probability and then minimize conditional success length, giving a principled accuracy-then-length ordering. The finite-class stabilization argument and the distribution-dependent Blackwell-factor bound are also valuable, and the multi-seed experimental protocol with paired evaluation seeds is a strength. However, the stated equality is stronger than the proof and is false; the empirical protocol selects gamma on training accuracy, so the experiments provide only partial support. With the theorem restated as an inclusion and the empirical claims softened, the paper would be a solid contribution.","major_comments":[{"comment":"The proof of Theorem 3.10 establishes only the inclusion: every Blackwell-optimal policy is a shortest-path policy. The converse is asserted but not derived, and it is false. Lemma A.13 gives J_gamma(pi)=p(pi)[1-epsilon(L(pi)-1)]+R_pi(epsilon) with |R_pi(epsilon)| <= C_H epsilon^2; when two policies have equal p and L, the first-order terms cancel and the ordering for gamma sufficiently close to 1 is determined by the O(epsilon^2) remainder, i.e., by the full success-time distribution, not just by L. Concretely, take H=5, two start states each with probability 1/2, and two deterministic policies: pi_A succeeds at times 2 and 4 on the two states, pi_B succeeds at times 1 and 5. Both have p=1 and L=3. With epsilon=1-gamma, J_gamma(pi_A)-J_gamma(pi_B)=0.5gamma+0.5gamma^3-0.5-0.5gamma^4 = -0.5epsilon^2(3-3epsilon+epsilon^2) < 0 for epsilon>0. Thus pi_A is in argmin_{Pi_maxp} L but is not Bla","section":"Theorem 3.10 / Lemma A.13 / Theorem A.14"},{"comment":"The empirical claim that discounted GRPO 'matches Pass@1 accuracy while substantially shortening responses' is weakened by the gamma-selection protocol. The authors select, per model/dataset, the discounted configuration whose training Pass@1 matches or exceeds the undiscounted run and then choose the one with the shortest mean response length; all tabled metrics are computed for that selected configuration. This is a selection-on-success protocol: on the training split, 'accuracy preserved' is true by construction for the chosen gamma, and the reported evaluation numbers are for the selected configuration only. The paper does not verify that the chosen gamma lies in the Blackwell interval (gamma_bw,1), nor does it use the Theorem 3.9 bound. Figure 1 provides some useful evidence of the full trade-off curve, but the main tables do not. I would ask the authors to either preselect gamma on","section":"Section 5 'Variance control and reporting'; Section 4 'Practical notes (i)'"},{"comment":"There is a theory-practice gap that should be acknowledged. The theoretical results are stated for the finite deployment class Sigma of greedy deterministic policies, while the training objective in Eq. (2) is optimized over stochastic softmax policies with a moving KL reference, and GRPO updates the stochastic policy. No result in the paper shows that the greedy policy induced by the trained stochastic policy converges to a Blackwell-optimal element of Sigma, or that optimizing Eq. (2) with a given gamma is equivalent to finding a Blackwell-optimal policy in Sigma. Additionally, the theoretical MDP discounts every step (gamma^{t-1}), whereas the practical recipe in Eq. (1) discounts only a masked count K(tau) of reasoning tokens; the theorem on shortest total response length does not literally apply to the masked objective. The paper should state these as explicit assumptions or heurist","section":"Section 3.2 / Eq. (2) / Section 4"}],"minor_comments":[{"comment":"The explicit bound on the Blackwell discount factor is not connected to the practical choice of gamma; because it depends on the full augmented state space, it is not intended to be computed. Please state more explicitly that it is an existence/qualitative bound, not a practical selector.","section":"Theorem 3.9"},{"comment":"Table 2 reports averages over 5 evaluation seeds per model, with no training-seed variance. Some cells show large accuracy differences (e.g., Phi-4 AMC 51.00 to 61.00). Add confidence intervals or standard errors, or note that such differences may be within noise.","section":"Section 5 / Table 2"},{"comment":"The phrase 'Experiments confirm our theoretical results' is too strong given the selection protocol and the theory-practice gap. Consider wording such as 'are consistent with the inclusion direction' or 'provide evidence that discounting can shorten responses without sacrificing accuracy for the selected gamma.'","section":"Abstract / Introduction"},{"comment":"The reported length reductions are in total response length, while the objective discounts only reasoning tokens. Clarify whether the reported length is total tokens or reasoning tokens, and report reasoning-token length separately if available.","section":"Section 4.3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope for a ML venue. The main issue is that Theorem 3.10 overstates what the proof shows; the inclusion direction is correct and valuable, but the equality is false. The empirical protocol also selects gamma based on training accuracy, which weakens the headline claim. I think these are fixable in revision, so major_revision rather than reject is appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this one. The practical claim—discount only the correctness reward and you can cut response length by 13-40% without losing accuracy—is plausible and the experiments support it, but the theoretical headline overreaches. Theorem 3.10 states that Blackwell-optimal policies equal the shortest-path policies (among accuracy maximizers). The proof only gives the inclusion: every Blackwell-optimal policy is shortest-path. The converse is false. The first-order expansion in Lemma A.13 has an O(ε²) remainder that can decide the ordering when two policies have the same success probability and same expected success length but different distributions of success times. Example: H=4, two policies both with p=1 and L=2.5, one with P(τ=2)=P(τ=3)=0.5, the other with P(τ=1)=0.25, P(τ=4)=0.75. The second has strictly lower discounted return for all γ close enough to 1, so it is shortest-path but not Blackwell optimal. So the equality fails; the inclusion is what the practical narrative actually needs, and that part holds.\n\nWhat's genuinely new: the restricted-policy-class Blackwell analysis, especially the distribution-over-prompts version of the Blackwell factor bound, and the recipe to discount only reasoning tokens while keeping formatting rewards undiscounted. The theory is self-contained and not circular. The empirical trend is impressive in its consistency: length reductions show up on GSM8K, MATH, AMC, AIME, MINERVA, OLYMPIAD across four models.\n\nThe soft spots beyond the theorem: γ is chosen as the value that best preserves training accuracy while shortening responses, so the 'accuracy preserved' claim is partly circular, though evaluation on held-out seeds mitigates it. There are no error bars or significance tests on the Pass@1 differences, which look like noise. No comparison to existing length-penalty methods (Arora-Zanette, etc.). The bound in Theorem 3.9 is enormously loose and not computed for the actual models; treat it as a qualitative existence result.\n\nThis is a solid contribution in need of revision. The fix for the theorem is simple: state it as an inclusion and add a remark about higher-order moments. Add error bars, report the selected γ values, and run at least one length-penalty baseline. Then it's a good paper.\n\nWho's it for: anyone working on efficient reasoning for LLMs, and RL people interested in Blackwell optimality in function classes. I'd bring it to reading group and cite the inclusion result. Send it to peer review with major revision requested.","headline":"Useful practical idea and a correct inclusion theorem, but the equality in Thm 3.10 is false as stated; the paper needs a revision, not a desk reject.","tokens_in":17610,"tokens_out":3418,"would_cite":true,"duration_ms":33687,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68Q32","90C40","68T05"],"pacs":[],"model":"deepseek-v4-flash","headline":"Discounting correctness rewards in verifier-based reasoning can shorten chains of thought without sacrificing accuracy, because Blackwell optimal policies are precisely the shortest successful paths among accuracy-maximizing policies.","keywords":["efficient reasoning","Blackwell optimality","discounted reinforcement learning","shortest path policy","chain-of-thought compression","verifier-based RL","response length","large reasoning models"],"falsifier":"Take two deterministic policies with success probability 1 and expected successful length 10: A always succeeds at step 10; B succeeds at step 1 with probability 0.5 and at step 19 with probability 0.5. For γ close to 1, B's discounted return 0.5(1+γ^18) exceeds A's γ^9, so B is γ-optimal while A is not, even though both are in argmin L. If this calculation is valid, the set equality in Theorem 3.10 fails.","tokens_in":16553,"feed_emoji":"⚡","tokens_out":9490,"duration_ms":90498,"temperature":0.7,"pith_summary":"The paper claims that in deterministic verifier problems with finite horizon and binary rewards, there is no inherent accuracy-length trade-off: a Blackwell optimal policy—one that is optimal for every discount factor sufficiently close to 1—simultaneously maximizes undiscounted accuracy and, among accuracy-maximizing policies, minimizes expected response length. It proves this for finite restricted policy classes and for the greedy deployment class induced by softmax training, and it gives an explicit bound on how close to 1 the discount factor must be. It then turns the theory into a training recipe: discount only the correctness reward over reasoning tokens, keep formatting rewards undiscounted, regularize with KL to a moving reference, and compare methods under matched token budgets. Experiments with group relative policy optimization report that discounted training matches Pass@1 accuracy while reducing mean response length by roughly 13–30% across six math benchmarks. A sympathetic reader would care because it suggests shorter reasoning is not a compromise but a target, and that the discount factor can be chosen empirically by bisection on training accuracy.","feed_headline":"Discounting shortens LLM answers without sacrificing accuracy","feed_subtitle":"Theory says accuracy and brevity are not in tension; six math benchmarks confirm shorter responses.","key_machinery":"The load-bearing machinery is the discounted return J_γ(π) treated as a polynomial in γ, plus Blackwell optimality as the selection criterion. For a finite policy class, each pairwise difference J_γ(π) − J_γ(π′) is a polynomial of degree at most H−1, so its sign is eventually constant as γ→1; this gives existence of a Blackwell region and a Blackwell discount factor γ_bw. For the greedy deployment class Σ, the paper encodes the value function via Cramer's rule on the augmented state space, uses integer-coefficient bounds on the resulting rational functions to separate polynomial roots, and derives an explicit lower bound on 1−γ_bw. A first-order Taylor expansion of γ^{τ−1} around γ=1 then ex","core_discovery":"The central claim is Theorem 3.10: in a deterministic verifier MDP with finite horizon and binary rewards, the set of Blackwell optimal policies equals the set of shortest-path policies over the accuracy-maximizing class, Π⋆bw = argmin_{π∈Π_maxp} L(π), where p(π) is success probability, L(π) is expected successful response length, and Π_maxp is the class of policies achieving maximal p. The proof expands the discounted return around γ=1 as J_γ(π) = p(π)(1 − ε(L(π)−1)) + O(ε²) with ε=1−γ, making the ordering lexicographic: accuracy first, then length. The same statement holds for the deployment class Σ of deterministic greedy policies induced by a softmax training class. For finite classes th","pith_inferences":["A direct test of the plateau prediction is to plot Pass@1 against mean response length for a fixed model and benchmark while sweeping γ; the theory predicts a flat region followed by a drop, and the width of the flat region is the practical margin for shortening.","The epsilon expansion opens a route to a refinement: if success-time distributions differ in higher moments, policies that succeed earlier in distribution should be preferred, suggesting a variance-aware version of the shortest-path criterion.","The bound in Theorem 3.9 is explicit but exponential in the augmented state-space size, so for realistic language-model state spaces it is unlikely to be computable; the empirical bisection on training accuracy is probably the only usable calibration, and it does not directly verify membership in the Blackwell region.","If the accuracy–length plateau is not specific to math, the same recipe should transfer to code or tool-use tasks with verifiable rewards, where shorter successful trajectories reduce latency and serving cost."],"forward_implications":["For any finite restricted policy class, or for the greedy deployment class Σ, there is a nonempty set of policies that simultaneously maximize undiscounted accuracy and minimize expected successful response length; every γ-optimal policy for γ ∈ (γ_bw, 1) is one of them.","The frequently reported trend that longer reasoning improves accuracy is, on this view, a regime effect: it should vanish inside the Blackwell region and reappear when the discount is chosen below γ_bw.","Because a small per-step negative reward is equivalent to discounting in deterministic verifier MDPs, properly tuned length-penalty methods inherit the same accuracy-then-length ordering.","The practical recipe of discounting only the correctness reward as a function of reasoning tokens, leaving formatting rewards undiscounted, KL-regularizing to a moving reference, and matching token budgets produces, in the paper's measurements, equal or better Pass@1 with 13–30% shorter mean responses across GSM8K, MATH, AMC, AIME, MINERVA, and OLYMPIAD.","The explicit bound on γ_bw gives a principled calibration rule: sweep γ downward from 1 and stop where training accuracy begins to dip; the theory predicts this selected γ lies in the Blackwell region when the deployment class is finite."],"fun_headline_variants":["Blackwell optimality shows shorter CoT can keep accuracy","Discounted RL trims reasoning tokens, preserving accuracy","Proving accuracy and brevity aren't at odds in LLM reasoning","Lexicographic returns: best accuracy, then shortest answer","Token cost penalty yields concise reasoning without loss"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The unstated premise, located in the proof of Theorem 3.10 via Lemma A.13, is that among policies with equal success probability and equal expected successful length, the O(ε²) remainder cannot reverse the γ-ordering for γ close to 1; this is false when success-time distributions differ in higher moments.","fun_headline_variants_meta":{"raw":{"variants":["Blackwell optimality shows shorter CoT can keep accuracy","Discounted RL trims reasoning tokens, preserving accuracy","Proving accuracy and brevity aren't at odds in LLM reasoning","Lexicographic returns: best accuracy, then shortest answer","Token cost penalty yields concise reasoning without loss"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000218,"raw_usage":{"total_tokens":1231,"prompt_tokens":652,"completion_tokens":579,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":396,"completion_tokens_details":{"reasoning_tokens":499}},"tokens_in":396,"tokens_out":579,"duration_ms":5835,"temperature":1.0,"reasoning_tokens":499,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T07:50:04.157236+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take two deterministic policies with success probability 1 and expected successful length 10: A always succeeds at step 10; B succeeds at step 1 with probability 0.5 and at step 19 with probability 0.5. For γ close to 1, B's discounted return 0.5(1+γ^18) exceeds A's γ^9, so B is γ-optimal while A is not, even though both are in argmin L. If this calculation is valid, the set equality in Theorem 3.10 fails.","supporting_citations":[],"review_version":1}