Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

Discounting correctness rewards in verifier-based reasoning can shorten chains of thought without sacrificing accuracy, because Blackwell optimal policies are precisely the shortest successful paths among accuracy-maximizing policies.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 07:50 UTC pith:Q65M6DBO

load-bearing objection Useful practical idea and a correct inclusion theorem, but the equality in Thm 3.10 is false as stated; the paper needs a revision, not a desk reject. the 3 major comments →

arxiv 2510.23486 v3 pith:Q65M6DBO submitted 2025-10-27 cs.LG

Learning to Reason Efficiently with Discounted Reinforcement Learning

Alex Ayoub , Kavosh Asadi , Dale Schuurmans , Csaba Szepesv\'ari , Karim Bouyarmane This is my paper
classification cs.LG MSC 68Q3290C4068T05
keywords efficient reasoningBlackwell optimalitydiscounted reinforcement learningshortest path policychain-of-thought compressionverifier-based RLresponse lengthlarge reasoning models
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that in deterministic verifier problems with finite horizon and binary rewards, there is no inherent accuracy-length trade-off: a Blackwell optimal policy—one that is optimal for every discount factor sufficiently close to 1—simultaneously maximizes undiscounted accuracy and, among accuracy-maximizing policies, minimizes expected response length. It proves this for finite restricted policy classes and for the greedy deployment class induced by softmax training, and it gives an explicit bound on how close to 1 the discount factor must be. It then turns the theory into a training recipe: discount only the correctness reward over reasoning tokens, keep formatting rewards undiscounted, regularize with KL to a moving reference, and compare methods under matched token budgets. Experiments with group relative policy optimization report that discounted training matches Pass@1 accuracy while reducing mean response length by roughly 13–30% across six math benchmarks. A sympathetic reader would care because it suggests shorter reasoning is not a compromise but a target, and that the discount factor can be chosen empirically by bisection on training accuracy.

Core claim

The central claim is Theorem 3.10: in a deterministic verifier MDP with finite horizon and binary rewards, the set of Blackwell optimal policies equals the set of shortest-path policies over the accuracy-maximizing class, Π⋆bw = argmin_{π∈Π_maxp} L(π), where p(π) is success probability, L(π) is expected successful response length, and Π_maxp is the class of policies achieving maximal p. The proof expands the discounted return around γ=1 as J_γ(π) = p(π)(1 − ε(L(π)−1)) + O(ε²) with ε=1−γ, making the ordering lexicographic: accuracy first, then length. The same statement holds for the deployment class Σ of deterministic greedy policies induced by a softmax training class. For finite classes th

What carries the argument

The load-bearing machinery is the discounted return J_γ(π) treated as a polynomial in γ, plus Blackwell optimality as the selection criterion. For a finite policy class, each pairwise difference J_γ(π) − J_γ(π′) is a polynomial of degree at most H−1, so its sign is eventually constant as γ→1; this gives existence of a Blackwell region and a Blackwell discount factor γ_bw. For the greedy deployment class Σ, the paper encodes the value function via Cramer's rule on the augmented state space, uses integer-coefficient bounds on the resulting rational functions to separate polynomial roots, and derives an explicit lower bound on 1−γ_bw. A first-order Taylor expansion of γ^{τ−1} around γ=1 then ex

Load-bearing premise

The unstated premise, located in the proof of Theorem 3.10 via Lemma A.13, is that among policies with equal success probability and equal expected successful length, the O(ε²) remainder cannot reverse the γ-ordering for γ close to 1; this is false when success-time distributions differ in higher moments.

What would settle it

Take two deterministic policies with success probability 1 and expected successful length 10: A always succeeds at step 10; B succeeds at step 1 with probability 0.5 and at step 19 with probability 0.5. For γ close to 1, B's discounted return 0.5(1+γ^18) exceeds A's γ^9, so B is γ-optimal while A is not, even though both are in argmin L. If this calculation is valid, the set equality in Theorem 3.10 fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • For any finite restricted policy class, or for the greedy deployment class Σ, there is a nonempty set of policies that simultaneously maximize undiscounted accuracy and minimize expected successful response length; every γ-optimal policy for γ ∈ (γ_bw, 1) is one of them.
  • The frequently reported trend that longer reasoning improves accuracy is, on this view, a regime effect: it should vanish inside the Blackwell region and reappear when the discount is chosen below γ_bw.
  • Because a small per-step negative reward is equivalent to discounting in deterministic verifier MDPs, properly tuned length-penalty methods inherit the same accuracy-then-length ordering.
  • The practical recipe of discounting only the correctness reward as a function of reasoning tokens, leaving formatting rewards undiscounted, KL-regularizing to a moving reference, and matching token budgets produces, in the paper's measurements, equal or better Pass@1 with 13–30% shorter mean responses across GSM8K, MATH, AMC, AIME, MINERVA, and OLYMPIAD.
  • The explicit bound on γ_bw gives a principled calibration rule: sweep γ downward from 1 and stop where training accuracy begins to dip; the theory predicts this selected γ lies in the Blackwell region when the deployment class is finite.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A direct test of the plateau prediction is to plot Pass@1 against mean response length for a fixed model and benchmark while sweeping γ; the theory predicts a flat region followed by a drop, and the width of the flat region is the practical margin for shortening.
  • The epsilon expansion opens a route to a refinement: if success-time distributions differ in higher moments, policies that succeed earlier in distribution should be preferred, suggesting a variance-aware version of the shortest-path criterion.
  • The bound in Theorem 3.9 is explicit but exponential in the augmented state-space size, so for realistic language-model state spaces it is unlikely to be computable; the empirical bisection on training accuracy is probably the only usable calibration, and it does not directly verify membership in the Blackwell region.
  • If the accuracy–length plateau is not specific to math, the same recipe should transfer to code or tool-use tasks with verifiable rewards, where shorter successful trajectories reduce latency and serving cost.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes discounted reinforcement learning as a principled way to shorten LLM reasoning traces without harming accuracy. It models verifier-based reasoning as a finite-horizon deterministic MDP with binary terminal reward and studies Blackwell optimality in possibly restricted policy classes. The main theoretical results are: (i) for finite policy classes, gamma-discounted optimal policies stabilize to a Blackwell-optimal set near gamma=1; (ii) an explicit, though extremely small, upper bound on the Blackwell discount factor for greedy deployment policies; and (iii) a characterization (Theorem 3.10) that Blackwell-optimal policies are exactly the shortest-path policies among accuracy maximizers. The authors instantiate this in GRPO by discounting only reasoning-token environment rewards, keeping KL regularization, and matching token budgets, and report experiments on GSM8K, MATH, AMC, AIME, MINERVA, and OLYMPIAD showing accuracy preservation with shorter responses.

Significance. If the inclusion direction of Theorem 3.10 is correct, the paper makes a useful conceptual point: in the near-undiscounted regime, an optimal policy must first maximize success probability and then minimize conditional success length, giving a principled accuracy-then-length ordering. The finite-class stabilization argument and the distribution-dependent Blackwell-factor bound are also valuable, and the multi-seed experimental protocol with paired evaluation seeds is a strength. However, the stated equality is stronger than the proof and is false; the empirical protocol selects gamma on training accuracy, so the experiments provide only partial support. With the theorem restated as an inclusion and the empirical claims softened, the paper would be a solid contribution.

major comments (3)
  1. [Theorem 3.10 / Lemma A.13 / Theorem A.14] The proof of Theorem 3.10 establishes only the inclusion: every Blackwell-optimal policy is a shortest-path policy. The converse is asserted but not derived, and it is false. Lemma A.13 gives J_gamma(pi)=p(pi)[1-epsilon(L(pi)-1)]+R_pi(epsilon) with |R_pi(epsilon)| <= C_H epsilon^2; when two policies have equal p and L, the first-order terms cancel and the ordering for gamma sufficiently close to 1 is determined by the O(epsilon^2) remainder, i.e., by the full success-time distribution, not just by L. Concretely, take H=5, two start states each with probability 1/2, and two deterministic policies: pi_A succeeds at times 2 and 4 on the two states, pi_B succeeds at times 1 and 5. Both have p=1 and L=3. With epsilon=1-gamma, J_gamma(pi_A)-J_gamma(pi_B)=0.5gamma+0.5gamma^3-0.5-0.5gamma^4 = -0.5epsilon^2(3-3epsilon+epsilon^2) < 0 for epsilon>0. Thus pi_A is in argmin_{Pi_maxp} L but is not Bla
  2. [Section 5 'Variance control and reporting'; Section 4 'Practical notes (i)'] The empirical claim that discounted GRPO 'matches Pass@1 accuracy while substantially shortening responses' is weakened by the gamma-selection protocol. The authors select, per model/dataset, the discounted configuration whose training Pass@1 matches or exceeds the undiscounted run and then choose the one with the shortest mean response length; all tabled metrics are computed for that selected configuration. This is a selection-on-success protocol: on the training split, 'accuracy preserved' is true by construction for the chosen gamma, and the reported evaluation numbers are for the selected configuration only. The paper does not verify that the chosen gamma lies in the Blackwell interval (gamma_bw,1), nor does it use the Theorem 3.9 bound. Figure 1 provides some useful evidence of the full trade-off curve, but the main tables do not. I would ask the authors to either preselect gamma on
  3. [Section 3.2 / Eq. (2) / Section 4] There is a theory-practice gap that should be acknowledged. The theoretical results are stated for the finite deployment class Sigma of greedy deterministic policies, while the training objective in Eq. (2) is optimized over stochastic softmax policies with a moving KL reference, and GRPO updates the stochastic policy. No result in the paper shows that the greedy policy induced by the trained stochastic policy converges to a Blackwell-optimal element of Sigma, or that optimizing Eq. (2) with a given gamma is equivalent to finding a Blackwell-optimal policy in Sigma. Additionally, the theoretical MDP discounts every step (gamma^{t-1}), whereas the practical recipe in Eq. (1) discounts only a masked count K(tau) of reasoning tokens; the theorem on shortest total response length does not literally apply to the masked objective. The paper should state these as explicit assumptions or heurist
minor comments (4)
  1. [Theorem 3.9] The explicit bound on the Blackwell discount factor is not connected to the practical choice of gamma; because it depends on the full augmented state space, it is not intended to be computed. Please state more explicitly that it is an existence/qualitative bound, not a practical selector.
  2. [Section 5 / Table 2] Table 2 reports averages over 5 evaluation seeds per model, with no training-seed variance. Some cells show large accuracy differences (e.g., Phi-4 AMC 51.00 to 61.00). Add confidence intervals or standard errors, or note that such differences may be within noise.
  3. [Abstract / Introduction] The phrase 'Experiments confirm our theoretical results' is too strong given the selection protocol and the theory-practice gap. Consider wording such as 'are consistent with the inclusion direction' or 'provide evidence that discounting can shorten responses without sacrificing accuracy for the selected gamma.'
  4. [Section 4.3] The reported length reductions are in total response length, while the objective discounts only reasoning tokens. Clarify whether the reported length is total tokens or reasoning tokens, and report reasoning-token length separately if available.

Circularity Check

0 steps flagged

No significant circularity: the theoretical derivation is self-contained, the empirical gamma sweep is transparent held-out model selection, and the flagged proof gap is a correctness issue rather than a circular step.

full rationale

The main derivation chain (Definitions 3.1--3.2, Theorem 3.4, Lemma 3.7, Lemma A.13, Theorem A.14) proceeds from an external notion of Blackwell optimality to a characterization of shortest-path policies; no equation is assumed equal to the conclusion and no fitted parameter is renamed as a prediction inside the theory. Citations such as Blackwell (1962), Puterman (2014), and Grand-Clément & Petrik (2023) are external classical results, and the sparse co-authored citations (e.g., Szepesvári 2022 for standard MDP background) are not load-bearing. The empirical protocol is also not a fit to the reported outcome: Section 5 states 'among all discounted settings whose training Pass@1 matches or exceeds that of the undiscounted run, we choose the one with the shortest mean response length. All tabled metrics are then computed on the evaluation seeds for the selected configuration.' Thus the training-side accuracy/length match is a selection criterion, but the tabled Pass@1 and length figures are held-out evaluation metrics, which can fail and therefore provide independent (though selection-biased) evidence. The concern about Lemma A.13's O(epsilon^2) remainder and the converse inclusion in Theorem 3.10 is a proof-gap/soundness criticism: the proof does not use the conclusion as an input, so it is not circularity. Overall, no significant circularity is present.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

No new physical or mathematical entities are postulated. The free parameters are training hyperparameters, most importantly γ, which is selected to fit the empirical accuracy target. The axioms are standard for Blackwell-optimality analysis plus domain-specific assumptions about the LLM reasoning MDP.

free parameters (3)
  • discount factor γ = not reported
    Chosen per model/dataset by bisection to preserve undiscounted training accuracy while shortening responses; central to the empirical claim but values are not disclosed.
  • KL regularization coefficient β = not reported
    Tuned to maximize the undiscounted model's accuracy and then applied to discounted methods; affects stability and final accuracy.
  • reference policy update frequency u = not reported
    Chosen via ablations to maximize undiscounted accuracy; affects how quickly the moving reference anchor follows the policy.
axioms (5)
  • domain assumption Finite horizon H, finite state and action sets, deterministic transitions, and binary verifier reward.
    Section 2 models reasoning as a finite-horizon MDP with deterministic next state St+1 = St At and terminal verifier reward 1 iff correct. This is a modeling choice; real LLM decoding is stochastic except under greedy deployment.
  • domain assumption The admissible policy class Π is finite, or the deployment class Σ induced by softmax training and greedy decoding is finite.
    Used in Theorem 3.4 and Lemma A.8 to guarantee existence of Blackwell optimal policies and to bound the Blackwell factor. The empirical model class is not shown to be exactly this finite class.
  • domain assumption Rational transition probabilities, rewards, and initial distribution over questions (Assumption 3.8).
    Needed for the explicit Blackwell-factor bound in Theorem 3.9; the paper notes deterministic binary rewards satisfy this with m=1, but the initial distribution over prompts in training is empirical.
  • domain assumption In deterministic MDPs with binary rewards, discounting is equivalent to a small per-step negative reward (Bertsekas, 2012).
    Used to extend conclusions to length-penalty methods in Section 3 and the conclusion; requires deterministic transitions and binary terminal reward.
  • standard math Cramer's-rule and polynomial-root machinery from Grand-Clément & Petrik (2023).
    Appendix A.11 imports their determinant formulas and root-separation bounds to prove Theorem 3.9.

pith-pipeline@v1.3.0-alltime-deepseek · 16254 in / 11935 out tokens · 135635 ms · 2026-08-04T07:50:04.157236+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Learning to Reason Efficiently with Discounted Reinforcement Learning." pith.science (2026). https://pith.science/paper/Q65M6DBO

@misc{pith2026251023486,
  author       = {Pith},
  title        = {Pith review of: Learning to Reason Efficiently with Discounted Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q65M6DBO}},
  note         = {Machine review of arXiv:2510.23486}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Large reasoning models (LRMs) often consume excessive tokens, inflating computational cost and latency. More broadly, in goal reaching sequential decision problems we often want to reach the goal quickly, and LRM reasoning can be viewed through this lens. We challenge the assumption that longer responses improve accuracy. By penalizing reasoning tokens using a discounted reinforcement learning setup (interpretable as a small token cost) and analyzing Blackwell optimality in restricted policy classes, we encourage concise yet accurate reasoning, analogous to preferring shorter successful trajectories in a stochastic shortest path problem. Experiments confirm our theoretical results that this approach shortens chains of thought while preserving accuracy.

Figures

Figures reproduced from arXiv: 2510.23486 by Alex Ayoub, Csaba Szepesv\'ari, Dale Schuurmans, Karim Bouyarmane, Kavosh Asadi.

Figure 1
Figure 1. Figure 1: GSM8K accuracy (blue, left) and tokens (orange, right) vs. discount (1 − γ). Variance control and reporting. To obtain stable estimates, we repeat each training run with 3 random training seeds and, for each trained model, evaluate with 10 independent sampling seeds on GSM8K and MATH; we report averages over 3 × 10 runs per condition and fix evaluation seeds across methods for paired comparisons. For AMC 2… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learning

    cs.LG 2026-06 unverdicted novelty 5.0

    Survey mapping RL techniques onto LLM training and highlighting gaps in value-based, off-policy, and bootstrapping methods.

Reference graph

Works this paper leans on

29 extracted references · 23 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Phi-4 techni- cal report.arXiv preprint arXiv:2412.08905,

    Marah Abdin, Jyoti Aneja, Harkirat Behl, S ´ebastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 techni- cal report.arXiv preprint arXiv:2412.08905,

    Pith/arXiv arXiv

  2. [3]

    Training language models to reason efficiently.arXiv preprint arXiv:2502.04463,

    Daman Arora and Andrea Zanette. Training language models to reason efficiently.arXiv preprint arXiv:2502.04463,

    arXiv

  3. [7]

    Concise reasoning via reinforcement learning.arXiv preprint arXiv:2504.05185,

    Mehdi Fatemi, Banafsheh Rafiee, Mingjie Tang, and Kartik Talamadupula. Concise reasoning via reinforcement learning.arXiv preprint arXiv:2504.05185,

    arXiv

  4. [9]

    Train long, think short: Curriculum learning for efficient rea- soning.arXiv preprint arXiv:2508.08940,

    Hasan Abed Al Kader Hammoud, Kumail Alhamoud, Abed Hammoud, Elie Bou-Zeid, Marzyeh Ghassemi, and Bernard Ghanem. Train long, think short: Curriculum learning for efficient rea- soning.arXiv preprint arXiv:2508.08940,

    Pith/arXiv arXiv

  5. [10]

    Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874,

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874,

    Pith/arXiv arXiv

  6. [12]

    Tulu 3: Pushing frontiers in open language model post-training.arXiv preprint arXiv:2411.15124,

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brah- man, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training.arXiv preprint arXiv:2411.15124,

    Pith/arXiv arXiv

  7. [13]

    How well do llms compress their own chain-of-thought? a token complexity approach.arXiv preprint arXiv:2503.01141,

    Ayeong Lee, Ethan Che, and Tianyi Peng. How well do llms compress their own chain-of-thought? a token complexity approach.arXiv preprint arXiv:2503.01141,

    Pith/arXiv arXiv

  8. [17]

    Ta- pered off-policy reinforce: Stable and efficient reinforcement learning for llms.arXiv preprint arXiv:2503.14286,

    Nicolas Le Roux, Marc G Bellemare, Jonathan Lebensold, Arnaud Bergeron, Joshua Greaves, Alex Fr ´echette, Carolyne Pelletier, Eric Thibodeau-Laufer, S ´andor Toth, and Sam Work. Ta- pered off-policy reinforce: Stable and efficient reinforcement learning for llms.arXiv preprint arXiv:2503.14286,

    Pith/arXiv arXiv

  9. [18]

    Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models.arXiv preprint arXiv:2402.03300,

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models.arXiv preprint arXiv:2402.03300,

    Pith/arXiv arXiv

  10. [19]

    Hawkeye: Efficient reasoning with model collaboration.arXiv preprint arXiv:2504.00424,

    Jianshu She, Zhuohao Li, Zhemin Huang, Qi Li, Peiran Xu, Haonan Li, and Qirong Ho. Hawkeye: Efficient reasoning with model collaboration.arXiv preprint arXiv:2504.00424,

    Pith/arXiv arXiv

  11. [20]

    Sample more to think less: Group filtered policy optimization for concise reasoning.arXiv preprint arXiv:2508.09726,

    Vaishnavi Shrivastava, Ahmed Awadallah, Vidhisha Balachandran, Shivam Garg, Harkirat Behl, and Dimitris Papailiopoulos. Sample more to think less: Group filtered policy optimization for concise reasoning.arXiv preprint arXiv:2508.09726,

    Pith/arXiv arXiv

  12. [22]

    Stop overthinking: A survey on efficient reasoning for large language models.arXiv preprint arXiv:2503.16419,

    Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen Zhong, Na Zou, et al. Stop overthinking: A survey on efficient reasoning for large language models.arXiv preprint arXiv:2503.16419,

    Pith/arXiv arXiv

  13. [24]

    Just enough thinking: Efficient reasoning with adaptive length penalties reinforcement learning.arXiv preprint arXiv:2506.05256,

    Violet Xiang, Chase Blagden, Rafael Rafailov, Nathan Lile, Sang Truong, Chelsea Finn, and Nick Haber. Just enough thinking: Efficient reasoning with adaptive length penalties reinforcement learning.arXiv preprint arXiv:2506.05256,

    Pith/arXiv arXiv

  14. [25]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    Pith/arXiv arXiv

  15. [26]

    Saber: Switchable and balanced training for efficient llm reasoning.arXiv preprint arXiv:2508.10026,

    Kai Zhao, Yanjun Zhao, Jiaming Song, Shien He, Lusheng Zhang, Qiang Zhang, and Tian- jiao Li. Saber: Switchable and balanced training for efficient llm reasoning.arXiv preprint arXiv:2508.10026,

    Pith/arXiv arXiv

  16. [28]

    A (possibly nonstationary) finite-horizon policyπ= (π t)H t=1 induces the stationary policy ˜π(a|(s, t)) =πt(a|s) (t≤H),˜π(· |absorb)arbitrary

    The initial distribution on augmented states is˜µwith˜µ((s,1)) =µ(s)and zero elsewhere. A (possibly nonstationary) finite-horizon policyπ= (π t)H t=1 induces the stationary policy ˜π(a|(s, t)) =πt(a|s) (t≤H),˜π(· |absorb)arbitrary. We fix a deterministic tie breaking rule onAand define the greedification map on augmented states Greed(π,(s, t))∈arg max a∈A...

    2023

  17. [29]

    By Proposition A.6 of Grand-Cl´ement & Petrik (2023),m | ˜S| n(·, s, π)has integer coefficients and NX k=0 [X k] m| ˜S| n(·, s, π) ≤ |˜S|r ∞ m| ˜S| 2| ˜S|

    Sincedeg ¯n≤ |˜S| −1anddegd≤ |˜S|, we havedegp≤N:= 2| ˜S| −1. By Proposition A.6 of Grand-Cl´ement & Petrik (2023),m | ˜S| n(·, s, π)has integer coefficients and NX k=0 [X k] m| ˜S| n(·, s, π) ≤ |˜S|r ∞ m| ˜S| 2| ˜S| . Thusm | ˜S| D˜µ¯n(·, π) =P s D˜µ˜µ(s) m| ˜S| n(·, s, π)has integer coefficients and coefficient-sum at mostD ˜µ| ˜S|r ∞ m| ˜S| 2| ˜S|. By ...

    2023

  18. [1962]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

    Pith/arXiv arXiv

  19. [1992]

    Effectively controlling reasoning models through thinking intervention.arXiv preprint arXiv:2503.24370,

    Tong Wu, Chong Xiang, Jiachen T Wang, G Edward Suh, and Prateek Mittal. Effectively controlling reasoning models through thinking intervention.arXiv preprint arXiv:2503.24370,

    Pith/arXiv arXiv

  20. [1996]

    A OMITTEDPROOFS We adapt classical Blackwell arguments (Zwick & Paterson, 1996; Puterman, 2014; Grand-Cl´ement & Petrik,

    1996

  21. [2007]

    Openai o1 system card.arXiv preprint arXiv:2412.16720,

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card.arXiv preprint arXiv:2412.16720,

    Pith/arXiv arXiv

  22. [2010]

    Thinking fast and right: Balancing accuracy and reasoning length with adaptive rewards.arXiv preprint arXiv:2505.18298,

    Jinyan Su and Claire Cardie. Thinking fast and right: Balancing accuracy and reasoning length with adaptive rewards.arXiv preprint arXiv:2505.18298,

    Pith/arXiv arXiv

  23. [2014]

    Concise: Confidence-guided compression in step-by-step efficient reasoning.arXiv preprint arXiv:2505.04881,

    Ziqing Qiao, Yongheng Deng, Jiali Zeng, Dong Wang, Lai Wei, Guanbo Wang, Fandong Meng, Jie Zhou, Ju Ren, and Yaoxue Zhang. Concise: Confidence-guided compression in step-by-step efficient reasoning.arXiv preprint arXiv:2505.04881,

    arXiv

  24. [2020]

    Scaling up rl: Unlocking di- verse reasoning in llms via prolonged training.arXiv preprint arXiv:2507.12507, 2025a

    Mingjie Liu, Shizhe Diao, Jian Hu, Ximing Lu, Xin Dong, Hao Zhang, Alexander Bukharin, Shaokun Zhang, Jiaqi Zeng, Makesh Narsimhan Sreedhar, et al. Scaling up rl: Unlocking di- verse reasoning in llms via prolonged training.arXiv preprint arXiv:2507.12507, 2025a. Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong. P...

    Pith/arXiv arXiv

  25. [2021]

    S-grpo: Early exit via reinforcement learning in reasoning models.arXiv preprint arXiv:2505.07686,

    Muzhi Dai, Chenxu Yang, and Qingyi Si. S-grpo: Early exit via reinforcement learning in reasoning models.arXiv preprint arXiv:2505.07686,

    Pith/arXiv arXiv

  26. [2022]

    Fast on the easy, deep on the hard: Efficient reasoning via powered length penalty.arXiv preprint arXiv:2506.10446,

    Zehui Ling, Deshu Chen, Hongwei Zhang, Yifeng Jiao, Xin Guo, and Yuan Cheng. Fast on the easy, deep on the hard: Efficient reasoning via powered length penalty.arXiv preprint arXiv:2506.10446,

    Pith/arXiv arXiv

  27. [2023]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Pith/arXiv arXiv

  28. [2024]

    L1: Controlling how long a reasoning model thinks with reinforcement learning.arXiv preprint arXiv:2503.04697,

    Pranjal Aggarwal and Sean Welleck. L1: Controlling how long a reasoning model thinks with reinforcement learning.arXiv preprint arXiv:2503.04697,

    Pith/arXiv arXiv

  29. [2025]

    Conciserl: Conciseness-guided reinforcement learning for efficient reasoning models.arXiv preprint arXiv:2505.17250,

    Razvan-Gabriel Dumitru, Darius Peteleaza, Vikas Yadav, and Liangming Pan. Conciserl: Conciseness-guided reinforcement learning for efficient reasoning models.arXiv preprint arXiv:2505.17250,

    Pith/arXiv arXiv