Pith. sign in

REVIEW 4 major objections 5 minor 8 cited by

Know When to Explore: Difficulty-Aware Certainty as a Guide for LLM Reinforcement Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that an LLM's self-certainty, modulated by per-problem difficulty, is a sufficient intrinsic reward to beat outcome-only RL baselines on hard math, with gains that grow when test-time compute is scaled.

desk verdict The DACE idea is genuinely novel, but the equations in Section 4 reverse the algorithm's stated mechanism; the paper needs major revision before its empirical claims can be trusted. read the letter →

arxiv 2509.00125 v1 pith:YNNPUD75 submitted 2025-08-29 cs.AI

Ang Li , Zhihang Yuan , Yang Zhang , Shouda Liu , Yisen Wang This is my paper
classification cs.AI
keywords difficulty-awareexplorationself-certaintyreinforcementlearningwithverifiablefeedbackGRPOexploration-exploitationtrade-offLLMmathematicalreasoningintrinsicrewardtest-timecomputescaling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Reinforcement learning for LLM reasoning typically pays only for correct final answers, so the model cannot tell a promising wrong path from a dead end, or a concise proof from a bloated one. This paper argues that the model's own self-certainty—measured as the negative average log-probability of its generated tokens—carries that missing granularity, provided it is read relative to task difficulty. DACE therefore estimates difficulty online from the policy's success rate and uses it to flip an intrinsic reward: hard problems get a penalty for high certainty (push to explore), easy problems get a reward for high certainty (push to exploit). On competition-level math benchmarks the method beats a strong GRPO baseline by up to 2.9 points, and the advantage grows when more answers are sampled at test time, which the authors take as evidence that exploration discovered genuinely new correct solution paths without sacrificing precision.

What carries the argument

The adaptive coefficient α(x; π) = α_scale * sgn(β_threshold - diff(x; π)) is the load-bearing piece. It converts a policy-relative difficulty estimate—the empirical failure rate over n sampled responses (Eq. 3)—into a sign flip on a certainty-based intrinsic reward, the negative average token log-probability (Eq. 4): a negative coefficient on hard problems drives certainty down (exploration), a positive coefficient on easy problems drives certainty up (exploitation). Because the difficulty estimate reuses the same n responses GRPO already samples for its policy update, the mechanism adds no extra rollouts.

What would settle it

Sort a policy's responses on AIME-style problems by self-certainty, then split by difficulty (success rate below or above 0.4). If, on hard problems, the highest-certainty responses are not less accurate than the lowest-certainty ones—or on easy problems high-certainty responses are not more accurate—then DACE's reward sign is wrong on those problems and the intrinsic reward fights the outcome reward. The paper asserts the correlation through two anecdotes in Figure 1 but never quantifies it; a direct measurement across difficulty bands settles the claim.

Watch

Extended reading notes

Core claim

Central claim: the exploration-exploitation balance in LLM reinforcement learning should be a per-problem, time-varying decision. DACE operationalizes this with an intrinsic reward R_int = α(x;π)*C(y,x;π), where C is the policy's self-certainty (Eq. 4) and the coefficient α = α_scale*sgn(β_threshold - diff(x;π)) flips sign with difficulty, estimated as the policy's empirical failure rate on that problem from n sampled responses (Eq. 3). Failing problems get certainty minimized (explore); mastered problems get certainty maximized (exploit). Over GRPO, DACE reports +1.3 on AIME25, +2.9 on AIME24, +0.5 on AMC23, parity on MATH-500, and a widening lead under test-time compute scaling. A threshol

Load-bearing premise

The load-bearing premise is that an LLM's self-certainty—the average log-probability of the words it generates—reliably tracks whether its answer is good and whether more exploration on that problem would help; the paper illustrates this with two examples but never measures the correlation.

Editorial extensions

If this is right

  • DACE beats a strong GRPO baseline on competition-level math: +1.3 on AIME25, +2.9 on AIME24, +0.5 on AMC23, with parity on MATH-500.
  • The advantage widens with test-time compute: on AIME25 the lead grows from +1.2 points at 16 samples to +3.3 points at 128 samples, indicating more diverse correct solution paths.
  • Fixed strategies are suboptimal at both ends: pure exploration produces roughly 3x longer, inefficient responses, while pure exploitation stalls in local optima; intermediate thresholds perform best.
  • Training dynamics show DACE injects a distinct exploratory phase mid-training—lower self-certainty, higher token entropy, longer responses—before metrics converge.
  • DACE estimates difficulty from the same group samples GRPO already draws, so the adaptive reward costs essentially no extra compute.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Appendix A.4 shows the raw certainty signal was prone to reward hacking and needed stabilizers (group-wise normalization, zero reward for answers containing executable code); a controlled ablation removing only the adaptive coefficient would reveal how much of the gain is the difficulty signal versus these stabilizers.
  • The paper's principle, if it holds for log-probability certainty, is testable as a drop-in for other intrinsic signals—self-consistency scores, process-reward estimates, or entropy bonuses—by wrapping them in the same difficulty-adaptive sign flip.
  • The threshold sweep's best average accuracy comes at β=0.8 rather than the default 0.4, and raising α also helps; this suggests the exact operating point matters less than having a working switch, and a continuous per-problem difficulty weight might beat the binary flip.
  • The per-problem difficulty estimate is noisy (n=16 responses per problem); comparing difficulty estimates from 8, 16, and 64 samples would show whether the adaptive reward's benefit scales with estimate quality or plateaus.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes DACE, an intrinsic-reward modification of GRPO for LLM math reasoning. It estimates per-task difficulty by the current policy's failure rate (Eq. 3) and defines a certainty metric as negative average log-probability (Eq. 4). A difficulty-dependent coefficient (Eqs. 5-6) is intended to penalize high certainty (encourage exploration) on hard tasks and reward high certainty on easy tasks. Experiments with Qwen2.5-7B on AIME25/AIME24/AMC23/MATH-500 report gains over GRPO, favorable test-time scaling, and ablations of the difficulty threshold and scaling factor.

Significance. The motivating idea--adaptively switching exploration/exploitation based on the policy's own difficulty estimate--is timely and, if implemented as intended, would be a simple plug-in addition to GRPO. The toy Gaussian environment in Section 3 provides a clean demonstration that fixed exploration/exploitation strategies are suboptimal across difficulty, and the test-time scaling result (Figure 3) is a useful falsifiable prediction. However, the paper's value is currently undermined by an algebraic sign inconsistency in the central objective, a lack of statistical rigor, and hyperparameter selection that is not clearly separated from evaluation. With corrections and stronger evidence, the contribution could be solid but not groundbreaking.

major comments (4)
  1. [§4, Eqs. (4)-(7)] The certainty definition has the wrong sign for the stated mechanism. Eq. (4) sets C(y,x;π)=-(1/|y|)Σ log π(y_j|...), so larger C means lower average token probability. Yet the text states 'Maximizing certainty encourages the policy to use high-probability tokens' and the bullets in §4 claim hard tasks (diff>β) lead to minimizing certainty. Substituting Eq. (4) into Eqs. (5)-(7), for hard tasks α=-α_scale and the intrinsic term is -α_scale·C = +α_scale·(average log p); maximizing this drives the policy toward higher-probability tokens (exploitation), the opposite of the claimed exploration. For easy tasks the algebra gives exploration. This reverses DACE's core behavior. The experimental training dynamics (Figure 4, lower self-certainty/higher entropy) suggest the implementation used the opposite sign, but the paper as written does not define the correct objective. Please correct Eq. (4)
  2. [§5.2, Table 3; §5, setup] The reported default hyperparameters are not the best in the ablation and appear to be selected on the evaluation benchmarks. Table 3a shows average accuracy is maximized at β=0.8 (47.60), not the default β=0.4 (46.89); Table 3b shows α=0.10 improves on α=0.05. If the default is arbitrary or chosen after seeing these numbers, the +1.3/+2.9 gains are not a fair estimate. Moreover, all experiments are single-seed and no error bars are reported, yet the headline differences (e.g., +0.5 on AMC23, -0.2 on MATH-500) are within plausible seed noise. Please report multiple seeds/standard deviations or justify why the differences are significant.
  3. [§1, Figure 1; §5.1, Table 2] The load-bearing premise that self-certainty tracks solution quality/difficulty is supported only by two illustrative examples, not quantified. A calibration analysis (e.g., correlation between C and correctness or between diff and outcome) would substantiate the mechanism. Also, the abstract's claim of consistent improvement is contradicted by MATH-500 (81.9 vs. GRPO 82.1, Table 2). The paper should either restrict the claim to competition-level benchmarks or explain the trade-off.
  4. [§5, baselines] The comparison against advanced baselines (Ent-Adv, Clip-Cov, KL-Cov, FR3E) is not controlled: results are taken from original papers and may use different base models, training data, or compute. The DACE-vs-GRPO comparison is the only same-setup comparison. The sentence 'state-of-the-art results on AIME25 and AMC23' should be qualified or supported with re-trained baselines under identical conditions.
minor comments (5)
  1. [§4, Figure 4] 'Self-certainty' is not defined for the plots. If it is C from Eq. (4), then lower values mean higher confidence, which contradicts the narrative of DACE being more exploratory. Please define the metric and reconcile the signs.
  2. [Figure 5] The legend uses 'CCRL' instead of 'DACE' in lines such as 'CCRL (Threshold=1.0)'.
  3. [Eq. (3)] Difficulty is estimated by sampling n responses. Please clarify whether n equals the GRPO group size of 16 and whether the same samples are reused for both difficulty estimation and policy update (as claimed in the text).
  4. [Appendix A.4] The description of group-wise z-score normalization and min-max scaling of certainty is not reflected in Eq. (5), which writes the intrinsic reward as α·C. Please state the exact normalized reward used in the objective; otherwise the training loss is not reproducible.
  5. [Table 2 / Figure 3] The main evaluation is described as 'mean@32', while Figure 3 reports pass@1/4/16/640. Clarify how mean@k is computed and why Figure 3's pass@640 is described in the text as mean@128.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: DACE's gains are measured on external benchmarks, not derived from its own definitions.

full rationale

The central claim that DACE improves math reasoning is an empirical result against AIME25/24, AMC23, and MATH-500, so it cannot reduce to the paper's equations by construction. The difficulty signal (Eq. 3) is a sampled failure rate and the certainty signal (Eq. 4) is a token-level log-probability; neither is fitted to the reported accuracies. The adaptive reward (Eqs. 5-7) is a policy-training objective, and the reported gains are measured after training, not algebraically implied by the reward definition. Hyperparameter sweeps (Table 3) are ablations; even if selection is optimistic, they do not turn the benchmark results into tautologies. There is no load-bearing self-citation: [8] and [27] are external prior works on self-certainty, and no uniqueness or ansatz is imported from the authors' own prior papers. The sign inconsistency between Eq. 4 and the prose in Section 4 is a genuine correctness/specification concern, but it is not circularity: the benchmark numbers are not made true by definition. Therefore the score is 0.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim rests on three unproven assumptions: the certainty-quality correlation, the success-rate difficulty proxy, and the informal transfer from the toy environment. These are not backed by formal argument or quantitative correlation analysis.

free parameters (2)
  • αscale (intrinsic reward scaling) = 0.05 (main runs)
    Controls the magnitude of the certainty reward; grid-searched in Table 3b, where 0.10 yields higher average accuracy than 0.05.
  • βthreshold (difficulty threshold) = 0.4 (main runs)
    Determines the split between hard and easy problems; grid-searched in Table 3a, where 0.8 yields the highest average accuracy.
assumptions (3)
  • domain assumption LLM self-certainty (negative average log-probability) correlates with solution quality and exploration value.
    Introduced in Section 1 and Figure 1; used to justify the intrinsic reward in Eq. 5. No quantitative evidence provided.
  • domain assumption The policy's empirical failure rate over n samples is a reliable proxy for task difficulty.
    Used in Eq. 3 to compute diff(x;π); assumes 16 samples are sufficient.
  • ad hoc to paper Findings from the toy Gaussian environment transfer to autoregressive LLM policies.
    The toy experiment in Section 3 uses a simple 1-D Gaussian policy and hand-crafted reward peaks; the mapping to LLM token probabilities is informal.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Know When to Explore: Difficulty-Aware Certainty as a Guide for LLM Reinforcement Learning." pith.science (2026). https://pith.science/paper/YNNPUD75

@misc{pith2026250900125,
  author       = {Pith},
  title        = {Pith review of: Know When to Explore: Difficulty-Aware Certainty as a Guide for LLM Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YNNPUD75}},
  note         = {Machine review of arXiv:2509.00125}
}
read the original abstract

Reinforcement Learning with Verifiable Feedback (RLVF) has become a key technique for enhancing the reasoning abilities of Large Language Models (LLMs). However, its reliance on sparse, outcome based rewards, which only indicate if a final answer is correct or not, fails to provide granular guidance on the reasoning process itself. This limitation hinders efficient learning, as the model cannot distinguish between high quality and inefficient solutions, nor can it learn effectively from different types of failures. To address this, we observe that an LLMs self-certainty often correlates with task difficulty and solution quality. We introduce Difficulty Aware Certainty guided Exploration (DACE), a novel RL algorithm that leverages this insight to dynamically balance the exploration exploitation trade-off. DACE assesses task difficulty online based on the policys success rate. It then uses this signal to modulate an intrinsic reward: for difficult tasks where the model is struggling, DACE encourages exploration by penalizing high certainty; for easier tasks, it encourages learning efficiency by rewarding high certainty. Experiments on challenging mathematical reasoning benchmarks (AIME, MATH) show that DACE significantly outperforms strong baselines. The DACE-trained models not only achieve higher accuracy but also demonstrate more robust performance when scaling test-time compute, validating that our adaptive approach fosters effective exploration without sacrificing precision.

Discussion (0). Sign in to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. From Reasoning Traces to Reusable Modules: Understanding Compositional Generalization in Language Model Reasoning

    cs.LG 2026-06 conditional novelty 6.5 of 10

    SFT supplies entangled compositional traces of atomic skills and routing modules; RL identifies those modules and enables recombination on novel compositions outside the SFT support.

  2. Formalizing Task-Space Complexity for Zero-Shot Generalization

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    Introduces signed divergence to bound generalization gaps and defines task-space complexity as the minimum source contexts needed for ε-coverage under local smoothness, with set-cover reduction and empirical validatio...

  3. From Reasoning Traces to Reusable Modules: Understanding Compositional Generalization in Language Model Reasoning

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    Introduces a hierarchical latent selection model showing SFT supplies raw module materials in compound traces while RL decomposes them to identify atomic modules and enable recombination for new reasoning configurations.

  4. Epistemic Uncertainty for Test-Time Discovery

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    UG-TTT adds epistemic uncertainty measured by adapter disagreement as an exploration bonus in RL for LLMs, raising maximum reward and diversity on scientific discovery benchmarks.

  5. HTPO: Towards Exploration-Exploitation Balanced Policy Optimization via Hierarchical Token-level Objective Control

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    HTPO introduces hierarchical token-level objective control in RLVR to balance exploration and exploitation by grouping tokens according to difficulty, correctness, and entropy, yielding up to 8.6% gains on AIME benchm...

  6. rePIRL: Learn PRM with Inverse RL for LLM Reasoning

    cs.LG 2026-02 unverdicted novelty 6.0 of 10

    rePIRL learns effective process reward models for LLM reasoning via a dual policy-PRM update process inspired by inverse RL, unifying online and offline methods with reported gains over prior approaches on math and co...

  7. Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learning

    cs.LG 2026-06 unverdicted novelty 5.0 of 10

    Survey mapping RL techniques onto LLM training and highlighting gaps in value-based, off-policy, and bootstrapping methods.

  8. rePIRL: Learn PRM with Inverse RL for LLM Reasoning

    cs.LG 2026-02 conditional novelty 4.0 of 10

    rePIRL learns token-level process rewards for LLM reasoning via a guided-cost-learning-style IRL objective, and shows these rewards improve reasoning policies on math/coding benchmarks.

Reference graph

Works this paper leans on

41 extracted references · 16 canonical work pages · cited by 6 Pith papers

  1. [1]

    Learning to reason with llms, 2024

    OpenAI. Learning to reason with llms, 2024

    2024

  2. [2]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    arXiv 2025

  3. [3]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, YK Li, Y Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    arXiv 2024

  4. [4]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115, 2024

    arXiv 2024

  5. [5]

    Let’s verify step by step

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. InThe TwelfthInternational Conference on Learning Representations, 2023

    work page 2023

  6. [6]

    Math- shepherd: Verify and reinforce llms step-by-step without human annotations.arXiv preprint arXiv:2312.08935, 2023

    Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math- shepherd: Verify and reinforce llms step-by-step without human annotations.arXiv preprint arXiv:2312.08935, 2023

    arXiv 2023

  7. [8]

    Scalable best-of-n selection for large language models via self-certainty

    Zhewei Kang, Xuandong Zhao, and Dawn Song. Scalable best-of-n selection for large language models via self-certainty. arXiv preprint arXiv:2502.18581, 2025

    arXiv 2025

  8. [9]

    The impact of intrinsic rewards on exploration in reinforcement learning

    Aya Kayal, Eduardo Pignatelli, and Laura Toni. The impact of intrinsic rewards on exploration in reinforcement learning. arXiv preprint arXiv:2501.11533, 2025

    arXiv 2025

Show all 41 references
  1. [10]

    Thompson sampling: An asymptotically optimal finite-time analysis

    Emilie Kaufmann, Nathaniel Korda, and Rémi Munos. Thompson sampling: An asymptotically optimal finite-time analysis. In ALT, 2012

    2012

  2. [11]

    Thompson sampling for contextual bandits with linear payoffs

    Shipra Agrawal and Navin Goyal. Thompson sampling for contextual bandits with linear payoffs. InICML, 2013

    2013

  3. [12]

    Exploration-exploitation in constrained mdps.arXiv preprint arXiv:2003.02189, 2020

    Yonathan Efroni, Shie Mannor, and Matteo Pirotta. Exploration-exploitation in constrained mdps.arXiv preprint arXiv:2003.02189, 2020

    2003 arXiv

  4. [13]

    Why is posterior sampling better than optimism for reinforcement learning? In ICML, 2017

    Ian Osband and Benjamin Van Roy. Why is posterior sampling better than optimism for reinforcement learning? In ICML, 2017

    2017

  5. [14]

    First return, then explore

    Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. First return, then explore. Nature, 2021

    2021

  6. [15]

    Automatic goal generation for reinforcement learning agents

    Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. InICML, 2018

    2018

  7. [16]

    MIT press Cambridge, 1998

    Richard S Sutton, Andrew G Barto, et al.Reinforcement learning: An introduction. MIT press Cambridge, 1998

    1998

  8. [17]

    Adaptiveε-greedy exploration in reinforcement learning based on value differences

    Michel Tokic. Adaptiveε-greedy exploration in reinforcement learning based on value differences. InAAAI, 2010

    2010

  9. [18]

    # exploration: A study of count-based exploration for deep reinforcement learning

    Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAI Xi Chen, Yan Duan, John Schulman, Filip DeTurck, and Pieter Abbeel. # exploration: A study of count-based exploration for deep reinforcement learning. In NeurIPS, 2017

    2017

  10. [19]

    Deep exploration via bootstrapped dqn

    Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. NeurIPS, 2016

    2016

  11. [20]

    Curious model-building control systems

    Jürgen Schmidhuber. Curious model-building control systems. InIJCNN, 1991

    1991

  12. [21]

    Curiosity-driven exploration by self-supervised prediction

    Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In ICML, 2017. 11

    2017

  13. [22]

    Exploration by random network distillation

    Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv:1810.12894, 2018

    2018 arXiv

  14. [23]

    Planning to explore via self-supervised world models

    Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via self-supervised world models. InICML, 2020

    2020

  15. [24]

    Vime: Variational information maximizing exploration

    Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration. InNeurIPS, 2016

    2016

  16. [25]

    Diversity is all you need: Learning skills without a reward function

    Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. InICLR, 2019

    2019

  17. [26]

    Ttrl: Test-time reinforcement learning

    Yuxin Zuo, Kaiyan Zhang, Li Sheng, Shang Qu, Ganqu Cui, Xuekai Zhu, Haozhan Li, Yuchen Zhang, Xinwei Long, Ermo Hua, Biqing Qi, Youbang Sun, Zhiyuan Ma, Lifan Yuan, Ning Ding, and Bowen Zhou. Ttrl: Test-time reinforcement learning. 2025

    2025

  18. [27]

    Learning to reason without external rewards

    Xuandong Zhao, Zhewei Kang, Aosong Feng, Sergey Levine, and Dawn Song. Learning to reason without external rewards. arXiv preprint arXiv:2505.19590, 2025

    2025 arXiv

  19. [28]

    Right question is already half the answer: Fully unsupervised llm reasoning incentivization.arXiv preprint arXiv:2504.05812, 2025

    Qingyang Zhang, Haitao Wu, Changqing Zhang, Peilin Zhao, and Yatao Bian. Right question is already half the answer: Fully unsupervised llm reasoning incentivization.arXiv preprint arXiv:2504.05812, 2025

    2025 arXiv

  20. [29]

    The unreasonable effectiveness of entropy minimization in llm reasoning.arXiv preprint arXiv:2505.15134, 2025

    Shivam Agarwal, Zimin Zhang, Lifan Yuan, Jiawei Han, and Hao Peng. The unreasonable effectiveness of entropy minimization in llm reasoning.arXiv preprint arXiv:2505.15134, 2025

    2025 arXiv

  21. [30]

    One-shot entropy minimization

    Zitian Gao, Lynx Chen, Joey Zhou, and Bryan Dai. One-shot entropy minimization. arXiv preprint arXiv:2505.20282, 2025

    2025 arXiv

  22. [31]

    Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for llm reasoning

    Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, Yuqiong Liu, An Yang, Andrew Zhao, Yang Yue, Shiji Song, Bowen Yu, Gao Huang, and Junyang Lin. Beyond the 80/20 rule: High-entropy minority tokens drive effe...

    2025 arXiv

  23. [32]

    First return, entropy-eliciting explore.arXiv preprint arXiv:2507.07017, 2025

    Tianyu Zheng, Tianshun Xing, Qingshui Gu, Taoran Liang, Xingwei Qu, Xin Zhou, Yizhi Li, Zhoufutu Wen, Chenghua Lin, Wenhao Huang, Qian Liu, Ge Zhang, and Zejun Ma. First return, entropy-eliciting explore.arXiv preprint arXiv:2507.07017, 2025

    2025 arXiv

  24. [33]

    Reasoning with exploration: An entropy perspective.arXiv preprint arXiv:2506.14758, 2025

    Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang, and Furu Wei. Reasoning with exploration: An entropy perspective.arXiv preprint arXiv:2506.14758, 2025

    2025 arXiv

  25. [34]

    Process reinforcement through implicit rewards.arXiv preprint arXiv:2502.01456, 2025

    Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards.arXiv preprint arXiv:2502.01456, 2025

    2025 arXiv

  26. [35]

    Free process rewards without process labels.arXiv preprint arXiv:2412.01981, 2024

    Lifan Yuan, Wendi Li, Huayu Chen, Ganqu Cui, Ning Ding, Kaiyan Zhang, Bowen Zhou, Zhiyuan Liu, and Hao Peng. Free process rewards without process labels.arXiv preprint arXiv:2412.01981, 2024

    2024 arXiv

  27. [36]

    Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv:2409.19256, 2024

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv:2409.19256, 2024

    2024 arXiv

  28. [37]

    Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476, 2025

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen...

    2025 arXiv

  29. [38]

    Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874, 2021

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874, 2021

    2021 arXiv

  30. [39]

    The entropy mechanism of reinforcement learning for reasoning language models.arXiv preprint arXiv:2505.22617, 2025

    Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, Zhiyuan Liu, Hao Peng, Lei Bai, Wanli Ouyang, Yu Cheng, Bowen Zhou, and Ning Ding. The entropy mechanism of reinforcement learning for reasoning language mo...

    2025 arXiv

  31. [40]

    Curriculum learning

    Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. InProceedings of the 26th annual international conference on machine learning, pages 41–48, 2009

    2009

  32. [41]

    Efficient memory management for large language model serving with pagedattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In SOSP, 2023

    2023

  33. [42]

    Math-Verify: Math Verification Library.https://github.com/huggingface/math-verify, 2025

    Hynek Kydlíček. Math-Verify: Math Verification Library.https://github.com/huggingface/math-verify, 2025. Version 0.6.1, License: Apache-2.0, Keywords: verification, math, evaluation. 13 Appendix A Additional Experimental Details A.1 Details of the Toy Model Experiment The moti...

    2025

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.