Pith. sign in

REVIEW 5 major objections 5 minor 3 cited by

No Free Lunch: Rethinking Internal Feedback for LLM Reasoning

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Reinforcement learning from internal feedback gives base LLMs a short accuracy boost, then the same entropy-minimizing pressure drives scores below the starting point.

desk verdict Useful empirical warning about RLIF for base LLMs, but the theory is borrowed and the headline 'matching RLVR' claim is unsubstantiated in the paper itself. read the letter →

arxiv 2506.17219 v2 pith:MURRQ7E7 submitted 2025-06-20 cs.LG cs.AI

Yanzhi Zhang , Zhaoxi Zhang , Haoxiang Guan , Yilin Cheng , Yitong Duan , Chen Wang , Yue Wang , Shuxin Zheng
show 1 more author
Jiyan He
This is my paper
classification cs.LGcs.AI
keywords reinforcementlearningfrominternalfeedbackpolicyentropyminimizationself-certaintyrewardLLMmathematicalreasoningoverconfidencetransitionalwordsRLIF
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Reinforcement learning from internal feedback (RLIF) replaces external correctness signals with the model's own confidence: self-certainty, token-level entropy, or trajectory-level entropy. The paper argues that all three rewards are partially equivalent because each drives down the policy's token-level entropy, and that this single mechanism explains a consistent training pattern: base LLM math accuracy rises for roughly the first 20 steps, then falls below the starting point, sometimes into collapse. Instruction-tuned models, which already have low policy entropy, gain little and simply degrade. The authors attribute the rise to suppressed underconfidence and better format compliance, and the fall to overconfidence and the disappearance of high-entropy transitional words that support multi-step reasoning. If correct, the work tells practitioners when unsupervised RLIF can substitute for verifiable-reward RL, and when it cannot.

What carries the argument

The load-bearing object is the dataset-averaged policy entropy, $H(\pi_\theta, \mathcal{D}) = -\mathbb{E}_{\mathcal{D}, \pi_\theta}[\log \pi_\theta(y_t \mid x, y_{<t})]$, together with the three reward proxies: self-certainty, the average KL divergence from a uniform distribution to the policy at each token; token-level entropy, the average negative Shannon entropy; and trajectory-level entropy, the normalized log-probability of the whole response. The theoretical engine is a lemma relating the one-step change in entropy to the covariance between the log-probability of an action and the parameter update, so that any reward making this covariance nonnegative drives entropy down; Propositions 1-3 show that each of the three rewards does exactly that under tabular softmax or natural-policy-gradient assumptions. The empirical engine is linear model merging between base and instruct weights, which produces models with a controlled range of initial policy entropies and lets the paper map entropy to trainability, and the notion of transitional words, high-entropy connectors such as 'wait', 'but', and 'suppose' whose declining frequency is used to trace the shift from underconfidence to overconfidence.

What would settle it

Run token-level-entropy RLIF on a base model for 80 steps while keeping the KL penalty fixed, and track GSM8K accuracy and policy entropy at every step; if accuracy keeps rising after step 20 while entropy keeps falling, or if the accuracy decline appears while entropy is flat, the claim that entropy minimization causes the collapse is wrong. As a second check, if the frequency of transitional words rises again or stays constant while accuracy falls, the overconfidence explanation is wrong.

Watch

Extended reading notes

Core claim

The paper's central claim is that RLIF is entropy minimization in disguise. Using token-level entropy as the reward is exactly equivalent to minimizing the model's expected policy entropy; under idealized tabular softmax policies, self-certainty and trajectory-level entropy also reduce policy entropy, via direct gradient updates and natural policy gradient updates respectively. Empirically, this entropy reduction first helps base models by removing hesitation and teaching them to produce clean, boxed answers within the length limit, but continued training suppresses the high-entropy transitional words that enable alternative reasoning paths, so accuracy peaks around step 20 and then declines below the untrained model. The same entropy logic explains why instruction-tuned models benefit little: their initial policy entropy is already low, as confirmed by model-merging experiments that interpolate between base and instruct weights and show gains shrink as the merge ratio rises. The paper concludes that the initial policy entropy of the model is the decisive factor for whether internal feedback helps, and that managing entropy is the key to avoiding collapse.

Load-bearing premise

The whole rise-then-fall explanation depends on the idealized tabular-softmax entropy theorems continuing to describe what actually happens to real LLMs trained with the optimizer and KL penalty used in large-scale practice; if real updates do not behave like the idealized ones, the proposed mechanism is unsupported.

Editorial extensions

If this is right

  • If the entropy-collapse account is right, RLIF should be scheduled as a short, early intervention; extending it beyond the peak step will undo the gains and can push accuracy below the untrained model.
  • Initial policy entropy, measured before training, should be the primary screening test for whether RLIF will help a given model, with high-entropy base models as candidates and low-entropy instruction-tuned models as non-candidates.
  • Transitional-word frequency can serve as an early warning signal: once it falls below the level observed at the accuracy peak, further training should be expected to hurt.
  • For instruction-tuned models, internal-feedback rewards are likely to be net harmful from the first step, so practitioners should not expect unsupervised intrinsic rewards to replace verifiable rewards on such models.
  • Because the three rewards are partially equivalent, implementation cost and stability can guide the choice among self-certainty, token-level, and trajectory-level entropy without expecting fundamentally different performance trajectories.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: the entropy-collapse mechanism suggests a direct intervention; rewarding or constraining the presence of a minimal set of transitional words could extend the early benefit beyond step 20 and is testable without any external labels.
  • Inference: if the mechanism is generic, the same rise-then-fall curve should appear on other free-form generation tasks such as code or open-ended question answering, not only math, because any sustained multi-step deliberation would suffer the same underconfidence-to-overconfidence shift.
  • Inference: the idealized equivalence among the three rewards may not hold under real optimizers with gradient clipping and a KL penalty, so token-level entropy and self-certainty could diverge in their exploration behavior at longer horizons even though their entropy effects look identical in the tabular analysis.
  • Inference: the model-merging results suggest that initial entropy is not the only relevant variable; in-domain knowledge may interact with entropy, so a low-entropy model with strong math-specific knowledge, like the Qwen2.5-1.5B-Math case, could keep improving even as entropy falls.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper studies Reinforcement Learning from Internal Feedback (RLIF) for LLM math reasoning, comparing three intrinsic reward signals: self-certainty, token-level entropy, and trajectory-level entropy. It claims theoretically that these signals are partially equivalent and all reduce policy entropy (Propositions 1-3), and empirically that RLIF improves base LLMs early in training, then degrades performance, while instruction-tuned models benefit little. The paper attributes this behavior to entropy minimization: early gains come from suppressing underconfidence and improving instruction following, while later degradation comes from overconfidence and reduced transitional-word usage. It also uses model merging to support the claim that initial policy entropy predicts RLIF effectiveness.

Significance. If the empirical patterns and theoretical mechanism were fully supported, the paper would make a useful contribution to the largely unsupervised RLIF literature, providing practical guidance on when internal rewards help or hurt. The strengths include transparent reporting of raw performance in Table 4, explicit acknowledgment of training collapse in several configurations, and concrete qualitative examples of underconfidence, balanced confidence, and overconfidence in Appendix B.5. The entropy-based explanation, though plausible, is not adequately established by the current theoretical analysis or the experimental design.

major comments (5)
  1. [Abstract and Section 4] The abstract claims RLIF can match or surpass RLVR techniques, but the paper contains no RLVR baseline. Table 4 reports only RLIF runs; there is no comparison with GRPO using verifiable rewards under the same setup, model, and step budget. This is a load-bearing claim in the abstract and must either be backed by a direct baseline or removed in favor of a more limited claim.
  2. [Section 3.4 and Appendix A.5-A.6] Propositions 2 and 3, which drive the paper's central mechanism, rely on Lemma 1 borrowed from Liu (2025) and Cui et al. (2025) and are proven only for tabular softmax policies updated by exact gradient or natural policy gradient. The experiments use VeRL/GRPO with a KL penalty (beta = 0.005), Adam optimization, and transformer policies, where rewards are recomputed under the current policy. None of these idealized assumptions hold in the empirical setup, so the theoretical result does not establish that the actual RLIF updates reduce entropy. The paper should either provide a transfer argument to the GRPO/Adam setting or explicitly reframe Propositions 2-3 as illustrative rather than as the explanation of the observed training dynamics.
  3. [Table 2 and Section 4.3] The totals in Table 2 are internally inconsistent, and the surrounding text reads the table incorrectly. For step 40, the columns give RA/WF = 0, WA/WF = 2, WA/RF = 233, and RA/RF = 261, so the right-answer total is 261 and the wrong-answer total is 235. The table's printed 'RA' and 'WA' columns instead show 235 and 261, reversing these totals. The text then says the number of right answers decreases from 291 to 235, which uses the reversed value. This error undermines the claim that reasoning capability degrades sharply at step 40 and must be corrected.
  4. [Table 4 and Section 4.2] The general claim that performance 'degrades even below the model before training' is not supported by the full results. For example, Qwen2.5-3B with self-certainty reaches 0.670 on GSM8K at step 80 versus 0.649 at step 0, and Qwen2.5-1.5B-Math improves on GSM8K from 0.265 to 0.779-0.830 at step 80 without dropping below the initial value. Several cells also show tiny changes (e.g., AIME2025 values of 0.0 to 0.038) that may be noise. The claim should be qualified to the specific model/method/dataset combinations that actually exhibit a below-start decline.
  5. [Section 4.2 and Figure 2] The 'early phase before 20 steps' characterization appears to be a post hoc division of the training curve, and no error bars or multiple-seed results are reported. Many improvements in Table 4 are small (a few accuracy points or less), and some differences are within plausible noise ranges. To support the claim that base models truly improve before step 20 and then degrade, the paper should provide variance estimates across seeds or otherwise justify that the observed differences are statistically meaningful.
minor comments (5)
  1. [Table 4 caption] The caption contains a typo: 'vaious benckmarks' should read 'various benchmarks'.
  2. [Figure 2 caption] The caption says the plotted values are accuracy improvements relative to the base model, but the y-axis is labeled 'Average Score (%)' and includes both positive and negative values. Please clarify whether the axis shows absolute scores or deltas.
  3. [Section 4.2] The text refers to 'Qwen2.5-Math' without specifying the exact model; from Table 4 this appears to be Qwen2.5-1.5B-Math. Please use consistent naming.
  4. [References] The reference list contains duplicate entries for Zhao et al. (2025), with slightly different arXiv identifiers, and a non-archival source for Lemma 1 (a Zhihu article, Liu 2025) is cited without a peer-reviewed or archival alternative. Please consolidate and verify all citations.
  5. [Section 3.4] The sentence 'From both theoretical and experimental perspectives, it is straightforward to demonstrate that RLIF directly leads to a reduction in policy entropy' is stronger than what the subsequent propositions establish, since the propositions assume idealized settings. Please soften the wording to match the actual scope of the proofs.

Circularity Check

1 steps flagged · score 2.0 of 10

One definitional theorem (Proposition 1) reduces to the reward definition; the empirical findings are otherwise held-out and not circular.

  1. self definitional [Section 3.4, Proposition 1; Eq. (3); Appendix A.4]
    "Proposition 1. Using r token-entropy(x, y) = − 1/|y| ∑^{|y|}_{t=1} H(πθ(·|x, y<t)) as the reward function in the LLM RL setting objective: arg maxθ Ex∼D,y∼πθ (·|x)[r(x, y)] is equivalent to minimizing the policy entropy."

    The reward r_token-entropy is defined in Eq. (3) as the negative average token-level policy entropy. Therefore, maximizing its expectation is identical to minimizing policy entropy by construction. The Appendix A.4 proof simply states that the expected reward equals −H(πθ,D), which is exactly a restatement of the definition. This is a definitional equivalence presented as a proposition, rather than a derived consequence.

full rationale

The paper's formal claims for token-level entropy (Proposition 1) are true by definition: the reward is the negative policy entropy, so optimizing it is entropy minimization. This is a minor self-definitional step, but it does not contaminate the empirical results, which are measured on held-out benchmarks (AIME2025, MATH500, GSM8K) and compared against the base model. Propositions 2 and 3 rely on standard tabular-softmax and natural-policy-gradient lemmas borrowed from external works (Agarwal et al. 2021; Liu 2025; Cui et al. 2025), not on fitted parameters or self-citations from this paper. The model-merging experiments test the 'initial entropy determines improvement' hypothesis on independently constructed interpolated models, so the central empirical claim is not circular. The transitional-word analysis is post hoc and correlational, but it is not used to fit a parameter or to define the success metric. Overall, aside from the definitional equivalence in Proposition 1, the derivation chain is self-contained and the empirical findings stand on their own.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No new theoretical entities are introduced; the 'transitional words' list is an empirical annotation. The central empirical claim is largely independent of fitted parameters; the explanatory theory, however, leans on unproved lemmas and idealized approximations.

assumptions (3)
  • domain assumption Lemma 1 (entropy change under parameter update for tabular softmax policy) is assumed from Liu (2025) and Cui et al. (2025); not proven in this paper.
    Propositions 2 and 3 rely on this lemma to connect reward-based parameter updates to entropy reduction. The lemma is cited, not derived, and only holds approximately under a first-order tabular-softmax idealization.
  • domain assumption The tabular softmax / natural policy gradient idealization transfers to real LLM training with Adam, KL penalty, and non-tabular policies.
    The theoretical claims that RLIF reduces policy entropy are proven only in idealized settings; the paper applies them to explain empirical behavior on Qwen models without verifying the idealizations hold.
  • domain assumption The 14-word transitional word list (But, However, Wait, etc.) is a valid proxy for reasoning depth and its decrease causes the performance decline.
    The paper attributes late-training degradation to overconfidence arising from reduced transitional word frequency, but provides no causal test linking these words to reasoning quality beyond correlation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of No Free Lunch: Rethinking Internal Feedback for LLM Reasoning." pith.science (2026). https://pith.science/paper/MURRQ7E7

@misc{pith2026250617219,
  author       = {Pith},
  title        = {Pith review of: No Free Lunch: Rethinking Internal Feedback for LLM Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MURRQ7E7}},
  note         = {Machine review of arXiv:2506.17219}
}
read the original abstract

Reinforcement learning has emerged as a powerful paradigm for post-training large language models (LLMs) to improve reasoning. Approaches like Reinforcement Learning from Human Feedback (RLHF) and Reinforcement Learning with Verifiable Rewards (RLVR) have shown strong results, but they require extensive external supervision. We investigate an alternative class of methods, Reinforcement Learning from Internal Feedback (RLIF), which relies solely on intrinsic model-derived signals instead of external rewards. In particular, we leverage unsupervised reward proxies such as token-level entropy, trajectory-level entropy, and self-certainty. Our theoretical analysis shows these internal objectives are partially equivalent, and we empirically evaluate various RLIF strategies on challenging math reasoning benchmarks. Experimental results demonstrate that RLIF can boost the reasoning performance of base LLMs at the beginning phase of the training, matching or surpassing RLVR techniques on these tasks. However, when training progresses, performance degrades even below the model before training. Moreover, we find that RLIF yields little improvement for instruction-tuned models, indicating diminishing returns of intrinsic feedback once an LLM is already instruction-tuned. We further analyze this limitation by mixing model weights and explain the reason of RLIF's training behaviors, providing practical guidelines for integrating internal feedback signals into LLM training. We hope our analysis of internal feedback will inform more principled and effective strategies for LLM post-training.

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Breaking the Self-Confirming Loop: Diagnosing and Mitigating Systemic Reward Bias in Self-Rewarding RL

    cs.LG 2025-10 conditional novelty 6.0 of 10

    Self-rewarding RL can be stabilized by ensembling multiple policy models' majority-vote rewards, reaching within 3.6% of verifiable-reward RL on math benchmarks.

  2. Towards Agents That Know When They Don't Know: Uncertainty as a Control Signal for Structured Reasoning

    cs.AI 2025-09 conditional novelty 6.0 of 10

    An LLM agent using retrieval and summary uncertainty as training rewards and inference filters produces more factual, useful multi-omics summaries and better downstream survival predictions.

  3. Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle

    cs.CL 2025-09 conditional novelty 3.0 of 10

    A survey that maps reinforcement learning methods, datasets, benchmarks, and open-source tools across the full training lifecycle of large language models, focusing on verifiable-reward reasoning.

Reference graph

Works this paper leans on

44 extracted references · 7 canonical work pages · cited by 3 Pith papers

  1. [1]

    Agarwal, S

    A. Agarwal, S. M. Kakade, J. D. Lee, and G. Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research, 22 0 (98): 0 1--76, 2021. URL http://jmlr.org/papers/v22/19-736.html

    work page 2021

  2. [2]

    Agarwal, Z

    S. Agarwal, Z. Zhang, L. Yuan, J. Han, and H. Peng. The unreasonable effectiveness of entropy minimization in llm reasoning, 2025

    work page 2025

  3. [3]

    Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022

    arXiv 2022

  4. [4]

    Balunović, J

    M. Balunović, J. Dekoninck, I. Petrov, N. Jovanović, and M. Vechev. Matharena: Evaluating llms on uncontaminated math competitions, Feb. 2025. URL https://matharena.ai/

    work page 2025

  5. [5]

    Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, M. Zhong, Q. Zhang, X. Zhu, L. Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 24185--24198, 2024

    work page 2024

  6. [6]

    Cobbe, V

    K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv 2021

  7. [7]

    G. Cui, Y. Zhang, J. Chen, L. Yuan, Z. Wang, Y. Zuo, H. Li, Y. Fan, H. Chen, W. Chen, Z. Liu, H. Peng, L. Bai, W. Ouyang, Y. Cheng, B. Zhou, and N. Ding. The entropy mechanism of reinforcement learning for reasoning language models, 2025. URL https://arxiv.org/abs/2505.22617

    arXiv 2025

  8. [8]

    Z. Gao, L. Chen, J. Zhou, and B. Dai. One-shot entropy minimization, 2025. URL https://arxiv.org/abs/2505.20282

    arXiv 2025

Show all 44 references
  1. [9]

    T. GLM, A. Zeng, B. Xu, B. Wang, C. Zhang, D. Yin, D. Zhang, D. Rojas, G. Feng, H. Zhao, et al. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793, 2024

    2024 arXiv

  2. [11]

    Hendrycks, C

    D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt. Measuring mathematical problem solving with the math dataset, 2021. URL https://arxiv.org/abs/2103.03874

    2021 arXiv

  3. [12]

    J. Hu. Reinforce++: A simple and efficient approach for aligning large language models. arXiv preprint arXiv:2501.03262, 2025

    2025 arXiv

  4. [13]

    Y. Ji, X. Tian, S. Zhao, H. Wang, S. Chen, Y. Peng, H. Zhao, and X. Li. Am-thinking-v1: Advancing the frontier of reasoning at 32b scale. arXiv preprint arXiv:2505.08311, 2025

    2025 arXiv

  5. [14]

    S. M. Kakade. A natural policy gradient. Advances in neural information processing systems, 2001

    2001

  6. [15]

    P. Li, M. Skripkin, A. Zubrey, A. Kuznetsov, and I. Oseledets. Confidence is all you need: Few-shot rl fine-tuning of language models, 2025. URL https://arxiv.org/abs/2506.06395

    2025 arXiv

  7. [16]

    Lightman, V

    H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023

    2023

  8. [17]

    A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024 a

    2024 arXiv

  9. [18]

    F. Liu, N. AlDahoul, G. Eady, Y. Zaki, and T. Rahwan. Self‑reflection makes large language models safer, less biased, and ideologically neutral. arXiv preprint arXiv:2406.10400, 2024 b

    2024 arXiv

  10. [19]

    J. Liu. How does rl policy entropy converge during iteration? https://zhuanlan.zhihu.com/p/28476703733, 2025. URL https://zhuanlan.zhihu.com/p/28476703733

    2025

  11. [20]

    Learning to reason with llms

    OpenAI . Learning to reason with llms. https://openai.com/index/learning-to-reason-with-llms/, 2024. Accessed: 2025-06-01

    2024

  12. [21]

    Petrov, Y

    A. Petrov, Y. Zhang, and N. Ahmed. Proof or bluff? evaluating llms on 2025 usa math olympiad. arXiv preprint arXiv:2503.21934, 2025

    2025 arXiv

  13. [22]

    Schulman, F

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  14. [23]

    Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  15. [24]

    Sheng, C

    G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, R. Zhang, Y. Peng, H. Lin, and C. Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19256, 2024

    2024 arXiv

  16. [25]

    Shojaee, I

    P. Shojaee, I. Mirzadeh, K. Alizadeh, M. Horton, S. Bengio, and M. Farajtabar. The illusion of thinking: Understanding the strengths and limitations of reasoning models via the lens of problem complexity. arXiv preprint arXiv:2506.06941, 2025

    2025 arXiv

  17. [26]

    X. Su, S. Xie, G. Liu, Y. Xia, R. Luo, P. Jin, Z. Ma, Y. Wang, Z. Wang, and Y. Liu. Trust region preference approximation: A simple and stable reinforcement learning algorithm for llm reasoning. arXiv preprint arXiv:2504.04524, 2025

    2025 arXiv

  18. [27]

    K. Team, A. Du, B. Gao, B. Xing, C. Jiang, C. Chen, C. Li, C. Xiao, C. Du, C. Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599, 2025

    2025 arXiv

  19. [28]

    Q. Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025. URL https://qwenlm.github.io/blog/qwq-32b/

    2025

  20. [29]

    Touvron, T

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi \`e re, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  21. [30]

    S. Wang, L. Yu, C. Gao, C. Zheng, S. Liu, R. Lu, K. Dang, X. Chen, J. Yang, Z. Zhang, et al. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for llm reasoning. arXiv preprint arXiv:2506.01939, 2025

    2025 arXiv

  22. [31]

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35: 0 24824--24837, 2022

    2022

  23. [32]

    F. Xu, Q. Hao, Z. Zong, J. Wang, Y. Zhang, J. Wang, X. Lan, J. Gong, T. Ouyang, F. Meng, C. Shao, Y. Yan, Q. Yang, Y. Song, S. Ren, X. Hu, Y. Li, J. Feng, C. Gao, and Y. Li. Towards large reasoning models: A survey of reinforced reasoning with large language models. arXiv prep...

    2025 arXiv

  24. [33]

    A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Xia, X. Ren, X. Ren, Y....

    2024 arXiv

  25. [34]

    A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025

    2025 arXiv

  26. [35]

    E. Yang, L. Shen, G. Guo, X. Wang, X. Cao, J. Zhang, and D. Tao. Model merging in llms, mllms, and beyond: Methods, theories, applications and opportunities, 2024 b . URL https://arxiv.org/abs/2408.07666

    2024 arXiv

  27. [36]

    Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, T. Fan, G. Liu, L. Liu, X. Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025

    2025 arXiv

  28. [37]

    Y. Yue, Z. Chen, R. Lu, A. Zhao, Z. Wang, Y. Yue, S. Song, and G. Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?, 2025 a . URL https://arxiv.org/abs/2504.13837

    2025 arXiv

  29. [38]

    Y. Yue, Y. Yuan, Q. Yu, X. Zuo, R. Zhu, W. Xu, J. Chen, C. Wang, T. Fan, Z. Du, et al. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks. arXiv preprint arXiv:2504.05118, 2025 b

    2025 arXiv

  30. [39]

    Zhang, H

    Q. Zhang, H. Wu, C. Zhang, P. Zhao, and Y. Bian. Right question is already half the answer: Fully unsupervised llm reasoning incentivization. arXiv preprint arXiv:2504.05812, 2025

    2025 arXiv

  31. [40]

    Zhao , Z

    X. Zhao , Z. Kang , A. Feng , S. Levine , and D. Song . Learning to Reason without External Rewards . arXiv e-prints, art. arXiv:2505.19590, May 2025. doi:10.48550/arXiv.2505.19590

    2025 doi arXiv

  32. [41]

    X. Zhao, Z. Kang, A. Feng, S. Levine, and D. Song. Learning to reason without external rewards. arXiv preprint arXiv:2505.19590, 2025

    2025 arXiv

  33. [42]

    Y. Zuo, K. Zhang, L. Sheng, S. Qu, G. Cui, X. Zhu, H. Li, Y. Zhang, X. Long, E. Hua, et al. Ttrl: Test-time reinforcement learning. arXiv preprint arXiv:2504.16084, 2025

    2025 arXiv

  34. [43]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  35. [44]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  36. [45]

    Learning to Reason without External Rewards

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2007 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.