pith. sign in

arxiv: 2506.08125 · v4 · submitted 2025-06-09 · 💻 cs.LG · cs.CL

Not All Tokens Matter: Towards Efficient LLM Reasoning via Token Significance in Reinforcement Learning

Hanbing Liu , Lang Cao , Yuanyi Ren , Mengyu Zhou , Haoyu Dong , Xiaojun Ma , Shi Han , Dongmei Zhang This is my paper

Pith reviewed 2026-05-19 10:02 UTC · model grok-4.3

classification 💻 cs.LG cs.CL
keywords LLM reasoningreinforcement learningtoken significancechain-of-thoughtlength optimizationpolicy optimizationefficient inference
0
0 comments X

The pith

Penalizing only low-significance tokens in RL training lets LLMs shorten reasoning chains while preserving or raising accuracy.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that uniform length penalties in reinforcement learning for LLMs waste effort on chain-of-thought tokens that add little to the final answer. By estimating which tokens matter and applying penalties selectively to the rest, plus a dynamic reward that loosens early and tightens later, the method trims output length. A sympathetic reader would care because current RL-tuned models often generate verbose explanations that raise compute cost without improving results. The experiments report shorter responses across benchmarks with no loss in correctness and sometimes gains.

Core claim

Observing that many chain-of-thought tokens contribute little to the final answer, the work introduces a significance-aware length reward that selectively penalizes insignificant tokens and a dynamic length reward that encourages detail early then shifts toward conciseness. When these are added to standard policy optimization, both reasoning efficiency and accuracy improve.

What carries the argument

Significance-aware length reward that estimates each token's contribution to the final answer and applies penalties only to low-contribution tokens.

If this is right

  • Response lengths drop substantially on standard reasoning benchmarks.
  • Correctness is preserved or improved compared with uniform length rewards.
  • Essential reasoning steps remain intact while redundant tokens are reduced.
  • The dynamic reward produces a gradual shift from detailed to concise outputs over training.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same token-level view could apply to other RLHF objectives beyond length, such as factual consistency.
  • Pre-computing rough significance scores from a smaller model might approximate the method at lower cost.
  • The findings suggest that sequence-level rewards in generative RL are often too coarse and that finer token analysis may be broadly useful.

Load-bearing premise

Token significance can be estimated reliably enough during training to guide penalties without removing steps needed for a correct answer.

What would settle it

Replace the significance estimate with random token selection in the reward and measure whether response accuracy falls more than under the proposed method on the same benchmarks.

Figures

Figures reproduced from arXiv: 2506.08125 by Dongmei Zhang, Hanbing Liu, Haoyu Dong, Lang Cao, Mengyu Zhou, Shi Han, Xiaojun Ma, Yuanyi Ren.

Figure 1
Figure 1. Figure 1: Performance overview of BINGO and other baselines. Left: Scatter plot of average accuracy versus response length for various methods. Points nearer the top-right corner represent a better balance of accuracy and efficiency. Right: Radar chart of length-normalized accuracy for each method. Greater radial distances denote higher efficiency. 1 Introduction Large language models (LLMs) [1, 2] have demonstrated… view at source ↗
Figure 2
Figure 2. Figure 2: Illustration of the BINGO framework. Given a generated CoT trace, the LLM first distinguishes between significant and insignificant tokens. A dynamic length reward is then computed based on token type and sample correctness. During the early exploration phase of training (k(t) ≥ β), the reward encourages extended reasoning for significant tokens in incorrect samples while penalizing insignificant tokens in… view at source ↗
Figure 3
Figure 3. Figure 3: Significant Length Ratio dynamics during training. The x-axis indicates training steps, and the y-axis denotes the proportion of significant tokens in the generated responses. Each subplot corresponds to one benchmark. The blue curve represents the baseline method (Vanilla PPO), and the red curve represents our approach (Ours). Existing baselines struggle with the trade-off between accuracy and brevity: Ap… view at source ↗
Figure 4
Figure 4. Figure 4: Penalty curve: q 1 − L Lmax . To evaluate reasoning efficiency, we adopt a length￾normalized accuracy metric, denoted as L-ACC, which balances correctness with brevity. Formally, it is defined as: L-Acc = Acc × r 1 − L Lmax , (45) where Acc ∈ [0, 1] denotes exact-match accuracy, L is the number of tokens in the model’s response, and Lmax is a dataset-specific upper bound on response length. The multiplicat… view at source ↗
Figure 5
Figure 5. Figure 5: Length–accuracy results for nine optimization algorithms on four datasets. Bars show the number of tokens generated at the checkpoint that yields the reported accuracy (left axis). Each bar is partitioned into significant (dark) and insignificant (light) segments, and the percentage above the bar indicates the share of significant tokens. The solid line (right axis) gives the corresponding answer accuracy.… view at source ↗
Figure 6
Figure 6. Figure 6: Token-level significance visualization for a sample reasoning task. Each token is colored based on its predicted significance: red indicates significant tokens (darker = more significant), and blue indicates insignificant tokens (darker = less significant). The response from BINGO (top) is shorter and more concentrated around meaningful reasoning steps, while the Vanilla PPO response (bottom) is longer and… view at source ↗
Figure 7
Figure 7. Figure 7: Response length trends during training across four datasets. The y-axis shows the number of tokens generated per response; the x-axis denotes training steps. The red line represents our method, and the blue line corresponds to Vanilla PPO. Across all tasks, our method consistently produces shorter and more stable responses, demonstrating improved reasoning efficiency without compromising task performance. … view at source ↗
Figure 8
Figure 8. Figure 8: Response length dynamics for correct vs. wrong samples during training. The x-axis indicates training steps, and the y-axis denotes response length in tokens. The blue line tracks correctly answered samples, while the yellow line tracks incorrectly answered samples. In the early stages, incorrect samples produce substantially longer responses, reflecting the effect of our length-incentive mechanism. As the… view at source ↗
Figure 9
Figure 9. Figure 9: Distribution of response lengths for correct vs. incorrect samples. Histograms show the frequency of token lengths in model outputs across four benchmarks. Each plot compares correct responses (blue) and incorrect responses (orange). The top row corresponds to our method, while the bottom row shows results from Vanilla PPO. Across all datasets, incorrect samples are more likely to produce longer outputs, w… view at source ↗
Figure 10
Figure 10. Figure 10: Effect of Reward Design on Incorrect Response Length. We visualize the average significant response length of incorrect predictions during training on four benchmarks. Compared to the variant without incentive, our full method produces longer responses for incorrect samples, suggesting that the significance-aware reward encourages more thorough exploration when the model is uncertain. In contrast, removin… view at source ↗
Figure 11
Figure 11. Figure 11: The figure shows a case study with three settings: base, PPO, and Bingo under the [PITH_FULL_IMAGE:figures/full_fig_p029_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: The figure shows a case study with three settings: base, PPO, and Bingo under the [PITH_FULL_IMAGE:figures/full_fig_p030_12.png] view at source ↗
read the original abstract

Large language models (LLMs) show strong reasoning abilities but often produce unnecessarily long explanations that reduce efficiency. Although reinforcement learning (RL) has been used to improve reasoning, most methods focus on accuracy and rely on uniform length-based rewards that overlook the differing contributions of individual tokens, often harming correctness. We revisit length optimization in RL through the perspective of token significance. Observing that many chain-of-thought (CoT) tokens contribute little to the final answer, we introduce a significance-aware length reward that selectively penalizes insignificance tokens, reducing redundancy while preserving essential reasoning. We also propose a dynamic length reward that encourages more detailed reasoning early in training and gradually shifts toward conciseness as learning progresses. Integrating these components into standard policy optimization yields a framework that improves both reasoning efficiency and accuracy. Experiments across multiple benchmarks demonstrate substantial reductions in response length while preserving or improving correctness, highlighting the importance of modeling token significance for efficient LLM reasoning.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper proposes integrating token significance estimation into RL-based length optimization for LLM reasoning. It introduces a significance-aware length reward that selectively penalizes low-significance CoT tokens and a dynamic length reward that shifts from encouraging detail early in training to conciseness later. These are combined with standard policy optimization to reduce response length while preserving or improving accuracy on reasoning benchmarks.

Significance. If the significance estimator reliably identifies redundant tokens without circular dependence on the policy, the framework could improve both efficiency and correctness over uniform length penalties, offering a practical advance in RL for long-form reasoning. The dynamic reward component is a notable design choice that addresses training dynamics.

major comments (2)
  1. [§3.2] §3.2 (significance estimation): The method for computing token significance is not externally validated against counterfactual answer changes or human-annotated essential steps. Without such a check (e.g., ablation removing high-significance tokens and measuring answer degradation), it is unclear whether the estimator captures causal contribution or merely correlates with the current policy's down-weighted tokens, risking circularity in the reward signal.
  2. [§4.1] §4.1 (experimental setup): The reported length reductions and accuracy gains lack error bars across multiple random seeds and do not include an ablation isolating the significance-aware reward from the dynamic reward. This makes it difficult to attribute improvements specifically to token significance modeling.
minor comments (2)
  1. Notation for the significance score s_t and its integration into the reward r_t should be defined explicitly in the main text rather than deferred to the appendix.
  2. Figure 2 (length vs. accuracy curves): Axis labels and legend entries are too small for readability; consider increasing font size or splitting into separate panels.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We appreciate the referee's insightful comments on our manuscript. We address each major comment below and plan to incorporate revisions to strengthen the presentation and validation of our proposed methods.

read point-by-point responses

  1. Referee: [§3.2] §3.2 (significance estimation): The method for computing token significance is not externally validated against counterfactual answer changes or human-annotated essential steps. Without such a check (e.g., ablation removing high-significance tokens and measuring answer degradation), it is unclear whether the estimator captures causal contribution or merely correlates with the current policy's down-weighted tokens, risking circularity in the reward signal.

    Authors: We thank the referee for this valuable feedback. We acknowledge that additional validation would help confirm the reliability of the token significance estimator. In the revised manuscript, we will include a new ablation study that removes high-significance tokens from the chain-of-thought and measures the degradation in reasoning accuracy. This will provide evidence of their causal contribution. We will also clarify in §3.2 how the significance is estimated to mitigate concerns about circular dependence on the policy. revision: yes

  2. Referee: [§4.1] §4.1 (experimental setup): The reported length reductions and accuracy gains lack error bars across multiple random seeds and do not include an ablation isolating the significance-aware reward from the dynamic reward. This makes it difficult to attribute improvements specifically to token significance modeling.

    Authors: We agree that reporting variability and isolating components is important for rigorous evaluation. In the revised version, we will rerun the experiments with multiple random seeds and report mean results with standard deviations (error bars). Additionally, we will add an ablation study that compares the full method against variants using only the significance-aware reward and only the dynamic length reward to better attribute the contributions of each component. revision: yes

Circularity Check

0 steps flagged

No significant circularity; empirical claims rest on external benchmarks

full rationale

The abstract and available description frame the contribution as an empirical RL framework that introduces significance-aware and dynamic length rewards, with results validated across multiple benchmarks showing reduced response length while preserving or improving correctness. No equations, self-citations, or derivation steps are visible that reduce a claimed prediction or uniqueness result to a fitted input or prior author work by construction. The central premise relies on experimental outcomes rather than an internal loop, satisfying the condition for a self-contained paper against external validation.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Review performed on abstract only; no explicit free parameters, axioms, or invented entities are stated in the provided text.

pith-pipeline@v0.9.0 · 5710 in / 992 out tokens · 21694 ms · 2026-05-19T10:02:36.710330+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Post Reasoning: Improving the Performance of Non-Thinking Models at No Cost

    cs.AI 2026-05 conditional novelty 7.0

    Post-Reasoning boosts LLM accuracy by reversing the usual answer-after-reasoning order, delivering mean relative gains of 17.37% across 117 model-benchmark pairs with zero extra cost.

  2. Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models

    cs.CL 2025-03 accept novelty 5.0

    A survey organizing techniques to achieve efficient reasoning in LLMs by shortening chain-of-thought outputs.

Reference graph

Works this paper leans on

56 extracted references · 56 canonical work pages · cited by 2 Pith papers · 7 internal anchors

  1. [1]

    Gpt-4 technical report, 2024

    OpenAI. Gpt-4 technical report, 2024

    work page 2024

  2. [2]

    Textbooks are all you need, 2023

    Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tau- man Kalai, Yin Tat Lee, and Yuanzhi Li. Textbooks are all you need, 2023

    work page 2023

  3. [3]

    Solving math word problems with process- and outcome-based feedback, 2022

    Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process- and outcome-based feedback, 2022

    work page 2022

  4. [4]

    Measuring Mathematical Problem Solving With the MATH Dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  5. [5]

    Aime problem set 1983-2024, 2023

    Hemish Veeraboina. Aime problem set 1983-2024, 2023

    work page 1983

  6. [6]

    Theoremqa: A theorem-driven question answering dataset

    Wenhu Chen, Ming Yin, Max Ku, Pan Lu, Yixin Wan, Xueguang Ma, Jianyu Xu, Xinyi Wang, and Tony Xia. Theoremqa: A theorem-driven question answering dataset. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7889–7901, Singapore, 2023. Association for Computational Linguistics

    work page 2023

  7. [7]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022

    work page 2022

  8. [8]

    Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models, 2022

    work page 2022

  9. [9]

    Le, Ed H

    Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V . Le, Ed H. Chi, Denny Zhou, and Jason Wei. Challeng- ing big-bench tasks and whether chain-of-thought can solve them, 2022

    work page 2022

  10. [10]

    A survey of efficient reasoning for large reasoning models: Language, multimodality, and beyond, 2025

    Xiaoye Qu, Yafu Li, Zhaochen Su, Weigao Sun, Jianhao Yan, Dongrui Liu, Ganqu Cui, Daizong Liu, Shuxian Liang, Junxian He, Peng Li, Wei Wei, Jing Shao, Chaochao Lu, Yue Zhang, Xian- Sheng Hua, Bowen Zhou, and Yu Cheng. A survey of efficient reasoning for large reasoning models: Language, multimodality, and beyond, 2025

    work page 2025

  11. [11]

    Stop overthinking: A survey on efficient reasoning for large language models, 2025

    Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen Zhong, Hanjie Chen, and Xia Hu. Stop overthinking: A survey on efficient reasoning for large language models, 2025

    work page 2025

  12. [12]

    From system 1 to system 2: A survey of reasoning large language models, 2025

    Zhong-Zhi Li, Duzhen Zhang, Ming-Liang Zhang, Jiaxin Zhang, Zengyan Liu, Yuxuan Yao, Haotian Xu, Junhao Zheng, Pei-Jie Wang, Xiuyi Chen, Yingying Zhang, Fei Yin, Jiahua Dong, Zhijiang Guo, Le Song, and Cheng-Lin Liu. From system 1 to system 2: A survey of reasoning large language models, 2025

    work page 2025

  13. [13]

    Harnessing the reasoning economy: A survey of efficient reasoning for large language models, 2025

    Rui Wang, Hongru Wang, Boyang Xue, Jianhui Pang, Shudong Liu, Yi Chen, Jiahao Qiu, Derek Fai Wong, Heng Ji, and Kam-Fai Wong. Harnessing the reasoning economy: A survey of efficient reasoning for large language models, 2025. 10

    work page 2025

  14. [14]

    Tokenskip: Controllable chain-of-thought compression in llms, 2025

    Heming Xia, Yongqi Li, Chak Tou Leong, Wenjie Wang, and Wenjie Li. Tokenskip: Controllable chain-of-thought compression in llms, 2025

    work page 2025

  15. [15]

    Twt: Thinking without tokens by habitual reasoning distillation with multi-teachers’ guidance, 2025

    Jingxian Xu, Mengyu Zhou, Weichang Liu, Hanbing Liu, Shi Han, and Dongmei Zhang. Twt: Thinking without tokens by habitual reasoning distillation with multi-teachers’ guidance, 2025

    work page 2025

  16. [16]

    Lightthinker: Thinking step-by-step compression, 2025

    Jintian Zhang, Yuqi Zhu, Mengshu Sun, Yujie Luo, Shuofei Qiao, Lun Du, Da Zheng, Huajun Chen, and Ningyu Zhang. Lightthinker: Thinking step-by-step compression, 2025

    work page 2025

  17. [17]

    C3ot: Generating shorter chain-of- thought without compromising effectiveness, 2024

    Yu Kang, Xianghui Sun, Liangyu Chen, and Wei Zou. C3ot: Generating shorter chain-of- thought without compromising effectiveness, 2024

    work page 2024

  18. [18]

    O1-pruner: Length-harmonizing fine-tuning for o1-like reasoning pruning, 2025

    Haotian Luo, Li Shen, Haiying He, Yibo Wang, Shiwei Liu, Wei Li, Naiqiang Tan, Xiaochun Cao, and Dacheng Tao. O1-pruner: Length-harmonizing fine-tuning for o1-like reasoning pruning, 2025

    work page 2025

  19. [19]

    Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, Chuning Tang, Congcong Wang, Dehao Zhang, Enming Yuan, Enzhe Lu, Fengxiang Tang, Flood Sung, Guangda Wei, Guokun Lai, Haiqing Guo, Han Zhu, Hao Ding, Hao Hu, Hao Yang, Hao Zhang, Haotian Yao, Haotian Zhao, Haoyu Lu, Haoze Li, Haoz...

    work page 2025

  20. [20]

    Training language models to reason efficiently, 2025

    Daman Arora and Andrea Zanette. Training language models to reason efficiently, 2025

    work page 2025

  21. [21]

    L1: Controlling how long a reasoning model thinks with reinforcement learning, 2025

    Pranjal Aggarwal and Sean Welleck. L1: Controlling how long a reasoning model thinks with reinforcement learning, 2025

    work page 2025

  22. [22]

    Demystifying long chain-of-thought reasoning in llms, 2025

    Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue. Demystifying long chain-of-thought reasoning in llms, 2025

    work page 2025

  23. [23]

    Dast: Difficulty-adaptive slow-thinking for large reasoning models, 2025

    Shuming Shi, Jian Zhang, Yi Shen, Kai Wang, Shiguo Lian, Ning Wang, Wenjing Zhang, Jieyun Huang, and Jiangze Yan. Dast: Difficulty-adaptive slow-thinking for large reasoning models, 2025

    work page 2025

  24. [24]

    Token dropping for efficient BERT pretraining

    Le Hou, Richard Yuanzhe Pang, Tianyi Zhou, Yuexin Wu, Xinying Song, Xiaodan Song, and Denny Zhou. Token dropping for efficient BERT pretraining. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors,Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3774–3784, Dublin, Irela...

    work page 2022

  25. [25]

    Rho-1: Not all tokens are what you need, 2025

    Zhenghao Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, Yelong Shen, Ruochen Xu, Chen Lin, Yujiu Yang, Jian Jiao, Nan Duan, and Weizhu Chen. Rho-1: Not all tokens are what you need, 2025

    work page 2025

  26. [26]

    s1: Simple test-time scaling, 2025

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling, 2025

    work page 2025

  27. [27]

    When more is less: Understanding chain-of-thought length in llms, 2025

    Yuyang Wu, Yifei Wang, Tianqi Du, Stefanie Jegelka, and Yisen Wang. When more is less: Understanding chain-of-thought length in llms, 2025

    work page 2025

  28. [28]

    Proximal Policy Optimization Algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. ArXiv, abs/1707.06347, 2017

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  29. [29]

    Littman, and Andrew W

    Leslie Pack Kaelbling, Michael L. Littman, and Andrew W. Moore. Reinforcement learning: A survey. J. Artif. Intell. Res., 4:237–285, 1996

    work page 1996

  30. [30]

    Deep reinforcement learning from human preferences

    Paul Francis Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. ArXiv, abs/1706.03741, 2017. 11

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  31. [31]

    Learning to summarize from human feedback

    Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan J. Lowe, Chelsea V oss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. ArXiv, abs/2009.01325, 2020

    work page internal anchor Pith review Pith/arXiv arXiv 2009

  32. [32]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. Training language models to follow instructions with h...

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  33. [33]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y . K. Li, Y . Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024

    work page 2024

  34. [34]

    REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization

    Jian Hu. Reinforce++: A simple and efficient approach for aligning large language models. ArXiv, abs/2501.03262, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  35. [35]

    Let’s verify step by step, 2023

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step, 2023

    work page 2023

  36. [36]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai D...

    work page 2025

  37. [37]

    Pengcheng Jiang, Jiacheng Lin, Lang Cao, R. Tian, S. Kang, Z. Wang, Jimeng Sun, and Jiawei Han. Deepretrieval: Hacking real search engines and retrievers with large language models via reinforcement learning. arXiv preprint arXiv: 2503.00223, 2025

    work page arXiv 2025

  38. [38]

    Fortune: Formula-driven reinforcement learning for symbolic table reasoning in language models

    Lang Cao, Jingxian Xu, Hanbing Liu, Jinyu Wang, Mengyu Zhou, Haoyu Dong, Shi Han, and Dongmei Zhang. Fortune: Formula-driven reinforcement learning for symbolic table reasoning in language models. arXiv preprint arXiv:2505.23667, 2025

    work page arXiv 2025

  39. [39]

    Chain-of-thought prompting elicits reasoning in large language models, 2023

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023

    work page 2023

  40. [40]

    Griffiths, Yuan Cao, and Karthik Narasimhan

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models, 2023. 12

    work page 2023

  41. [41]

    GraphReason: Enhancing reasoning capabilities of large language models through a graph-based verification approach

    Lang Cao. GraphReason: Enhancing reasoning capabilities of large language models through a graph-based verification approach. In Bhavana Dalvi Mishra, Greg Durrett, Peter Jansen, Ben Lipkin, Danilo Neves Ribeiro, Lionel Wong, Xi Ye, and Wenting Zhao, editors,Proceedings of the 2nd Workshop on Natural Language Reasoning and Structured Explanations (@ACL 20...

    work page 2024

  42. [42]

    Self-consistency improves chain of thought reasoning in language models, 2023

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models, 2023

    work page 2023

  43. [43]

    Token-budget-aware llm reasoning, 2025

    Tingxu Han, Zhenting Wang, Chunrong Fang, Shiyu Zhao, Shiqing Ma, and Zhenyu Chen. Token-budget-aware llm reasoning, 2025

    work page 2025

  44. [44]

    Reasoning models can be effective without thinking, 2025

    Wenjie Ma, Jingxuan He, Charlie Snell, Tyler Griggs, Sewon Min, and Matei Zaharia. Reasoning models can be effective without thinking, 2025

    work page 2025

  45. [45]

    Chain of draft: Thinking faster by writing less, 2025

    Silei Xu, Wenhao Xie, Lingxiao Zhao, and Pengcheng He. Chain of draft: Thinking faster by writing less, 2025

    work page 2025

  46. [46]

    Vicky Zhao, Lili Qiu, and Dongmei Zhang

    Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Menglin Xia, Xufang Luo, Jue Zhang, Qingwei Lin, Victor Rühle, Yuqing Yang, Chin-Yew Lin, H. Vicky Zhao, Lili Qiu, and Dongmei Zhang. Llmlingua-2: Data distillation for efficient and faithful task-agnostic prompt compression, 2024

    work page 2024

  47. [47]

    Training Verifiers to Solve Math Word Problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christo- pher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  48. [48]

    Compressing context to enhance inference efficiency of large language models

    Yucheng Li, Bo Dong, Frank Guerin, and Chenghua Lin. Compressing context to enhance inference efficiency of large language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6342–6353, Singapore, December 2023. Association for Computational Linguistics

    work page 2023

  49. [49]

    Llmlingua: Compress- ing prompts for accelerated inference of large language models, 2023

    Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. Llmlingua: Compress- ing prompts for accelerated inference of large language models, 2023

    work page 2023

  50. [50]

    insignificant

    Yucheng Li, Bo Dong, Chenghua Lin, and Frank Guerin. Compressing context to enhance inference efficiency of large language models, 2023. 13 Contents of Appendix A Limitations and Future Work 15 B Broader Impacts and Safeguards 15 C Discussion of Token Significance Measurement 16 D Theoretical Analysis of Significance-Aware Length Reward 16 E Theoretical D...

    work page 2023

  51. [51]

    Why does encouraging longer chains of thought (CoT) during early training help exploration?

    work page

  52. [52]

    Why does applying a fixed length penalty throughout training limit performance?

    work page

  53. [53]

    Why does dynamically flipping the reward from positive to negative upon convergence yield better accuracy–efficiency trade-offs?

    work page

  54. [54]

    Longer CoT Enables Richer Exploration. Let Pt(L) be the model’s distribution over output lengths at training stept, and define A(L) = Pr ˆz(y) = z | L(y) = L (27) as the expected accuracy (e.g., exact match) given output length L. Empirically, A(L) follows a saturating “S-curve”: A′(L) > 0 for L < L ⋆, A ′(L) ≈ 0 for L ≥ L⋆, (28) 18 where A′(L) denotes th...

    work page

  55. [55]

    Consider a fixed length penalty λ > 0, giving the reward Jstatic(L) = A(L) − λL

    Static Length Penalty Causes Premature Compression. Consider a fixed length penalty λ > 0, giving the reward Jstatic(L) = A(L) − λL. (30) The optimal length Ls under this objective satisfies A′(Ls) = λ. (31) Since A′(L) vanishes for L ≥ L⋆, any λ > 0 forces Ls < L ⋆, implying A(Ls) < A(L⋆). (32) As A′(L) > 0 for L < L ⋆, the reward Jstatic(L) reaches its ...

    work page

  56. [56]

    Let’s think step by step and output the final answer within \boxed

    Dynamic Penalty Supports a Two-Phase Curriculum. We introduce a time-dependent penalty λt: λt = ( 0, t < t 0 (exploration phase), α (t − t0), t ≥ t0 (compression phase), (33) where t0 is the step at which validation accuracy (training batch accuracy used in the experiments) stabilizes, i.e. when ˙At = Acct − Acct−∆ ∆ < β. (34) Phase I (Exploration). Durin...

    work page arXiv 2000