Pith. sign in

REVIEW 4 major objections 7 minor 2 cited by

Do Thinking Tokens Help or Trap? Towards More Efficient Large Reasoning Model

T0 review · 4 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Thinking tokens like 'wait' and 'hmm' are not necessary for reasoning, and a new training method that suppresses them improves both math accuracy and token efficiency.

desk verdict A promising token-efficiency method with a checkpoint-selection leak that inflates the headline gains; worth refereeing but not worth quoting the numbers yet. read the letter →

arxiv 2506.23840 v1 pith:2YWT5KNY submitted 2025-06-30 cs.CL cs.AI

Bowen Ding , Yuhan Chen , Futing Wang , Lingfeng Ming , Tao Lin This is my paper
classification cs.CLcs.AI
keywords largereasoningmodelsthinkingtokensoverthinkingtraptokenefficiencypreferenceoptimizationreinforcementlearningmathematical
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether the discourse markers that large reasoning models use to signal reflection, such as 'wait', 'hmm', and 'however', actually help or hurt. On a 1.5-billion-parameter distilled reasoning model, the authors find that wrong answers contain roughly twice as many thinking tokens as right answers, and that forcibly suppressing these tokens barely changes accuracy while saving about 1,000 tokens per response. They name this failure mode the thinking trap: an overconfident model keeps generating thinking tokens, and each one increases the probability of the next, producing loops of reflection and backtracking. The paper then proposes a reinforcement-learning method, DuP-PO, that trains the model to regulate thinking-token use instead of banning it. Across six math benchmarks the trained model gains 4.0 points of average accuracy while cutting response length by 15.4 percent.

What carries the argument

The load-bearing object is DuP-PO (Dual Policy Preference Optimization), an extension of Group Relative Policy Optimization. It samples each question twice: once from the normal policy and once from a 'rectified' policy that zeroes the logits of the eight predefined thinking tokens, guaranteeing that the model sees concise successful answers. A token-level advantage scaling then amplifies rewards for short correct thinking-free responses, magnifies penalties on thinking tokens inside wrong thinking-heavy responses, and zeroes rewards for thinking tokens when a concise correct answer already exists. A policy-shaping step recalibrates the old-policy probability of thinking tokens so that suppression gradients are not clipped away precisely when the model is most overconfident about them. Thinking tokens are defined as reflection markers ('wait', 'hmm', 'hold on', 'okay') and thought-transition markers ('alternatively', 'maybe', 'but', 'however').

What would settle it

Run the same thinking-token suppression on a much larger reasoning model, or on code-generation and open-ended tasks, and measure accuracy: if suppressing 'wait', 'hmm', and the rest lowers accuracy by more than a small margin, or fails to shorten responses, the claim that thinking tokens are unnecessary for effective reasoning is refuted.

Watch

Extended reading notes

Core claim

The central claim is that thinking tokens are not required for effective reasoning in large reasoning models; they often trigger reflection and backtracking loops that waste tokens without improving answers. In the authors' controlled experiments, suppressing a predefined set of thinking tokens during decoding left accuracy nearly unchanged while sharply reducing response length, and incorrect responses averaged twice as many thinking tokens as correct ones. The mechanism identified is a self-reinforcing cascade: the model assigns high probability (around 0.88 on average) to tokens like 'wait', and inserting a single 'wait' raises the predicted probability of later 'wait' tokens by roughly 100-fold over the next 20 positions. On this evidence the paper concludes that the thinking trap, not reflection itself, is what the model must learn to avoid, and that a training objective can suppress trap-inducing tokens while keeping accuracy.

Load-bearing premise

The whole chain rests on the premise that one 1.5-billion-parameter distilled model and a fixed list of eight thinking tokens represent how large reasoning models actually think, so the trap the paper removes is the trap that matters.

Editorial extensions

If this is right

  • Trainers can reduce overthinking without curated short-chain-of-thought data: DuP-PO uses only 1,000 medium-difficulty problems and 80 reinforcement-learning steps.
  • Test-time logit penalties on thinking tokens already preserve accuracy on simple math, so inference-only efficiency is possible; the paper's training step turns that preserved accuracy into actual gains.
  • DuP-PO's token savings are largest where overthinking is worst: MATH500 drops 24.7% of tokens with a 3.5-point accuracy gain, while harder AIME sets save 8-11% with smaller but consistent gains.
  • The method beats its base GRPO: it gains 1.3 more accuracy points, uses fewer tokens, and needs fewer training steps (80 versus 90).
  • Since incorrect responses carry twice as many thinking tokens, tracking thinking-token density during rollout can serve as an early signal of reasoning failure.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same rectified-policy idea could be turned into an adaptive inference-time controller that suppresses reflection markers only when the model's own confidence is high, replacing the paper's global suppression rule.
  • If the cascade mechanism is specific to thinking tokens, inserting semantically empty filler tokens should not produce the same 100-fold amplification; testing that contrast would isolate whether reflection meaning, not mere repetition, drives the trap.
  • The paper's evidence is limited to one 1.5-billion-parameter model and six math benchmarks; if the thinking-token/accuracy correlation reverses in code generation or open-ended reasoning, suppression may need to be conditioned on task type.
  • Token-level advantage scaling suggests a general design principle: attributing reward to specific token classes inside a trajectory, rather than to the whole trajectory, is what lets efficiency improve without sacrificing accuracy.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper argues that 'thinking tokens' such as wait, hmm, and however are not necessary for effective reasoning in large reasoning models and can trigger unproductive reflection loops, a phenomenon the authors call the thinking trap. It supports this with a token-level analysis of 6,023 responses from DeepSeek-R1-Distill-Qwen-1.5B, a training-free ThinkTokenPenalty intervention, and a new GRPO extension called DuP-PO that combines dual-policy sampling, token-level advantage scaling, and policy shaping to regulate thinking tokens. Experiments on six math benchmarks report that DuP-PO improves average accuracy by 4.0 points while reducing response length by 15.4% relative to the base model, and that it outperforms GRPO by 1.3 points at 80 versus 90 training steps. The paper explicitly limits its conclusions to the 1.5B model in Section 7.

Significance. If the central claim survives scrutiny, it is significant: it challenges the common assumption that more thinking is always better and offers a lightweight RL recipe for improving both accuracy and token efficiency of small LRMs. The paper deserves credit for directly testing the causal role of thinking tokens through a controlled logits-penalty intervention, for comparing against GRPO under matched conditions, and for being explicit that the evidence is limited to a 1.5B model. The core claim is also falsifiable: if suppressing thinking tokens degrades accuracy at scale or in other domains, the claim would fail. However, the quantitative support is currently weakened by a validation/test overlap for AIME24, missing statistics on the main observation, and the absence of ablations isolating the three mechanisms of DuP-PO. The token-reduction figures are also partly a designed-in consequence of the rectified policy and advantage scaling, rather than purely a learned behavior, and the paper should state this separation explicitly.

major comments (4)
  1. [Section 5.1 and Table 1] AIME24 is used both as the validation set for checkpoint selection and as a reported benchmark. The text says 'We utilise AIME24 as the validation set' and selects the GRPO 90-step and DuP-PO 80-step checkpoints on that set; Section 6 then reports AIME24 as one of the six evaluation benchmarks. This is a test-set selection leak: the reported AIME24 score is the result of choosing the best checkpoint on that same set, and with only 30 problems (Avg@32) the selection noise is large. Because the other five benchmarks are evaluated at those same selected checkpoints, their numbers inherit the bias, and the headline 4.0-point average gain and 15.4% token reduction are therefore optimistically biased. The authors should select checkpoints on a separate validation set or report complete training curves over all checkpoints, and should either exclude AIME24 from the test table or clearly label it as a development-set result.
  2. [Section 3.1 and Table 1] The paper's motivating observation that 'Incorrect responses contain twice as many thinking tokens as correct responses' is given without any counts, standard deviations, or significance test for the 6,023 responses. The subsequent ThinkTokenPenalty experiment reports 'minimal degradation' and 'approximately 1,000 token reductions' without exact accuracy values or the failure-type denominators behind the 86% to 37% reduction. In fact, Table 1 shows ThinkTokenPenalty losing 1.5 points on MATH500 (77.7 vs. 79.2) while gaining only 0.1 points on average, so the claim of preserving accuracy needs explicit confidence intervals or a significance test. Adding a supporting table with per-condition accuracy, token statistics, and effect sizes is necessary to make the thinking-trap premise load-bearing.
  3. [Section 4.4 and Table 1] DuP-PO combines three innovations, but no ablation is reported. The full method is compared only to GRPO, so the 1.3-point accuracy gain and the token savings cannot be attributed to the fine-grained thinking-token control; any of the three components, especially the rectified policy's hard removal of thinking tokens during rollouts, could account for most of the efficiency gain. Reporting one-factor-at-a-time ablations, such as DuP-PO without Policy Shaping, without Advantage Scaling, and with only normal-policy rollouts, is required to support the claim that the specific thinking-token mechanism rather than a generic conciseness bias drives the results.
  4. [Section 4.4, Eq. (6)] The token savings are partly designed into the method rather than learned. The Rectified Policy sets thinking-token logits to negative infinity, the enhancement case multiplies advantages on preferred rectified trajectories by alpha greater than 1, and the return-to-zero case deletes advantages on thinking tokens in preferred normal trajectories whenever a correct rectified trajectory exists. The paper should therefore separate token reduction enforced by the intervention from token reduction produced by the trained policy, and report both; otherwise the efficiency claim in Table 1 conflates design choices with empirical findings.
minor comments (7)
  1. [Table 1] The column header 'MINER V A' is a typo for 'MINERVA', and 'OLYMPIADBENCH' is truncated; the standard deviations promised by 'all reported results represent the average of three inference runs' are not shown.
  2. [Section 5.3.2, Eq. (9)] The reward function introduces a 0.1 formatting bonus, so the effective rewards are 0, 0.1, 1.0, and 1.1 rather than the binary 0/1 in Eq. (1); the paper should clarify how Eq. (2)'s group normalization operates over these four reward levels.
  3. [Section 6.3] The claim of faster training is not measured; comparing step 80 with step 90 is not a control for training compute, and the rollout budget of the GRPO baseline is not specified. Please report wall-clock time or total sample counts, or remove the 'training speed' claim.
  4. [Section 3.2, Figure 2b] The '100-fold increase' in wait probability after insertion is reported without control insertions of a non-thinking token and without variance over the 100 responses, so the causal claim is weaker than stated.
  5. [Section 4.1] Section 4.1 is empty: the heading 'Preliminary' is immediately followed by Section 4.2 'GRPO', so the subsection should either be populated with notation or removed.
  6. [References] Several works are listed twice with different arXiv identifiers (Muennighoff et al. 2025a/2025b, Wang et al. 2025b/2025c, and Luo et al. 2025a/2025b); these duplicates should be merged or disambiguated.
  7. [Abstract] The abstract states 'five popular math reasoning benchmarks' while Section 5.1 and Table 1 list six benchmarks; the count should be corrected.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the thinking-trap claim rests on an independent token-suppression intervention, and DuP-PO's preference for thinking-token-free responses is disclosed as its mechanism rather than used as evidence for the trap.

full rationale

The paper's central claim that thinking tokens are not necessary is tested directly in Section 3.1 via ThinkTokenPenalty, a logits-level intervention that suppresses a predefined set of thinking tokens (wait, hmm, hold on, alternatively, maybe, however, but, okay). The observed result that accuracy remains stable while token usage drops is an empirical outcome, not an identity: suppressing those tokens does not by construction preserve or improve accuracy. The correlational observation that incorrect responses contain more thinking tokens is only used to motivate the intervention, not as the proof. DuP-PO's objective in Equation (6) does encode a preference for concise, thinking-token-free responses through the enhancement factor for rectified-policy successes and the return-to-zero rule for thinking tokens in preferred normal-policy trajectories, so the reported 15.4% token reduction is partly by design. However, the paper does not present DuP-PO's token savings as evidence for the thinking trap; that evidence is the independent TTP experiment and the mechanism analysis in Section 3.2. The accuracy gains from DuP-PO are not forced by construction because the reward function is correctness-grounded and the comparison against GRPO is empirical. The use of AIME24 for checkpoint selection and then as a reported benchmark in Table 1 is a test-set-selection leak that may inflate the 4.0-point average and is a legitimate correctness concern, but it is not definitional circularity: the reported AIME24 number is not identical to the selection criterion by any equation, and the qualitative thinking-trap finding does not depend on it. There is no load-bearing self-citation chain and no imported uniqueness theorem; the LUFFY-inspired policy shaping is an implementation detail borrowed from external work, not an unverified premise that forces the conclusion. Overall, no step in the derivation chain reduces to its own input.

Assumptions & free parameters 7 free parameters · 6 assumptions · 1 invented entities

The central claim rests on a small set of hand-set constants (alpha, beta, gamma), a predefined thinking-token set, and the representativeness of a single 1.5B model. These are not fitted to test data in the usual sense, but no sensitivity analysis is reported, so the method's robustness is unmeasured. The empirical 'derivation' is not circular, but it is narrow in scope.

free parameters (7)
  • Advantage enhancement factor alpha = 2
    Hand-set constant that multiplies advantages for preferred trajectories from the rectified policy (Section 4.4.2, Eq. 6). No sensitivity analysis reported.
  • Advantage suppression factor beta = 2
    Hand-set constant that magnifies negative advantages for thinking tokens in unsuccessful normal-policy trajectories (Section 4.4.2, Eq. 6).
  • Old-policy calibration constant gamma = 0.1
    Used in Eq. 7 to reshape old-policy thinking-token probabilities; the paper assumes thinking-token probabilities remain above gamma during training to guarantee gradient flow.
  • GRPO clip range epsilon = 0.2
    Standard clip bound used in Eq. 3 and Eq. 8; the policy-shaping guarantee depends on this value.
  • Data selection thresholds = correctness 0.25-0.5; avg length >8192 tokens
    Curated DuPPO-1K from DAPO-MATH-17K by filtering for medium difficulty and thinking-trap-prone long responses (Section 5.1).
  • Rollout split N=4, M=4 = 4/4
    Number of trajectories sampled from rectified and normal policies in each rollout (Section 5.3.1); no study of sensitivity is reported.
  • Entropy coefficient = 0.01
    Training hyperparameter in Section 5.3.1; likely minor but unexamined.
assumptions (6)
  • domain assumption The eight listed tokens (wait, hmm, hold on, alternatively, maybe, however, but, okay) constitute a sufficient definition of 'thinking tokens' for both analysis and training.
    Section 3.1 and Section 5.3.1: this set drives the TTP intervention, the rectified policy, and the advantage scaling; if the set excludes important thinking behaviors, the conclusions are incomplete.
  • domain assumption DeepSeek-R1-Distill-Qwen-1.5B is representative of LRMs for the thinking-trap phenomenon.
    All experiments use this single model; Section 7 defers larger models to future work, so generalization is assumed.
  • domain assumption The rule-based reward (MathVerify parsing plus formatting bonus) correctly captures answer quality for all benchmarks.
    Section 5.3.2 defines reward with is_equivalent and formatting; if parsing misjudges correctness, training signals are biased.
  • domain assumption Trajectories generated by the rectified policy (logits of thinking tokens set to -inf) are valid samples whose rewards are comparable to normal-policy trajectories.
    Section 4.4.1: dual-policy sampling treats rectified-policy trajectories as ordinary responses; forced token removal may create unnatural outputs, yet advantages from them are used to train the policy.
  • standard math Standard RL/GRPO machinery (policy gradient, clipping, advantage normalization) is sound.
    Section 4.2 invokes standard GRPO definitions without proof; accepted background.
  • domain assumption The insertion experiment (adding a wait token into responses with no wait tokens) estimates the counterfactual cascade effect.
    Section 3.2 reports a 100-fold probability increase after inserting one wait token on 100 responses; this assumes insertion is a valid counterfactual intervention.
invented entities (1)
  • thinking trap
    purpose: Conceptual label for unproductive recursive self-reflection loops that consume tokens without improving correctness.
    The paper defines the trap through its own observations (Sections 3.1-3.2) and demonstrates it in one model; no external benchmark or independent replication is provided. It is a useful description, not a measured physical quantity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Do Thinking Tokens Help or Trap? Towards More Efficient Large Reasoning Model." pith.science (2026). https://pith.science/paper/2YWT5KNY

@misc{pith2026250623840,
  author       = {Pith},
  title        = {Pith review of: Do Thinking Tokens Help or Trap? Towards More Efficient Large Reasoning Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2YWT5KNY}},
  note         = {Machine review of arXiv:2506.23840}
}
read the original abstract

Large Reasoning Models (LRMs) excel at solving complex problems but face an overthinking dilemma. When handling simple tasks, they often produce verbose responses overloaded with thinking tokens (e.g., wait, however). These tokens trigger unnecessary high-level reasoning behaviors like reflection and backtracking, reducing efficiency. In this work, our pilot study reveals that these thinking-token-induced behaviors are not essential for effective problem-solving and may even hinder correct reasoning within constrained token budgets. We identify this phenomenon as the thinking trap. To mitigate this issue, we propose Dual Policy Preference Optimization (DuP-PO), a novel algorithm featuring: (1) A rollout sampling strategy that guarantees balanced exposure to responses with and without thinking tokens; (2) A fine-grained advantage control technique to dynamically regulate the prediction of target tokens; (3) A policy shaping method ensuring stable gradient contributions from thinking tokens. Experimental results on five popular math reasoning benchmarks show that DuP-PO performs well on the popular LRM, which significantly improves their token efficiency during reasoning, while achieving superior performance of the base model.

Figures

Figures reproduced from arXiv: 2506.23840 by the authors.

Figure 1
Figure 1. The Illustration of Trapped v.s. Effi￾cient Reasoning. An example from MATH500 where correct inference gets stuck in redundant verification loops, failing to produce a final answer within token limits. conventional Large Language Models (LLMs), LRMs consistently generate discourse mark￾ers such as “wait, hmm, however” within their reasoning processes, which we term thinking tokens (elaborated in Section 5.3.1). Thes… view at source ↗
Figure 2
Figure 2. Analysis of thinking token effects in DeepSeek-R1-Distill-Qwen-1.5B. (2a) The heatmap of token probability. Darker backgrounds indicate higher probabilities, showing systematic over-confidence in thinking tokens like wait. (2b) The illustration of wait cascading effect. Blue circles show the original wait probabilities, red squares show 100-fold increases after inserting a single wait token, demonstrating auto-regre… view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Attention Degradation, Function Token Anchoring, and the Limits of Attention-Based Intervention in Large Language Models

    cs.AI 2026-07 conditional novelty 6.0 of 10

    In GPT-2, OPT-1.3B, and LLaMA-3.2 models, boosting attention to function-token positions by 16-24% does not improve contextual retrieval, so short-range attention degradation is descriptive rather than causal.

  2. Towards Concise and Adaptive Thinking in Large Reasoning Models: A Survey

    cs.AI 2025-07 conditional novelty 3.0 of 10

    A comprehensive review that categorizes methods for shortening and adaptively triggering chain-of-thought reasoning in large language models.

Reference graph

Works this paper leans on

51 extracted references · 5 canonical work pages · cited by 2 Pith papers

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Pranjal Aggarwal and Sean Welleck. 2025. L1: Controlling how long a reasoning model thinks with reinforcement learning. arXiv preprint arXiv:2503.04697

    arXiv 2025

  4. [4]

    Sanghwan Bae, Jiwoo Hong, Min Young Lee, Hanbyul Kim, JeongYeon Nam, and Donghyun Kwak. 2025. https://arxiv.org/abs/2504.03380 Online difficulty filtering for reasoning oriented reinforcement learning . Preprint, arXiv:2504.03380

    arXiv 2025

  5. [5]

    Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. 2025. https://arxiv.org/abs/2412.21187 Do not think that much for 2+3=? on the overthinking of o1-like llms . Preprint, arXiv:2412.21187

    arXiv 2025

  6. [6]

    Yingqian Cui, Pengfei He, Jingying Zeng, Hui Liu, Xianfeng Tang, Zhenwei Dai, Yan Han, Chen Luo, Jing Huang, Zhen Li, and 1 others. 2025. Stepwise perplexity-guided refinement for efficient chain-of-thought reasoning in large language models. arXiv preprint arXiv:2502.13260

    arXiv 2025

  7. [7]

    DeepSeek - AI. 2025. https://doi.org/10.48550/ARXIV.2501.12948 Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning . CoRR, abs/2501.12948

    arXiv doi:10.48550/arxiv.2501.12948 2025

  8. [8]

    Gongfan Fang, Xinyin Ma, and Xinchao Wang. 2025. https://arxiv.org/abs/2505.13379 Thinkless: Llm learns when to think . Preprint, arXiv:2505.13379

    arXiv 2025

Show all 51 references
  1. [9]

    Sicheng Feng, Gongfan Fang, Xinyin Ma, and Xinchao Wang. 2025. https://arxiv.org/abs/2504.10903 Efficient reasoning models: A survey . Preprint, arXiv:2504.10903

    2025

  2. [10]

    Yichao Fu, Junda Chen, Siqi Zhu, Zheyu Fu, Zhongdongming Dai, Aurick Qiao, and Hao Zhang. 2024. Efficiently serving llm reasoning programs with certaindex. arXiv preprint arXiv:2412.20993

    2024 arXiv

  3. [11]

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. 2024. https://arxiv.org/abs/2402.14008 Olympiadbench: A challenging benchmark for promoting agi with olym...

    2024 arXiv

  4. [12]

    Bairu Hou, Yang Zhang, Jiabao Ji, Yujian Liu, Kaizhi Qian, Jacob Andreas, and Shiyu Chang. 2025. Thinkprune: Pruning long chain-of-thought of llms via reinforcement learning. arXiv preprint arXiv:2504.01296

    2025 arXiv

  5. [13]

    Jia LI , Edward Beeching , Lewis Tunstall , Ben Lipkin , Roman Soletskyi , Shengyi Costa Huang , Kashif Rasul , Longhui Yu , Albert Jiang , Ziju Shen , Zihan Qin , Bin Dong , Li Zhou , Yann Fleureau , Guillaume Lample , and Stanislas Polu . 2024. https://github.com/project-num...

    2024

  6. [14]

    Yu Kang, Xianghui Sun, Liangyu Chen, and Wei Zou. 2025. C3ot: Generating shorter chain-of-thought without compromising effectiveness. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 24312--24320

    2025

  7. [15]

    Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. 2022. https://arxiv.org/abs/2206.14858 Solving quantitative re...

    2022 arXiv

  8. [16]

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. https://openreview.net/forum?id=v8L0pN6EOi Let's verify step by step . In The Twelfth International Conference on Learning Re...

    2024

  9. [17]

    Zichen Liu, Changyu Chen, Wenjun Li, Tianyu Pang, Chao Du, and Min Lin. 2025. There may not be aha moment in r1-zero-like training — a pilot study. https://oatllm.notion.site/oat-zero. Notion Blog

    2025

  10. [19]

    Haotian Luo, Li Shen, Haiying He, Yibo Wang, Shiwei Liu, Wei Li, Naiqiang Tan, Xiaochun Cao, and Dacheng Tao. 2025 b . O1-pruner: Length-harmonizing fine-tuning for o1-like reasoning pruning. arXiv preprint arXiv:2501.12570

    2025 arXiv

  11. [20]

    Wenjie Ma, Jingxuan He, Charlie Snell, Tyler Griggs, Sewon Min, and Matei Zaharia. 2025 a . Reasoning models can be effective without thinking. arXiv preprint arXiv:2504.09858

    2025 arXiv

  12. [21]

    Xinyin Ma, Guangnian Wan, Runpeng Yu, Gongfan Fang, and Xinchao Wang. 2025 b . Cot-valve: Length-compressible chain-of-thought tuning. arXiv preprint arXiv:2502.09601

    2025 arXiv

  13. [22]

    Cand \` e s, and Tatsunori Hashimoto

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei - Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel J. Cand \` e s, and Tatsunori Hashimoto. 2025 a . https://doi.org/10.48550/ARXIV.2501.19393 s1: Simple test-time scaling . CoRR, abs/2501.19393

    2025 doi arXiv

  14. [23]

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. 2025 b . https://arxiv.org/abs/2501.19393 s1: Simple test-time scaling . Preprint, arXiv:2501.19393

    2025 arXiv

  15. [24]

    Tergel Munkhbat, Namgyu Ho, Seo Hyun Kim, Yongjin Yang, Yujin Kim, and Se-Young Yun. 2025. Self-training elicits concise reasoning in large language models. arXiv preprint arXiv:2502.20122

    2025 arXiv

  16. [25]

    Open-R1-Team . 2025. https://huggingface.co/blog/open-r1/mini-r1-contdown-game Mini r1 countdown game . Blog post

    2025

  17. [26]

    OpenAI . 2024. Learning to reason with llms. https://openai.com/index/learning-to-reason-with-llms/. Accessed: 2025-05-07

    2024

  18. [27]

    Chen Qian, Dongrui Liu, Haochen Wen, Zhen Bai, Yong Liu, and Jing Shao. 2025. https://arxiv.org/abs/2506.02867 Demystifying reasoning dynamics with mutual information: Thinking tokens are information peaks in llm reasoning . Preprint, arXiv:2506.02867

    2025 arXiv

  19. [28]

    Yuxiao Qu, Matthew YR Yang, Amrith Setlur, Lewis Tunstall, Edward Emanuel Beeching, Ruslan Salakhutdinov, and Aviral Kumar. 2025. Optimizing test-time compute via meta reinforcement fine-tuning. arXiv preprint arXiv:2503.07572

    2025 arXiv

  20. [29]

    Qwen Team . 2025. Qwq-32b-preview. https://qwenlm.github.io/blog/qwq-32b-preview/. Accessed: 15 March 2025

    2025

  21. [30]

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. https://api.semanticscholar.org/CorpusID:28695052 Proximal policy optimization algorithms . ArXiv, abs/1707.06347

    2017 arXiv

  22. [31]

    ByteDance Seed, :, Jiaze Chen, Tiantian Fan, Xin Liu, Lingjun Liu, Zhiqi Lin, Mingxuan Wang, Chengyi Wang, Xiangpeng Wei, Wenyuan Xu, Yufeng Yuan, Yu Yue, Lin Yan, Qiying Yu, Xiaochen Zuo, Chi Zhang, Ruofei Zhu, Zhecheng An, and 255 others. 2025. https://arxiv.org/abs/2504.139...

    2025

  23. [32]

    Yi Shen, Jian Zhang, Jieyun Huang, Shuming Shi, Wenjing Zhang, Jiangze Yan, Ning Wang, Kai Wang, Zhaoxiang Liu, and Shiguo Lian. 2025. Dast: Difficulty-adaptive slow-thinking for large reasoning models. arXiv preprint arXiv:2503.04472

    2025

  24. [33]

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. 2024. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19256

    2024 arXiv

  25. [34]

    Jinyan Su and Claire Cardie. 2025. https://arxiv.org/abs/2505.18298 Thinking fast and right: Balancing accuracy and reasoning length with adaptive rewards . Preprint, arXiv:2505.18298

    2025 arXiv

  26. [35]

    Yang Sui, Yu - Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen Zhong, Hanjie Chen, and Xia Ben Hu. 2025. https://doi.org/10.48550/ARXIV.2503.16419 Stop overthinking: A survey on efficient reasoning for large language models . ...

    2025 doi arXiv

  27. [36]

    Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, and 1 others. 2025. Kimi k1.5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599

    2025 arXiv

  28. [37]

    Songjun Tu, Jiahao Lin, Qichao Zhang, Xiangyu Tian, Linjing Li, Xiangyuan Lan, and Dongbin Zhao. 2025. Learning when to think: Shaping adaptive reasoning in r1-style models via multi-stage rl. arXiv preprint arXiv:2505.10832

    2025

  29. [38]

    Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, Yuqiong Liu, An Yang, Andrew Zhao, Yang Yue, Shiji Song, Bowen Yu, Gao Huang, and Junyang Lin. 2025 a . https://arxiv.org/abs/2506.01939 Beyond the 80/20 rul...

    2025 arXiv

  30. [40]

    Yue Wang, Qiuzhi Liu, Jiahao Xu, Tian Liang, Xingyu Chen, Zhiwei He, Linfeng Song, Dian Yu, Juntao Li, Zhuosheng Zhang, and 1 others. 2025 c . Thoughts are all over the place: On the underthinking of o1-like llms. arXiv preprint arXiv:2501.18585

    2025 arXiv

  31. [41]

    Heming Xia, Yongqi Li, Chak Tou Leong, Wenjie Wang, and Wenjie Li. 2025. Tokenskip: Controllable chain-of-thought compression in llms. arXiv preprint arXiv:2502.12067

    2025

  32. [42]

    Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. 2025. Learning to reason under off-policy guidance. arXiv preprint arXiv:2504.14945

    2025 arXiv

  33. [43]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025 a . https://arxiv.org/abs/2505.09388...

    2025 arXiv

  34. [44]

    Chenxu Yang, Qingyi Si, Yongjie Duan, Zheliang Zhu, Chenyu Zhu, Qiaowei Li, Zheng Lin, Li Cao, and Weiping Wang. 2025 b . Dynamic early exit in reasoning models. arXiv preprint arXiv:2504.15895

    2025

  35. [45]

    Junjie Yang, Ke Lin, and Xing Yu. 2025 c . Think when you need: Self-adaptive chain-of-thought learning. arXiv preprint arXiv:2504.03234

    2025 arXiv

  36. [46]

    Wong, and Di Wang

    Shu Yang, Junchao Wu, Xin Chen, Yunze Xiao, Xinyi Yang, Derek F. Wong, and Di Wang. 2025 d . https://arxiv.org/abs/2504.02956 Understanding aha moments: from external observations to internal mechanisms . Preprint, arXiv:2504.02956

    2025 arXiv

  37. [47]

    Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue. 2025. Demystifying long chain-of-thought reasoning in llms. arXiv preprint arXiv:2502.03373

    2025 arXiv

  38. [48]

    Ping Yu, Jing Xu, Jason Weston, and Ilia Kulikov. 2024. Distilling system 2 into system 1. arXiv preprint arXiv:2407.06023

    2024 arXiv

  39. [49]

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, and 16 others. 2025. https://arxiv.org/abs/2503...

    2025 arXiv

  40. [50]

    Yu Yue, Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du, Xiangpeng Wei, Xiangyu Yu, Gaohong Liu, Juncai Liu, Lingjun Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Chi Zhang, and 8 others. 2025. https://arxiv.org/abs/2504...

    2025 arXiv

  41. [51]

    Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. 2025. https://arxiv.org/abs/2503.18892 Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild . Preprint, arXiv:2503.18892

    2025 arXiv

  42. [52]

    Jiajie Zhang, Nianyi Lin, Lei Hou, Ling Feng, and Juanzi Li. 2025. Adaptthink: Reasoning models can learn when to think. arXiv preprint arXiv:2505.13417

    2025 arXiv

  43. [53]

    aha moment

    Hengguang Zhou, Xirui Li, Ruochen Wang, Minhao Cheng, Tianyi Zhou, and Cho-Jui Hsieh. 2025. https://arxiv.org/abs/2503.05132 R1-zero's "aha moment" in visual reasoning on a 2b non-sft model . Preprint, arXiv:2503.05132

    2025 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.