pith. sign in

arxiv: 2506.11274 · v2 · pith:QMZQ7JO6new · submitted 2025-06-12 · 💻 cs.CL · cs.LG

Learning a Continue-Thinking Token for Enhanced Test-Time Scaling

Liran Ringel , Elad Tolochinsky , Yaniv Romano This is my paper

Pith reviewed 2026-05-19 09:05 UTC · model grok-4.3

classification 💻 cs.CL cs.LG
keywords test-time scalingcontinue-thinking tokenreinforcement learningmath reasoninglanguage modelsbudget forcingtoken embedding
0
0 comments X

The pith

A learned continue-thinking token triggers more effective extended reasoning than fixed overrides like 'Wait'.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tests whether a model can be taught its own signal to keep reasoning longer at inference time instead of stopping. They add one new token to a frozen distilled reasoning model and train only its embedding with reinforcement learning. This produces larger accuracy gains on math benchmarks than either the base model or the common trick of replacing the end-of-thought marker with a fixed word. A sympathetic reader would care because the method uses almost no extra parameters yet unlocks more of the benefit from additional test-time compute. It suggests that some of the power of longer reasoning chains can be captured by learning a compact, model-specific continuation cue.

Core claim

By augmenting a distilled version of DeepSeek-R1 with a single learned <|continue-thinking|> token and training only its embedding via reinforcement learning while keeping all model weights frozen, the authors show that this token elicits extended reasoning steps that improve accuracy on standard math benchmarks more than a fixed-token budget-forcing baseline. Where the fixed-token approach already raises accuracy, the learned token delivers markedly larger gains, for instance 4.2 percent absolute improvement on GSM8K versus 1.3 percent from the fixed method.

What carries the argument

The learned <|continue-thinking|> token whose embedding is optimized via reinforcement learning to prompt continued reasoning while the remainder of the model stays frozen.

If this is right

  • The learned token produces larger accuracy improvements than fixed-token budget forcing on benchmarks where extra reasoning already helps.
  • On GSM8K the absolute gain reaches 4.2 percent over the base model compared with 1.3 percent for the fixed token.
  • Test-time scaling is achieved by updating only a single token embedding without changing any model parameters.
  • The benefit is clearest precisely in the settings where manually forcing additional steps already yields some improvement.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same single-token training approach could be tried on other base models or on non-math reasoning tasks to test transfer.
  • Multiple learned continuation tokens might eventually be trained to control distinct aspects of reasoning behavior.
  • If the token encodes a general 'think longer' signal, it could reduce reliance on hand-crafted prompt overrides across different inference budgets.

Load-bearing premise

The accuracy gains come from the token genuinely prompting useful extended reasoning steps rather than from side effects of the reinforcement learning procedure or from the particular reward and benchmark choices.

What would settle it

If inserting the learned token produces no measurable increase in reasoning-chain length or quality compared with the fixed-token baseline on the same problems, yet accuracy still rises, the claim that the token triggers genuinely extended reasoning would be falsified.

Figures

Figures reproduced from arXiv: 2506.11274 by Elad Tolochinsky, Liran Ringel, Yaniv Romano.

Figure 1
Figure 1. Figure 1: Text generation with budget forcing: Whenever the model outputs a [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Accuracy of different methods as a function of the average number of tokens generated by each method. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Comparison of generated sequence length distributions across methods and datasets and their cor [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Word cloud of the first token gen￾erated immediately after injecting the learned <|continue-thinking|> token, across all datasets. <|continue-thinking|> token. The most com￾mon continuations often prompt the model to self￾verify or reconsider its previous steps, indicating that the token effectively encourages reflective rea￾soning and backtracking. The reasoning trace depicted in [PITH_FULL_IMAGE:figures… view at source ↗
Figure 5
Figure 5. Figure 5: GSM8K reasoning trace demonstrating the positive impact of <|continue_thinking|> token. Blue indicates the original reasoning, yielding an in￾correct answer of 7,938. Green shows the continuation after the special token was added, leading to the correct answer of 294. model to the correct conclusion. See Appendix D for the full reasoning traces and additional exam￾ples. 5 Conclusions In this work, we have … view at source ↗
Figure 6
Figure 6. Figure 6: GSM8K reasoning trace demonstrating the positive impact of the [PITH_FULL_IMAGE:figures/full_fig_p015_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: GSM8K reasoning trace of the baseline model for the same question as in [PITH_FULL_IMAGE:figures/full_fig_p016_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: GSM8K reasoning trace demonstrating that the [PITH_FULL_IMAGE:figures/full_fig_p017_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: GSM8K reasoning trace of the baseline model for the same question as in [PITH_FULL_IMAGE:figures/full_fig_p018_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: GSM8K reasoning trace of a wrong answer given by both models. [PITH_FULL_IMAGE:figures/full_fig_p019_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: GSM8K reasoning trace of the baseline model for the same question as in [PITH_FULL_IMAGE:figures/full_fig_p020_11.png] view at source ↗
read the original abstract

Test-time scaling has emerged as an effective approach for improving language model performance by utilizing additional compute at inference time. Recent studies have shown that overriding end-of-thinking tokens (e.g., replacing "</think>" with "Wait") can extend reasoning steps and improve accuracy. In this work, we explore whether a dedicated continue-thinking token can be learned to trigger extended reasoning. We augment a distilled version of DeepSeek-R1 with a single learned "<|continue-thinking|>" token, training only its embedding via reinforcement learning while keeping the model weights frozen. Our experiments show that this learned token achieves improved accuracy on standard math benchmarks compared to both the baseline model and a test-time scaling approach that uses a fixed token (e.g., "Wait") for budget forcing. In particular, we observe that in cases where the fixed-token approach enhances the base model's accuracy, our method achieves a markedly greater improvement. For example, on the GSM8K benchmark, the fixed-token approach yields a 1.3% absolute improvement in accuracy, whereas our learned-token method achieves a 4.2% improvement over the base model that does not use budget forcing.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The manuscript proposes augmenting a frozen distilled DeepSeek-R1 model with a single learned <|continue-thinking|> token whose embedding is trained via reinforcement learning. It reports that this token yields larger accuracy gains on math reasoning benchmarks than a fixed-token budget-forcing baseline (e.g., 4.2 % vs. 1.3 % absolute improvement on GSM8K relative to the no-budget-forcing base model).

Significance. If the gains are attributable to the learned token inducing longer, higher-quality reasoning traces rather than incidental RL effects, the approach would constitute a low-parameter, model-frozen method for improving test-time scaling. The decision to train only the embedding while freezing all other weights is a clear efficiency strength that merits explicit credit.

major comments (2)
  1. [Results] Results / GSM8K comparison: the headline 4.2 % vs. 1.3 % absolute gains are stated without error bars, number of runs, or any statistical test. Because the central empirical claim rests on this specific difference, the absence of variance information makes it impossible to judge whether the reported superiority is robust.
  2. [Method] Method / ablation design: no control is described in which an embedding is trained with the identical RL procedure and reward but is not given the continue-thinking interpretation (e.g., a neutral or randomly initialized token). Without this isolation, it remains possible that the observed improvement arises from generic reward optimization on the embedding rather than from the token learning to trigger extended reasoning.
minor comments (1)
  1. [Abstract] The abstract refers to “standard math benchmarks” yet only details GSM8K; listing the complete set of evaluated datasets and their corresponding gains would improve completeness.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive comments on our manuscript. We address each major comment below.

read point-by-point responses

  1. Referee: [Results] Results / GSM8K comparison: the headline 4.2 % vs. 1.3 % absolute gains are stated without error bars, number of runs, or any statistical test. Because the central empirical claim rests on this specific difference, the absence of variance information makes it impossible to judge whether the reported superiority is robust.

    Authors: We agree that variance information is necessary to assess the robustness of the reported gains. In the revised manuscript, we will perform the experiments across multiple random seeds, report error bars (standard deviation), specify the number of runs, and include a statistical test to evaluate whether the 4.2% improvement is significantly larger than the 1.3% improvement from the fixed-token baseline. revision: yes

  2. Referee: [Method] Method / ablation design: no control is described in which an embedding is trained with the identical RL procedure and reward but is not given the continue-thinking interpretation (e.g., a neutral or randomly initialized token). Without this isolation, it remains possible that the observed improvement arises from generic reward optimization on the embedding rather than from the token learning to trigger extended reasoning.

    Authors: We recognize the importance of this control to rule out generic effects of RL on the embedding. Our current approach trains the embedding specifically for its role in continuing the thinking process by overriding the end token. To address the referee's point, we will add an ablation in which we train a different embedding using the same RL procedure and reward signal but without linking it to the continue-thinking mechanism, and report the comparative results to show that the performance gains are due to the learned continue-thinking behavior. revision: yes

Circularity Check

0 steps flagged

No significant circularity; empirical comparisons are independent of fitted inputs

full rationale

The paper describes an empirical procedure: a single token embedding is trained via RL on a frozen distilled DeepSeek-R1 model, then accuracy is measured on GSM8K and other benchmarks against a no-budget-forcing baseline and a fixed-token baseline. These reported deltas (e.g., 4.2 % vs 1.3 %) are direct experimental outcomes, not quantities derived from equations that reduce by construction to the trained embedding or reward signal. No self-definitional steps, fitted-input-called-prediction patterns, or load-bearing self-citations appear in the method or results chain. The evaluation remains falsifiable by re-running the RL training and benchmark measurements under the stated protocol.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 0 invented entities

The central claim rests on the learned embedding vector and on the assumption that RL on that vector alone produces a token that reliably extends useful reasoning.

free parameters (1)
  • continue-thinking token embedding
    The only trainable parameters; their values are fitted by the RL procedure.
axioms (1)
  • domain assumption Reinforcement learning on a single token embedding can produce a signal that extends reasoning steps in a useful way
    Invoked when the authors decide to train only the embedding and evaluate on downstream accuracy.

pith-pipeline@v0.9.0 · 5733 in / 1263 out tokens · 29924 ms · 2026-05-19T09:05:58.135777+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

15 extracted references · 15 canonical work pages · 1 internal anchor

  1. [1]

    In Advances in Neural Information Processing Systems

    Solving quantitative reasoning problems with language models. In Advances in Neural Information Processing Systems. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harri- son Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe

    work page

  2. [2]

    s1: Simple test-time scaling

    Let’s verify step by step. In International Conference on Learning Representations. Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, and Li Erran Li. 2025. Deep- scaler: Surpassing o1-preview with a 1.5B model by scaling RL. https://www.notion.so/Deepscal er-Surpassing-o1-preview-wit...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  3. [3]

    Let’s think dot by dot: Hidden computation in transformer language models.arXiv preprint arXiv:2404.15758,

    Let’s think dot by dot: Hidden computa- tion in transformer language models. arXiv preprint arXiv:2404.15758. Amrith Setlur, Nived Rajaraman, Sergey Levine, and Aviral Kumar. 2025. Scaling test-time compute with- out verification or RL is suboptimal. arXiv preprint arXiv:2502.12118. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Ha...

    work page arXiv 2025

  4. [4]

    how much does Lloyd make on eggs per week

    Guiding language model reasoning with plan- ning tokens. arXiv preprint arXiv:2310.05707. Yiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren, Lucas Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, and 1 others. 2025. Re- inforcement learning for reasoning in large language models with one training example. arXiv preprint arXiv:2504.20571. J...

    work page arXiv 2025

  5. [5]

    Calculate weekly egg production: 252 eggs/day * 7 days = 1,764 eggs/week

    work page

  6. [6]

    Convert weekly eggs to dozens: 1,764 eggs / 12 = 147 dozen

    work page

  7. [7]

    Figure 6: GSM8K reasoning trace demonstrating the positive impact of the <|continue_thinking|> token

    Calculate weekly revenue: 147 dozen * $2/dozen = $294 Answer: Lloyd makes \boxed{294} dollars per week from his egg sales. Figure 6: GSM8K reasoning trace demonstrating the positive impact of the <|continue_thinking|> token. Blue indicates the original reasoning, yielding an incorrect answer of 7,938. Green shows the continuation after the special token w...

    work page

  8. [8]

    Calculate the total number of eggs produced per day: Eggs per day = Number of chickens × Eggs per chicken per day Eggs per day = 27 × 252 = 6 ,804 eggs

    work page

  9. [9]

    Determine the number of dozens of eggs produced per day: Dozens per day = Eggs per day 12 Dozens per day = 6,804 12 = 567 dozen

    work page

  10. [10]

    Calculate the daily revenue from selling eggs: Revenue per day = Dozens per day × Selling price per dozen Revenue per day = 567 × 2 = 1 ,134 dollars

    work page

  11. [11]

    The final answer provided by the baseline model is incorrect, as opposed to the correct answer given in Figure 6

    Find the weekly revenue by multiplying daily revenue by the number of days in a week: Revenue per week = Revenue per day × 7 Revenue per week = 1,134 × 7 = 7 ,938 dollars Final Answer: \boxed{7,938 dollars} Figure 7: GSM8K reasoning trace of the baseline model for the same question as in Figure 6. The final answer provided by the baseline model is incorre...

    work page

  12. [12]

    Calculate the number of shells found by team Alphas: 40% of 700 = 0 .4 × 700 = 280 shells

    work page

  13. [13]

    Subtract the shells found by Alphas from the total:: 700 − 280 = 420 shells remaining

    work page

  14. [14]

    Calculate the number of shells found by team The finders: 60% of 420 = 0 .6 × 420 = 252 shells

    work page

  15. [15]

    Figure 8: GSM8K reasoning trace demonstrating that the <|continue_thinking|> token does not generate many tokens when the model is confident

    Subtract the shells found by The finders from the remaining shells: 420 − 252 = 168 shells Conclusion: Team Gogetters found \boxed{168} shells. Figure 8: GSM8K reasoning trace demonstrating that the <|continue_thinking|> token does not generate many tokens when the model is confident. Blue indicates the original reasoning. Green shows the short continuati...

    work page