Pith. sign in

REVIEW 4 major objections 5 minor 29 references

Hierarchical Budget Policy Optimization for Adaptive Reasoning

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Partitioning rollouts into token-budget hierarchies lets a 1.5B reasoning model cut token use up to 60.6% while improving accuracy by 3.14 points.

desk verdict Useful empirical results undercut by an algorithm-description mismatch and missing uncertainty reporting. read the letter →

arxiv 2507.15844 v3 pith:3PZX3FDI submitted 2025-07-21 cs.AI cs.CL

Shangke Lyu , Linjuan Wu , Yuchen Yan , Xingyu Wu , Hao Li , Yongliang Shen , Peisheng Jiang , Weiming Lu
show 2 more authors
Jun Xiao Yueting Zhuang
This is my paper
classification cs.AIcs.CL
keywords hierarchicalbudgetpolicyoptimizationadaptivereasoningtokenefficiencyreinforcementlearningGRPOchain-of-thoughtlengthcontroldepth
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that a reasoning model can be made dramatically cheaper without losing capability if, during reinforcement learning, rollouts are split into token-budget groups—512, 1024, 2048, and 2560 tokens—each with its own reward function instead of a single global length penalty. The authors argue that uniform length penalties collapse the exploration space by pushing policies away from long reasoning paths, whereas a hierarchy preserves both short and long paths and lets the model learn which length fits which problem. On four math benchmarks, the approach reduces average token use by up to 60.6% while improving accuracy by 3.14 percentage points on one 1.5B base model, and matches accuracy with 50.2% fewer tokens on another. It also reports emergent adaptation: one trained model spends roughly 670 tokens on a grade-school math benchmark but about 5,606 tokens on contest-level AIME problems, with no explicit difficulty classifier. A reader should care because the result suggests token efficiency and reasoning quality need not trade off if training preserves diverse reasoning lengths.

What carries the argument

The load-bearing machinery is the budget hierarchy itself: rollout samples are partitioned into $k$ subgroups with budgets $b_i \in \{512,1024,2048,2560\}$ tokens, implemented by appending the prompt "I will answer the question within $b_i$ tokens" after the reasoning tag. Each subgroup uses the piecewise reward $R(n_{\mathrm{gen}}|b)$: a correct response within budget receives the constant ceiling $f_2(b)$, and a correct response over budget receives the cosine-decay-minus-deviation value $f_1(n_{\mathrm{gen}},b)$, with incorrect responses getting zero. Training then uses two advantage signals: the intra-subgroup advantage $\mu_i - R_{b_i}$ compares a subgroup's mean reward to its budget baseline, while the inter-subgroup advantage $(R_{i,j}-\bar{R})/\mathrm{std}(R)$ compares each response to the global mean. The combination is what lets the model compare reasoning paths of different lengths and learn which budget level a problem warrants.

What would settle it

An ablation that trains with the same budget-aware reward but omits the budget prompt would settle the claim: if per-benchmark token counts become uniform or accuracy falls back to the length-penalty baseline, the adaptive behavior is a response to the prompt rather than a learned reasoning-depth policy.

Watch

Extended reading notes

Core claim

The central claim is that efficiency and reasoning capability can be optimized together by replacing a single length penalty with a hierarchy of budget-constrained exploration spaces. For each query, HBPO draws 16 responses split into four subgroups with budget prompts stating the token allowance, and scores each response with a piecewise reward: correct answers inside the budget get the ceiling value $f_2(b)=\beta\cos(\pi b/2L_{\max})$, which is non-decreasing and preserves exploration; correct answers that exceed the budget get $f_1(n_{\mathrm{gen}},b)=\beta\cos(\pi n_{\mathrm{gen}}/2L_{\max})-\alpha|n_{\mathrm{gen}}-b|$, which pushes the response back toward its budget. Advantages are decomposed into an intra-subgroup term comparing against the budget baseline and an inter-subgroup term comparing against the global mean, so the policy learns both to be efficient within a budget and to prefer the budget level that fits the problem. On DeepSeek-R1-Distill-Qwen-1.5B the method reports 59.4% average accuracy with 3,120 tokens versus the base 56.3% with 7,921 tokens; on DeepScaleR-Preview-1.5B it reports 63.7% accuracy with 2,364 tokens versus 63.7% with 4,744. The paper attributes the resulting token-count differences across benchmarks—790 on GSM8K versus 3,988 on AIME25 for the DeepScaleR model—to emergent adaptive reasoning learned from the hierarchical rewards.

Load-bearing premise

The claim that adaptive token allocation is an emergent skill depends on the assumption that training with the inserted budget prompt teaches the model a general sense of when long reasoning is worthwhile, so that the prompt can be removed at evaluation and the model still chooses appropriate lengths.

Editorial extensions

If this is right

  • On the R1-distill 1.5B base, HBPO reports average accuracy rising from 56.3% to 59.4% while average token use falls from 7,921 to 3,120, a 60.6% reduction.
  • On the DeepScaleR 1.5B base, accuracy stays at 63.7% while token use falls from 4,744 to 2,364, a 50.2% reduction, and AIME25 accuracy rises from 30.0% to 31.1%.
  • When explicitly prompted to minimize tokens, the HBPO-trained DeepScaleR model keeps 59.4% average accuracy with 947 tokens, whereas the untrained base model drops to 51.7% under the same prompting.
  • The policy allocates tokens by difficulty in natural reasoning, using 790 tokens on GSM8K versus 3,988 on AIME25 for the DeepScaleR model, with no explicit difficulty labels.
  • The number of budget levels matters: four budgets outperform one, two, six, and eight budgets at the same average budget, indicating an optimum between exploration diversity and per-subgroup sample size.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported benchmark-level token counts could be driven partly by dataset style cues, such as the model recognizing the genre of a problem, rather than by a genuine estimate of difficulty; testing on a single mixed-difficulty dataset would separate those explanations.
  • The same hierarchical reward design could be applied to other controllable resources, such as number of reasoning steps, tool calls, or API cost, though the paper only demonstrates tokens and only on math benchmarks plus one scientific reasoning benchmark.
  • Because correct answers within a budget all receive the same reward regardless of how many tokens they used, the method leaves intra-budget compression on the table; an extension rewarding shorter correct answers within each budget could push token counts lower still.
  • The phrase 'emergent adaptive behavior' is the paper's interpretation; the mechanism is better described as learned budget selection driven by inter-subgroup advantage, and whether that counts as emergence depends on how strictly one defines emergence.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Hierarchical Budget Policy Optimization (HBPO), a GRPO-style RL method that trains a reasoning model with rollouts partitioned into token-budget subgroups (512, 1024, 2048, 2560 tokens). A piecewise reward assigns a constant reward for correct within-budget responses, cosine decay with a deviation penalty for over-budget correct responses, and zero otherwise. The authors claim that this hierarchical structure preserves exploration diversity, teaches within-budget efficiency, and enables the model to learn problem-specific reasoning depth. On four math benchmarks with two 1.5B base models, they report up to 60.6% token reduction with a 3.1-point average accuracy gain, and they interpret the varying per-benchmark token counts as emergent adaptive behavior.

Significance. If the empirical results hold, HBPO would be a practically relevant contribution to efficient reasoning: it is simple, builds on the standard GRPO framework, compares against many recent baselines, and the authors point to a public repository. The paper also identifies a real problem, namely that uniform length penalties can collapse exploration in RL-trained reasoning models. However, the significance is conditional because the described intra-budget learning mechanism is not implemented in the pseudocode, key reward hyperparameters are missing, the evaluation lacks any variance or significance measures, and no ablation isolates the budget prompt from the reward shaping. Those issues prevent the current version from establishing the central claims.

major comments (4)
  1. [§3.3, Algorithm 1, Eq. (5)] Algorithm 1 computes Aintra_i = μ_i − R_bi as a scalar for the whole subgroup (lines 10–12) and then forms A_i,j = Aintra_i + Ainter_i,j. Thus the intra-subgroup term adds the same constant to every response in the subgroup and cannot 'measure how well responses perform relative to their budget expectation' for individual responses. For all correct responses with n_gen ≤ b, Equation (1) assigns the same reward f2(b), and Equation (3) sets R_bi = f2(b); hence all such responses receive identical advantages regardless of how much shorter they are than the budget. The policy therefore receives no within-budget efficiency signal for correct answers, which contradicts the mechanism described in §3.2 and the interpretation of Table 3 in §5.1. If the implementation instead uses a per-response intra-subgroup advantage, the paper misdescribes the method and the experiments cannot be reproduced from the pseudocode.
  2. [§3.2, §4.1] The reward functions in Equations (2) and (3) contain two free hyperparameters, α and β, but their values are never reported. The reward shape is a central design choice: it determines the relative attractiveness of different budgets and the strength of the over-budget penalty. Without α and β, and without a sensitivity analysis over these parameters, the main efficiency-accuracy results in Tables 1 and 2 are not reproducible, and the reader cannot assess how robust the reported trade-offs are to reward shaping.
  3. [§4.2, Tables 1–5] All results are reported from what appears to be a single training run and a single evaluation pass, with no error bars, no multiple seeds, and no significance tests. Several headline comparisons are small (e.g., 63.7% vs. 61.3% average accuracy over AutoThink in Table 1; a 0.8-point drop between the 4-budget and 6-budget configurations in Table 3), so without variance estimates these differences are not statistically grounded. The paper should report at least three seeds with means and standard deviations, or justify why the trends are stable.
  4. [§3.1, §4.2, Tables 1–2] The training procedure conditions generation on budget prompts such as 'I will answer the question within b_i tokens', but natural-reasoning evaluation removes these prompts. No ablation varies the prompt and the reward independently, so the paper does not establish whether the observed token allocation comes from the hierarchical reward, from prompt conditioning during training, or from a mixture of both. In addition, the base models already allocate tokens unevenly across benchmarks (e.g., DeepScaleR baseline uses 1,684 tokens on GSM8K and 9,023 on AIME25 in Table 1), so the claim of 'emergent adaptive behavior' should be quantified as a change in the token-allocation pattern relative to the base prior, not merely as the existence of per-benchmark variation.
minor comments (5)
  1. [§3] The introduction to Section 3 contains typographical errors with doubled parentheses: '(Section 3.2))' and '(Section 3.3))'.
  2. [§3.1 and Algorithm 1] The prompt wording is inconsistent: Section 3.1 writes 'I will answer the question within b_i tokens' while Algorithm 1 line 5 writes 'I will answer within b_i tokens'. Please make the exact prompt text consistent.
  3. [Abstract and Table 1] The abstract states that HBPO improves accuracy 'by 3.14%', but Table 1 shows an increase of 3.1 percentage points on one base model and no average accuracy change on the other. Please express this as percentage points rather than percent relative gain, or clarify the calculation.
  4. [§4.1, Table 4] The baseline 'Classic Reward' in Table 4 is not defined in the experimental setup. Please specify what reward function it uses and how it differs from 'Cosine Reward'.
  5. [References] The AdaR1 entry in the baselines list contains a malformed citation '( ?)'. The reference list should be completed and checked for consistency.

Circularity Check

2 steps flagged · score 4.0 of 10

No load-bearing self-citation or fitted-input-as-prediction exists; however, the 'emergent adaptive behavior' headline restates the reward's designed cross-budget preference, and Algorithm 1's per-group constant advantage removes the claimed within-budget efficiency signal by construction, so the paper merits a moderate partial-circularity score of 4.

  1. self definitional [Abstract; Section 3.2.2 (Eq. 1-3); Section 4.2]
    "HBPO exhibits emergent adaptive behavior where models automatically adjust reasoning depth based on problem complexity ... When ngen < min(bi), all budgets yield rewards determined by f2, and smaller budgets receive higher rewards due to the monotonic decrease of the cosine function over the interval. This preference for smaller budgets on short responses encourages efficiency for simple problems."

    The headline 'emergent adaptive behavior' is the direct target of the reward engineered in Section 3.2.2: Eq. 1-3 assign higher f2 rewards to short responses under small budgets, and larger budgets are favored as length grows through smaller deviation penalties in f1. The reward therefore constructs the problem-complexity-to-length mapping that the paper later reports as a discovered phenomenon. The test-time token counts (670 tokens on GSM8K vs. 5,606 on AIME25) are the empirical realization of this designed gradient, so the claimed adaptivity is built into the input reward by construction, and calling it 'emergent' relabels the objective as a result.

  2. other [Algorithm 1 (lines 10-15); Eq. 1; Section 3.3]
    "Compute intra-subgroup advantage: Aintra i = µi − Rbi ... Normalize final advantage: Ai,j = Aintra i + Ainter i,j ... This term encourages optimization within each budget constraint, teaching the model to reason efficiently given a specific token allocation."

    By Algorithm 1, Aintra_i is one scalar per subgroup (no j index), and by Eq. 1 every correct response with ngen ≤ b receives reward f2(b) = Rbi. Therefore every correct within-budget response in subgroup i has the same final advantage, because Ainter_i,j = (Ri,j − global mean)/std(R) is also identical for such responses. A 100-token and a 900-token correct answer under the same budget are indistinguishable to the optimizer, so no within-budget token-efficiency gradient exists under the pseudocode. The claimed derivation of efficient reasoning from the hierarchical advantage is thus constant by the paper's own equations; only crossing the budget (f1) carries a length signal.

full rationale

Most of this paper's derivation chain is empirical and self-contained: the reward (Eq. 1-3) is a fixed design choice, not fitted to the evaluation benchmarks; the token-reduction and accuracy numbers are measured after training on the reported benchmarks; and ablations (Tables 3-4) provide independent evidence that hierarchy granularity and the budget-aware reward matter. There are no load-bearing self-citations and no uniqueness theorems imported from prior work, so the self-citation and imported-uniqueness patterns do not apply. The manuscript also contains no explicit limitation passage; the concerns below arise from internal inconsistencies between the pseudocode and the prose claims. Two circularity-adjacent issues are flagged. First, the headline 'emergent adaptive behavior' is the reward's designed objective relabeled: Section 3.2.2 explicitly engineers the cross-budget preference (small budgets reward short responses more; large budgets avoid deviation penalties on long ones), so the adaptive token allocation is constructed into the input rather than discovered independently; the word 'emergent' overstates this, although the token counts themselves are genuinely measured. Second, the claimed within-budget efficiency learning is not implemented as written: Algorithm 1 defines Aintra_i as a per-subgroup constant and Eq. 1 gives all correct within-budget responses the identical reward f2(b), so the intra-budget advantage cannot distinguish a short from a long correct answer, making the mechanistic explanation for the hierarchy's benefit (Sections 3.3 and the Table 3 discussion) constant by construction. Because the main quantitative claims remain genuine empirical outcomes with independent ablations and external benchmarks, this is partial circularity rather than full reduction, reflected in a score of 4 rather than 6 or higher.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central claim rests on standard RL machinery (GRPO), a hand-designed reward with unreported alpha and beta, and a manual choice of four budget levels. The paper does not introduce new theoretical entities. The main unstated assumptions are that one epoch of training transfers and that the budget prompts generalize to unprompted inference.

free parameters (4)
  • alpha (α) = not reported
    Deviation sensitivity in Eq. 2 for over-budget penalty; hand-chosen, value omitted.
  • beta (β) = not reported
    Scaling factor in Eq. 2-3; hand-chosen, value omitted.
  • Budget set B = {512, 1024, 2048, 2560}
    Four levels manually selected; sensitivity analysis only varies the number of budgets while keeping average 1536.
  • Lmax training context = 4096 tokens
    Maximum generation length during training; affects cosine decay scaling and truncation.
assumptions (3)
  • standard math Group Relative Policy Optimization (GRPO) with the clipped surrogate objective (Eq. 6) is a valid policy update rule that improves the expected reward.
    Invoked in Section 3.3, the method inherits GRPO's objective without proof; standard in the RLHF literature.
  • domain assumption The reward function defined in Eq. 1-3 creates the intended preference ordering: short correct answers are preferred under small budgets and long correct answers under large budgets.
    Section 3.2 claims the piecewise structure creates a natural gradient of incentives; this is an assumption about how the reward shapes policy, not proven.
  • domain assumption The DeepScaleR training set (40K math problems) is representative enough that one epoch of training (629 steps) yields a generalizable reasoning policy for the four evaluation benchmarks.
    Section 4.1 describes the training setup; the paper assumes a single epoch is sufficient for the adaptive behavior to emerge and transfer.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Hierarchical Budget Policy Optimization for Adaptive Reasoning." pith.science (2026). https://pith.science/paper/3PZX3FDI

@misc{pith2026250715844,
  author       = {Pith},
  title        = {Pith review of: Hierarchical Budget Policy Optimization for Adaptive Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3PZX3FDI}},
  note         = {Machine review of arXiv:2507.15844}
}
read the original abstract

Large reasoning models achieve remarkable performance through extensive chain-of-thought generation, yet they suffer from a critical inefficiency: applying uniformly extensive reasoning regardless of problem complexity. We present Hierarchical Budget Policy Optimization (HBPO), a reinforcement learning framework that enables models to learn problem-specific reasoning depths without sacrificing capability. Unlike existing approaches that impose rigid constraints or rely on discrete mode selection, HBPO partitions the exploration space into budget-constrained hierarchies (512-2560 tokens), each with differentiated reward structures that preserve both efficiency incentives and reasoning capabilities. This design addresses a fundamental challenge in efficient reasoning training: traditional length penalties systematically bias models away from necessary long reasoning paths, causing exploration space collapse. Through hierarchical sampling and budget-aware rewards, HBPO maintains exploration diversity while teaching models to recognize when extended deliberation is warranted. Extensive experiments demonstrate that HBPO reduces average token usage by up to 60.6% while improving accuracy by 3.14% across four reasoning benchmarks. Most notably, HBPO exhibits emergent adaptive behavior where models automatically adjust reasoning depth based on problem complexity. Our results suggest that reasoning efficiency and capability are not inherently conflicting, and can be simultaneously optimized through appropriately structured hierarchical training that preserves exploration diversity.

Figures

Figures reproduced from arXiv: 2507.15844 by the authors.

Figure 1
Figure 1. HBPO provides budget-aware reward through hierarchical budget exploration, which [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of Hierarchical Budget Policy Optimization. Given a query, HBPO generates [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Training dynamics. (Left) Entropy Comparison of different hierarchical structures. (Right) [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Reasoning pattern analysis across methods and problem difficulties. Thinking proportions [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

29 extracted references · 3 canonical work pages

  1. [1]

    L1: controlling how long A reasoning model thinks with reinforcement learning

    Pranjal Aggarwal and Sean Welleck. L1: controlling how long A reasoning model thinks with reinforcement learning. CoRR, abs/2503.04697,

    arXiv

  2. [3]

    URL https://doi.org/10

    doi: 10.48550/ARXIV .2502.04463. URL https://doi.org/10. 48550/arXiv.2502.04463. Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng, Jiannan Guan, Peng Wang, Mengkang Hu, Yuhang Zhou, Te Gao, and Wanxiang Che. Towards reasoning era: A survey of long chain- of-thought for reasoning large language models. CoRR, abs/2503.09567,

    doi:10.48550/arxiv.2502.04463

  3. [4]

    URLhttps://doi.org/10.48550/arXiv.2503.09567

    doi: 10.48550/ ARXIV .2503.09567. URLhttps://doi.org/10.48550/arXiv.2503.09567. Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. Do NOT think that much for 2+3=? on the overthinking of o1-like llms. CoRR, abs/2412.21187,

    arXiv doi:10.48550/arxiv.2503.09567

  4. [7]

    URL https://doi.org/ 10.48550/arXiv.2501.12948

    doi: 10.48550/ARXIV .2501.12948. URL https://doi.org/ 10.48550/arXiv.2501.12948. Gongfan Fang, Xinyin Ma, and Xinchao Wang. Thinkless: LLM learns when to think. CoRR, abs/2505.13379,

    arXiv doi:10.48550/arxiv.2501.12948

  5. [8]

    URL https://doi.org/10

    doi: 10.48550/ARXIV .2505.13379. URL https://doi.org/10. 48550/arXiv.2505.13379. Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, Zhengyang Tang, Benyou Wang, Daoguang Zan, Shanghaoran Quan, Ge Zhang, Lei Sha, Yichang Zhang, Xuancheng Ren, Tianyu Liu, and Baobao Chang. Omni-math: A universal...

    arXiv doi:10.48550/arxiv.2505.13379

  6. [9]

    Thinkprune: Pruning long chain-of-thought of llms via reinforcement learning

    Bairu Hou, Yang Zhang, Jiabao Ji, Yujian Liu, Kaizhi Qian, Jacob Andreas, and Shiyu Chang. Thinkprune: Pruning long chain-of-thought of llms via reinforcement learning. CoRR, abs/2504.01296,

    arXiv

  7. [10]

    URL https://doi.org/10

    doi: 10.48550/ARXIV .2504.01296. URL https://doi.org/10. 48550/arXiv.2504.01296. Chengyu Huang, Zhengxin Zhang, and Claire Cardie. HAPO: training language models to reason concisely via history-aware policy optimization. CoRR, abs/2505.11225,

    arXiv doi:10.48550/arxiv.2504.01296

  8. [11]

    URLhttps://doi.org/10.48550/arXiv.2505.11225

    doi: 10.48550/ ARXIV .2505.11225. URLhttps://doi.org/10.48550/arXiv.2505.11225. Lingjie Jiang, Xun Wu, Shaohan Huang, Qingxiu Dong, Zewen Chi, Li Dong, Xingxing Zhang, Tengchao Lv, Lei Cui, and Furu Wei. Think only when you need with large hybrid-reasoning models. CoRR, abs/2505.14631,

    doi:10.48550/arxiv.2505.11225

Show all 29 references
  1. [12]

    URL https:// doi.org/10.48550/arXiv.2505.14631

    doi: 10.48550/ARXIV .2505.14631. URL https:// doi.org/10.48550/arXiv.2505.14631. Zheng Li, Qingxiu Dong, Jingyuan Ma, Di Zhang, and Zhifang Sui. Selfbudgeter: Adaptive token allocation for efficient LLM reasoning. CoRR, abs/2505.11274,

    doi arXiv

  2. [13]

    2505.11274

    doi: 10.48550/ARXIV . 2505.11274. URL https://doi.org/10.48550/arXiv.2505.11274. Guosheng Liang, Longguang Zhong, Ziyi Yang, and Xiaojun Quan. Thinkswitcher: When to think hard, when to think fast. CoRR, abs/2505.14183,

    doi arXiv

  3. [14]

    URL https://doi.org/10.48550/arXiv.2505.14183

    doi: 10.48550/ARXIV .2505.14183. URL https://doi.org/10.48550/arXiv.2505.14183. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step,

    doi arXiv

  4. [16]

    URL https://doi.org/10.48550/arXiv.2505.11896

    doi: 10.48550/ARXIV .2505.11896. URL https://doi.org/10.48550/arXiv.2505.11896. Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Tang, Manan Roongta, Colin Cai, Jef- frey Luo, Tianjun Zhang, Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing o1- previ...

    doi arXiv

  5. [17]

    Wenjie Ma, Jingxuan He, Charlie Snell, Tyler Griggs, Sewon Min, and Matei Zaharia

    Notion Blog. Wenjie Ma, Jingxuan He, Charlie Snell, Tyler Griggs, Sewon Min, and Matei Zaharia. Reasoning models can be effective without thinking. CoRR, abs/2504.09858,

    arXiv

  6. [18]

    2504.09858

    doi: 10.48550/ARXIV . 2504.09858. URL https://doi.org/10.48550/arXiv.2504.09858. 12 Preprint Yingqian Min, Zhipeng Chen, Jinhao Jiang, Jie Chen, Jia Deng, Yiwen Hu, Yiru Tang, Jiapeng Wang, Xiaoxue Cheng, Huatong Song, Wayne Xin Zhao, Zheng Liu, Zhongyuan Wang, and Ji- Rong We...

    doi arXiv

  7. [19]

    URL https: //doi.org/10.48550/arXiv.2412.09413

    doi: 10.48550/ARXIV .2412.09413. URL https: //doi.org/10.48550/arXiv.2412.09413. Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel J. Cand`es, and Tatsunori Hashimoto. s1: Simple test- time scal...

    doi arXiv

  8. [20]

    URL https: //doi.org/10.48550/arXiv.2501.19393

    doi: 10.48550/ARXIV .2501.19393. URL https: //doi.org/10.48550/arXiv.2501.19393. OpenAI. Learning to reason with llms. OpenAI Blog ,

    doi arXiv

  9. [21]

    Accessed: 2025-07-22

    URL https://openai.com/ index/learning-to-reason-with-llms/ . Accessed: 2025-07-22. Ziqing Qiao, Yongheng Deng, Jiali Zeng, Dong Wang, Lai Wei, Fandong Meng, Jie Zhou, Ju Ren, and Yaoxue Zhang. Concise: Confidence-guided compression in step-by-step efficient reasoning. CoRR, a...

    2025

  10. [22]

    URL https://doi.org/ 10.48550/arXiv.2505.04881

    doi: 10.48550/ARXIV .2505.04881. URL https://doi.org/ 10.48550/arXiv.2505.04881. Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19256,

    doi

  11. [23]

    Learning when to think: Shaping adaptive reasoning in r1-style models via multi-stage RL

    Songjun Tu, Jiahao Lin, Qichao Zhang, Xiangyu Tian, Linjing Li, Xiangyuan Lan, and Dongbin Zhao. Learning when to think: Shaping adaptive reasoning in r1-style models via multi-stage RL. CoRR, abs/2505.10832,

  12. [24]

    URL https://doi.org/ 10.48550/arXiv.2505.10832

    doi: 10.48550/ARXIV .2505.10832. URL https://doi.org/ 10.48550/arXiv.2505.10832. Yi Wang, Junxiao Liu, Shimao Zhang, Jiajun Chen, and Shujian Huang. PATS: process-level adaptive thinking mode switching. CoRR, abs/2505.19250,

    doi

  13. [25]

    doi: 10.48550/ARXIV .2505. 19250. URL https://doi.org/10.48550/arXiv.2505.19250. Siye Wu, Jian Xie, Yikai Zhang, Aili Chen, Kai Zhang, Yu Su, and Yanghua Xiao. ARM: adaptive reasoning model. CoRR, abs/2505.20258,

    doi arXiv

  14. [26]

    URL https://doi.org/10.48550/arXiv.2505.20258

    doi: 10.48550/ARXIV .2505.20258. URL https://doi.org/10.48550/arXiv.2505.20258. Yuhui Xu, Hanze Dong, Lei Wang, Doyen Sahoo, Junnan Li, and Caiming Xiong. Scalable chain of thoughts via elastic reasoning. CoRR, abs/2505.05315,

    doi

  15. [27]

    URL https://doi.org/10.48550/arXiv.2505.05315

    doi: 10.48550/ARXIV .2505.05315. URL https://doi.org/10.48550/arXiv.2505.05315. Chenxu Yang, Qingyi Si, Yongjie Duan, Zheliang Zhu, Chenyu Zhu, Zheng Lin, Li Cao, and Weiping Wang. Dynamic early exit in reasoning models. CoRR, abs/2504.15895,

    doi arXiv

  16. [28]

    URLhttps://doi.org/10.48550/arXiv.2504.15895

    48550/ARXIV .2504.15895. URLhttps://doi.org/10.48550/arXiv.2504.15895. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, ...

    doi

  17. [29]

    URLhttps://doi.org/10.48550/arXiv.2503.14476

    48550/ARXIV .2503.14476. URLhttps://doi.org/10.48550/arXiv.2503.14476. Jiajie Zhang, Nianyi Lin, Lei Hou, Ling Feng, and Juanzi Li. Adaptthink: Reasoning models can learn when to think. CoRR, abs/2505.13417, 2025a. doi: 10.48550/ARXIV .2505.13417. URL https://doi.org/10.48550/...

    doi arXiv

  18. [2021]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. CoRR, abs/2501.12948,

    arXiv

  19. [2023]

    Adacot: Pareto-optimal adaptive chain-of-thought triggering via reinforcement learning

    Chenwei Lou, Zewei Sun, Xinnian Liang, Meng Qu, Wei Shen, Wenqi Wang, Yuntao Li, Qingping Yang, and Shuangzhi Wu. Adacot: Pareto-optimal adaptive chain-of-thought triggering via reinforcement learning. CoRR, abs/2505.11896,

    arXiv

  20. [2024]

    URL https://doi.org/10.48550/arXiv.2412

    doi: 10.48550/ARXIV .2412.21187. URL https://doi.org/10.48550/arXiv.2412. 21187. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John 11 Preprint Schul...

    doi arXiv

  21. [2025]

    URL https://doi.org/10.48550/arXiv.2503.04697

    doi: 10.48550/ARXIV .2503.04697. URL https://doi.org/10.48550/arXiv.2503.04697. Daman Arora and Andrea Zanette. Training language models to reason efficiently. CoRR, abs/2502.04463,

    doi arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.