Pith. sign in

REVIEW 3 major objections 5 minor 13 cited by

This paper argues that the reasoning bottleneck in LLMs is not over-thinking but tunnel vision, and that parallel thought paths scale better than longer sequential thought.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

ParaThinker trains LLMs for native parallel reasoning and reports 7 to 12 percent higher accuracy on math benchmarks over sequential thinking with modest latency overhead.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection A promising parallel-reasoning architecture whose headline width-beats-depth claim is not yet backed by adequate statistics. the 3 major comments →

arxiv 2509.04475 v1 pith:XTAQ4HDN submitted 2025-08-30 cs.CL cs.AI

ParaThinker: Native Parallel Thinking as a New Paradigm to Scale LLM Test-time Compute

Hao Wen , Yifan Su , Feifei Zhang , Yunxin Liu , Yunhao Liu , Ya-Qin Zhang , Yuanchun Li This is my paper
classification cs.CL cs.AI
keywords parallel reasoningtest-time compute scalingchain-of-thoughttunnel visionsupervised fine-tuningthought diversityKV cache reusemathematical reasoning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ParaThinker claims that a reasoning LLM's test-time scaling ceiling comes not from its underlying capability but from how the extra compute is spent: once a chain of thought commits to early tokens, later tokens rarely escape the chosen path—a failure the paper calls Tunnel Vision. The paper's fix is to train the model to generate several independent reasoning paths at once and then synthesize them into one answer, turning test-time scaling from a depth game into a width game. On AIME 2024, AIME 2025, AMC 2023, and MATH-500, this parallel training lifts a 1.5B model by 12.3% and a 7B model by 7.5% on average over sequential reasoning with eight paths, at only 7.1% extra latency. If true, this gives a practical, hardware-friendly way to get more reasoning out of the same model, and lets smaller models overtake larger ones.

Core claim

On its own terms, the paper establishes a new scaling direction: native thought parallelism. It first shows that on AIME 2024, a single sequential chain of thought saturates as its token budget grows, while majority voting over the same total budget keeps climbing, and that continuing a solution from an erroneous prefix decreases accuracy the longer the prefix is. It then builds ParaThinker, which supervises the model to emit P distinct reasoning paths in one forward pass—each opened by a learned <think i> token and separated by thought-specific positional embeddings—and then to read all P paths from their cached key-values and summarize them into a final answer. With eight parallel paths, P

What carries the argument

The load-bearing mechanism is a two-stage attention design inside an otherwise standard transformer. Trainable control tokens (<think i>, </think i>, <summary>) mark each reasoning stream; a learned thought-specific embedding T^(j) is added to the key and value of every token in path j before RoPE rotation, so the summarizer can tell which stream a token came from without inflating positional indices; and a two-phase attention mask blocks all inter-path attention during reasoning while allowing full attention over every path during summarization. Because the parallel paths share the prompt's KV cache and are decoded as one batch, the extra paths add little latency, and the summarizer reuses

Load-bearing premise

The fine-tuning corpus is assembled from public reasoning datasets with no reported overlap analysis against the evaluation benchmarks, so the reported gains could partly come from the model having seen the test problems during training.

What would settle it

Take a fresh set of hard competition-style math problems written after the training data was collected, run ParaThinker with 1 vs 8 paths under equal total token budgets, and check whether the 8-path advantage persists; if it disappears, the central scaling claim is not about parallel thinking but about data overlap.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Training a model to synthesize parallel thoughts is a viable alternative to majority voting for tasks without a single verifiable answer, since it requires no external verifier.
  • The accuracy-vs-token-budget curve keeps improving with path count after sequential scaling has plateaued, so width is a new axis along which test-time scaling laws should be drawn.
  • Because decoding is memory-bandwidth-bound, batching several paths costs far less than linearly more time, making parallel reasoning affordable at deployment.
  • ParaThinker and majority voting compose: voting over ParaThinker's paths gives further gains (up to 66.7% on AIME 2024), so the method does not replace self-consistency but improves the candidates it votes over.
  • The same SFT recipe transfers to a 7B model, suggesting the mechanism is not tied to one model size.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If Tunnel Vision is the correct mechanism, the paper's prefix-continuation experiment predicts that interventions which bias the first few tokens (e.g., different problem restatements) should change final accuracy more than interventions later in the chain; this is testable on other models.
  • Since <think i> tokens are randomly assigned during training, the paper's method treats all paths as exchangeable; an explicit diversity objective (for example, rewarding paths that disagree or cover different strategies) is a natural next step the paper does not explore.
  • The training set draws from public reasoning corpora with no reported decontamination, so the cleanest test is to evaluate a fresh set of hard problems; if the 8-path gain shrinks there, the measured advantage may be partly memorization.
  • The thought-specific positional embedding effectively learns a soft cluster identity for each path; a simpler alternative that labels paths in the prompt rather than in the key-value space would test whether this architectural choice is necessary.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes ParaThinker, an end-to-end framework that trains an LLM to produce multiple independent reasoning paths in parallel and then summarize them into a final answer. The method uses trainable control tokens (<think i>), thought-specific positional embeddings, and a two-phase attention mask. The authors argue that sequential test-time scaling suffers from 'Tunnel Vision,' where early tokens lock the model into a suboptimal path, and that scaling in width (parallel paths) is both more effective and more efficient than scaling in depth. Experiments on AIME 2024, AIME 2025, AMC 2023, and MATH-500 with Qwen-2.5 1.5B/7B models report accuracy gains over sequential decoding and majority voting, with small latency overhead.

Significance. If the quantitative claims hold, this is a useful contribution: it offers a concrete, efficiently-deployable recipe for parallel reasoning without an external verifier, and it provides a falsifiable comparison of width versus depth scaling. The release of source code and the sequential SFT control in Table 5 are strengths. The reported gains over majority voting are nontrivial, and the latency results suggest practical value. However, the central claim is currently supported only by point estimates without statistical uncertainty, and the training-data hygiene relative to the evaluation benchmarks is not documented. The core idea is sound enough to merit revision rather than rejection.

major comments (3)
  1. [§5.1, Table 1]
  2. [§5.1 Training Details]
  3. [§4.1, §5.1]
minor comments (5)
  1. [Table 2]
  2. [Table 3]
  3. [§5.1]
  4. [§5.3, Figure 4]
  5. [§5.4]

Circularity Check

1 steps flagged

Minor test-set selection on AIME 2024; no central circularity.

specific steps
  1. fitted input called prediction [Section 5.4 (Table 4) and Table 1 (Section 5.2)]
    "As shown in Table 4, the First-Finish strategy yields the best performance. ... First-Finish (Default)"

    The default First-Finish termination strategy was selected because it maximized accuracy on AIME 2024 (Table 4: P=8 gives 48.1 vs 42.5 for Last-Finish; P=4 gives 43.3 vs 37.1). The same AIME 2024 numbers are then reported in Table 1 as evidence that ParaThinker-1.5B (8x16K) reaches 48.1 and beats sequential LLMs. For this benchmark, the headline result is therefore the argmax over the strategies tried on that exact dataset, not an independent prediction. The width-over-depth claim retains independent support on AIME 2025, AMC 2023, and MATH-500, so the circularity is partial and benchmark-specific.

full rationale

The central claim—that parallel scaling (width) outperforms sequential scaling (depth)—is supported by direct benchmark measurements, not by fitting equations to the target result. ParaThinker's components (control tokens, thought embeddings, attention masks, SFT pipeline) are architectural and training choices; they are not derived from the conclusion. The 'Tunnel Vision' effect is an empirical prefix-sensitivity observation, and the comparison with majority voting plus the Table 5 data ablation provide independent controls. The only circularity-adjacent issue is the termination strategy: Table 4 selects First-Finish by maximizing AIME 2024 accuracy, and Table 1 uses that same AIME 2024 performance as evidence. This is a test-set selection for one benchmark, partially inflating that specific number, but it does not collapse the whole argument because other benchmarks and the qualitative scaling trend are not used for that selection. Self-citations (e.g., Sun et al. 2025, Wen et al. 2025) appear in background/related-work statements and are not load-bearing. No uniqueness theorems are imported from the authors' prior work, and no known result is merely renamed. Hence no significant central circularity; score reflects the one benchmark-specific fitted selection.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 2 invented entities

The central claim relies on data quality and cleanliness, the correctness of the two-phase attention design, and the representativeness of the base models. The key free choices are temperature, path count, and token budget; the key domain assumptions are generalization from qwen-distill models and absence of data contamination.

free parameters (3)
  • Sampling temperature = 0.5 (1.5B), 0.6 (7B)
    Chosen by hand; affects pass@1 estimates and path diversity; not swept in main results.
  • Number of parallel paths P = up to 8 at test time; trained with P sampled from {2,4,6}
    The headline accuracy gains are reported for P=8; scaling beyond 8 is extrapolation.
  • Per-path token budget B = 8K to 128K
    The comparison between parallel and sequential methods is budget-dependent; at small B parallel can be worse (Table 2).
axioms (5)
  • domain assumption DeepSeek-R1-distill-Qwen-1.5B and 7B are representative of current reasoning LLMs.
    Results are expected to generalize to other reasoning LLMs; no cross-architecture validation is provided.
  • domain assumption The SFT training data does not overlap with the evaluation benchmarks.
    No decontamination is reported; if overlap exists, accuracy gains may be inflated. Section 5.1 lists training sources and evaluation sets separately without overlap analysis.
  • domain assumption The two-phase attention mask and First-Finish termination preserve path diversity.
    The design assumes truncation at the first completed path does not harm reasoning quality; Section 4.3 and Section 5.4 provide only a single ablation.
  • domain assumption The teacher model gpt-oss-20b produces diverse, high-quality solutions.
    Six paths per problem are sampled from the teacher; if teacher paths are homogeneous, the student may not learn diversity.
  • standard math RoPE algebra used in Eq. 7 is valid.
    Standard rotary position embedding decomposition; assumed correct.
invented entities (2)
  • Learnable control tokens <think i> no independent evidence
    purpose: Trigger distinct reasoning paths and delimit summarization
    Internal model tokens; only supported by the paper's own benchmarks.
  • Thought-specific positional embeddings T^(j) no independent evidence
    purpose: Disambiguate parallel reasoning paths during summarization
    Learned per-path vectors; no external evidence outside ParaThinker.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of ParaThinker: Native Parallel Thinking as a New Paradigm to Scale LLM Test-time Compute." pith.science (2026). https://pith.science/paper/XTAQ4HDN

@misc{pith2026250904475,
  author       = {Pith},
  title        = {Pith review of: ParaThinker: Native Parallel Thinking as a New Paradigm to Scale LLM Test-time Compute},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XTAQ4HDN}},
  note         = {Machine review of arXiv:2509.04475}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Recent advances in Large Language Models (LLMs) have been driven by test-time compute scaling - a strategy that improves reasoning by generating longer, sequential thought processes. While effective, this approach encounters a significant bottleneck as computation increases, where further computation offers only marginal performance gains. We argue this ceiling is not an inherent limit of the model's capability but a flaw in the scaling strategy itself, a phenomenon we term "Tunnel Vision", where a model's imperfect initial steps lock it into a suboptimal reasoning path. To overcome this, we introduce a new scaling paradigm: native thought parallelism. We present ParaThinker, an end-to-end framework that trains an LLM to generate multiple, diverse reasoning paths in parallel and synthesize them into a superior final answer. By exploring different lines of thoughts simultaneously, ParaThinker effectively sidesteps the Tunnel Vision issue and unlocks the model's latent reasoning potential. Our approach demonstrates that scaling compute in parallel (width) is a more effective and efficient way to superior reasoning than simply scaling sequentially (depth). On challenging reasoning benchmarks, ParaThinker achieves substantial accuracy improvements over sequential LLMs (12.3% for 1.5B and 7.5% for 7B models on average with 8 parallel paths), while adding only negligible latency overhead (7.1%). This enables smaller models to surpass much larger counterparts and establishes parallel thinking as a critical, efficient dimension for scaling future LLMs.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 13 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. LLMs Improving LLMs: Agentic Discovery for Test-Time Scaling

    cs.CL 2026-05 conditional novelty 8.0

    AutoTTS discovers width-depth test-time scaling controllers through agentic search in a pre-collected trajectory environment, yielding better accuracy-cost tradeoffs than hand-designed baselines on math reasoning task...

  2. LaneRoPE: Positional Encoding for Collaborative Parallel Reasoning and Generation

    cs.AI 2026-05 unverdicted novelty 7.0

    LaneRoPE adds an inter-sequence attention mask and extended RoPE to enable collaborative parallel sequence generation in LLMs, yielding accuracy gains on math reasoning under length limits.

  3. LLMs Improving LLMs: Agentic Discovery for Test-Time Scaling

    cs.CL 2026-05 unverdicted novelty 7.0

    AutoTTS discovers superior test-time scaling strategies for LLMs via cheap controller synthesis in a pre-collected trajectory environment, outperforming manual baselines on math benchmarks with low discovery cost.

  4. Visual Para-Thinker: Divide-and-Conquer Reasoning for Visual Comprehension

    cs.CV 2026-02 unverdicted novelty 7.0

    Visual Para-Thinker is the first parallel reasoning framework for MLLMs that uses visual partitioning strategies, Pa-Attention, and LPRoPE to extend test-time scaling benefits to visual comprehension tasks.

  5. Fishing Out Free Riders: Shapley-Based Reward Attribution for Parallel Reasoning via Reinforcement Learning

    cs.AI 2026-07 conditional novelty 6.0

    Rewarding each parallel reasoning path by Monte-Carlo-Shapley marginal contribution, scored by a generative reward model, lifts Pass@16 on AIME24/AIME25/AMC23 by 4-90% relative over Parallel-R1 with a fifth of the tra...

  6. Small RL Controller, Large Language Model: RL-Guided Adaptive Sampling for Test-Time Scaling

    cs.CL 2026-06 unverdicted novelty 6.0

    RL-trained lightweight controller using answer statistics improves trade-offs among correctness, latency, and total samples in adaptive sampling for LLM test-time scaling.

  7. OpenDeepThink: Parallel Reasoning via Bradley-Terry Aggregation

    cs.AI 2026-05 conditional novelty 6.0

    OpenDeepThink uses Bradley-Terry aggregation of LLM pairwise judgments to rank and evolve parallel reasoning traces, improving Gemini 3.1 Pro Codeforces Elo by 405 points over eight rounds.

  8. OpenDeepThink: Parallel Reasoning via Bradley-Terry Aggregation

    cs.AI 2026-05 unverdicted novelty 6.0

    OpenDeepThink improves LLM reasoning by ranking parallel candidate traces via Bradley-Terry aggregation of LLM pairwise judgments, achieving a +405 Codeforces Elo gain on Gemini 3.1 Pro after eight rounds.

  9. LACE: Lattice Attention for Cross-thread Exploration

    cs.AI 2026-04 unverdicted novelty 6.0

    LACE enables parallel reasoning paths in LLMs to communicate via lattice attention and error-correct using synthetic training data, improving accuracy by over 7 points over standard parallel search.

  10. LACE: Lattice Attention for Cross-thread Exploration

    cs.AI 2026-04 unverdicted novelty 5.0

    LACE enables concurrent reasoning paths in LLMs to interact via lattice attention and a synthetic training pipeline, raising accuracy more than 7 points over independent parallel search.

  11. LACE: Lattice Attention for Cross-thread Exploration

    cs.AI 2026-04 unverdicted novelty 5.0

    LACE adds lattice attention to let parallel LLM reasoning threads interact and correct errors, raising accuracy over 7 points versus standard independent sampling.

  12. Understanding Performance Gap Between Parallel and Sequential Sampling in Large Reasoning Models

    cs.CL 2026-04 unverdicted novelty 5.0

    Lack of exploration from conditioning on prior answers is the primary reason parallel sampling outperforms sequential sampling in large reasoning models.

  13. Evolutionary Profiles for Protein Fitness Prediction

    cs.LG 2025-10 unverdicted novelty 5.0

    EvoIF integrates within-family and cross-family evolutionary signals into a compact model to achieve competitive or state-of-the-art zero-shot fitness prediction on ProteinGym using only 0.15% of typical training data.

Reference graph

Works this paper leans on

58 extracted references · 7 canonical work pages · cited by 9 Pith papers · 1 internal anchor

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Block diffusion: Interpolating between autoregressive and diffusion language models

    Marianne Arriola, Subham Sekhar Sahoo, Aaron Gokaslan, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Justin T Chiu, and Volodymyr Kuleshov. Block diffusion: Interpolating between autoregressive and diffusion language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=tyEyYT267x

    2025

  3. [3]

    Accelerating large language model decoding with speculative sampling

    Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318, 2023

    Pith/arXiv arXiv 2023

  4. [4]

    Are more llm calls all you need? towards the scaling properties of compound ai systems

    Lingjiao Chen, Jared Davis, Boris Hanin, Peter Bailis, Ion Stoica, Matei Zaharia, and James Zou. Are more llm calls all you need? towards the scaling properties of compound ai systems. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp.\ 45767--45790...

    work page 2024

  5. [5]

    Parallel scaling law for language models, 2025 a

    Mouxiang Chen, Binyuan Hui, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Jianling Sun, Junyang Lin, and Zhongxin Liu. Parallel scaling law for language models, 2025 a . URL https://arxiv.org/abs/2505.10475

    Pith/arXiv arXiv 2025

  6. [6]

    Do not think that much for 2+3=? on the overthinking of o1-like llms, 2025 b

    Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. Do not think that much for 2+3=? on the overthinking of o1-like llms, 2025 b . URL https://arxiv.org/abs/2412.21187

    Pith/arXiv arXiv 2025

  7. [7]

    Premise order matters in reasoning with large language models

    Xinyun Chen, Ryan Andrew Chi, Xuezhi Wang, and Denny Zhou. Premise order matters in reasoning with large language models. In Forty-first International Conference on Machine Learning, 2024 b . URL https://openreview.net/forum?id=4zAHgkiCQg

    work page 2024

  8. [8]

    Gonzalez

    Alejandro Cuadron, Dacheng Li, Wenjie Ma, Xingyao Wang, Yichuan Wang, Siyuan Zhuang, Shu Liu, Luis Gaspar Schroeder, Tian Xia, Huanzhi Mao, Nicholas Thumiger, Aditya Desai, Ion Stoica, Ana Klimovic, Graham Neubig, and Joseph E. Gonzalez. The danger of overthinking: Examining the reasoning-action dilemma in agentic tasks, 2025. URL https://arxiv.org/abs/2502.08235

    Pith/arXiv arXiv 2025

  9. [9]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025. URL https://arxiv.org/abs/2501.12948

    Pith/arXiv arXiv 2025

  10. [10]

    Missing premise exacerbates overthinking: Are reasoning models losing critical thinking skill?, 2025

    Chenrui Fan, Ming Li, Lichao Sun, and Tianyi Zhou. Missing premise exacerbates overthinking: Are reasoning models losing critical thinking skill?, 2025. URL https://arxiv.org/abs/2504.06514

    Pith/arXiv arXiv 2025

  11. [11]

    Theoretical benefit and limitation of diffusion language model, 2025

    Guhao Feng, Yihan Geng, Jian Guan, Wei Wu, Liwei Wang, and Di He. Theoretical benefit and limitation of diffusion language model, 2025. URL https://arxiv.org/abs/2502.09622

    Pith/arXiv arXiv 2025

  12. [12]

    Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein

    Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R. Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up test-time compute with latent reasoning: A recurrent depth approach. CoRR, abs/2502.05171, February 2025. URL https://doi.org/10.48550/arXiv.2502.05171

    Pith/arXiv arXiv doi:10.48550/arxiv.2502.05171 2025

  13. [13]

    Does thinking more always help? understanding test-time scaling in reasoning models, 2025

    Soumya Suvra Ghosal, Souradip Chakraborty, Avinash Reddy, Yifu Lu, Mengdi Wang, Dinesh Manocha, Furong Huang, Mohammad Ghavamzadeh, and Amrit Singh Bedi. Does thinking more always help? understanding test-time scaling in reasoning models, 2025. URL https://arxiv.org/abs/2506.04210

    arXiv 2025

  14. [14]

    Gemini 2.5: Our most intelligent ai model

    Google. Gemini 2.5: Our most intelligent ai model. https://blog.google/technology/google-deepmind/gemini-model-thinking-updates-march-2025/#gemini-2-5-thinking, March 2025

    work page 2025

  15. [15]

    rstar-math: Small llms can master math reasoning with self-evolved deep thinking, 2025

    Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao Yang. rstar-math: Small llms can master math reasoning with self-evolved deep thinking, 2025. URL https://arxiv.org/abs/2501.04519

    Pith/arXiv arXiv 2025

  16. [16]

    D iffusion BERT : Improving generative masked language models with diffusion models

    Zhengfu He, Tianxiang Sun, Qiong Tang, Kuanning Wang, Xuanjing Huang, and Xipeng Qiu. D iffusion BERT : Improving generative masked language models with diffusion models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 4521-...

    doi:10.18653/v1/2023.acl-long.248 2023

  17. [17]

    Deepmath-103k: A large-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning

    Zhiwei He, Tian Liang, Jiahao Xu, Qiuzhi Liu, Xingyu Chen, Yue Wang, Linfeng Song, Dian Yu, Zhenwen Liang, Wenxuan Wang, et al. Deepmath-103k: A large-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning. arXiv preprint arXiv:2504.11456, 2025

    Pith/arXiv arXiv 2025

  18. [18]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021

    2021

  19. [19]

    Open r1: A fully open reproduction of deepseek-r1, January 2025

    Hugging Face . Open r1: A fully open reproduction of deepseek-r1, January 2025. URL https://github.com/huggingface/open-r1

    2025

  20. [20]

    Cheng, Zack Ankner, Nikunj Saunshi, Blake M

    Tian Jin, Ellie Y. Cheng, Zack Ankner, Nikunj Saunshi, Blake M. Elias, Amir Yazdanbakhsh, Jonathan Ragan-Kelley, Suvinay Subramanian, and Michael Carbin. Learning to keep a promise: Scaling language model decoding parallelism with learned asynchronous decoding, 2025. URL https://arxiv.org/abs/2502.11517

    Pith/arXiv arXiv 2025

  21. [21]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023

    2023

  22. [22]

    Bespoke-stratos: The unreasonable effectiveness of reasoning distillation

    Bespoke Labs. Bespoke-stratos: The unreasonable effectiveness of reasoning distillation. www.bespokelabs.ai/blog/bespoke-stratos-the-unreasonable-effectiveness-of-reasoning-distillation, 2025. Accessed: 2025-01-22

    2025

  23. [23]

    Fast inference from transformers via speculative decoding

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pp.\ 19274--19286. PMLR, 2023

    2023

  24. [24]

    Selfbudgeter: Adaptive token allocation for efficient llm reasoning

    Zheng Li, Qingxiu Dong, Jingyuan Ma, Di Zhang, and Zhifang Sui. Selfbudgeter: Adaptive token allocation for efficient llm reasoning. arXiv preprint arXiv:2505.11274, 2025

    Pith/arXiv arXiv 2025

  25. [25]

    Let's verify step by step, 2023

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step, 2023. URL https://arxiv.org/abs/2305.20050

    Pith/arXiv arXiv 2023

  26. [26]

    s1: Simple test-time scaling

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Cand \`e s, and Tatsunori Hashimoto. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393, 2025

    Pith/arXiv arXiv 2025

  27. [27]

    Openai o1 system card

    OpenAI. Openai o1 system card. 2024. URL https://arxiv.org/abs/2412.16720

    Pith/arXiv arXiv 2024

  28. [28]

    gpt-oss-120b & gpt-oss-20b model card, 2025

    OpenAI. gpt-oss-120b & gpt-oss-20b model card, 2025. URL https://arxiv.org/abs/2508.10925

    Pith/arXiv arXiv 2025

  29. [29]

    Learning adaptive parallel reasoning with language models, 2025

    Jiayi Pan, Xiuyu Li, Long Lian, Charlie Snell, Yifei Zhou, Adam Yala, Trevor Darrell, Kurt Keutzer, and Alane Suhr. Learning adaptive parallel reasoning with language models, 2025. URL https://arxiv.org/abs/2504.15466

    Pith/arXiv arXiv 2025

  30. [30]

    Safety alignment should be made more than just a few tokens deep

    Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, and Peter Henderson. Safety alignment should be made more than just a few tokens deep. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=6Mxhg9PtDE

    work page 2025

  31. [31]

    Yuxiao Qu, Matthew Y. R. Yang, Amrith Setlur, Lewis Tunstall, Edward Emanuel Beeching, Ruslan Salakhutdinov, and Aviral Kumar. Optimizing test-time compute via meta reinforcement fine-tuning, 2025. URL https://arxiv.org/abs/2503.07572

    Pith/arXiv arXiv 2025

  32. [32]

    Qwen2.5 technical report, 2025

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li,...

    Pith/arXiv arXiv 2025

  33. [33]

    Hogwild! inference: Parallel llm generation via concurrent attention, 2025

    Gleb Rodionov, Roman Garipov, Alina Shutova, George Yakushev, Erik Schultheis, Vage Egiazarian, Anton Sinitsin, Denis Kuznedelev, and Dan Alistarh. Hogwild! inference: Parallel llm generation via concurrent attention, 2025. URL https://arxiv.org/abs/2504.06261

    arXiv 2025

  34. [34]

    Magicdec: Breaking the latency-throughput tradeoff for long context generation with speculative decoding

    Ranajoy Sadhukhan, Jian Chen, Zhuoming Chen, Vashisth Tiwari, Ruihang Lai, Jinyuan Shi, Ian Yen, Avner May, Tianqi Chen, and Beidi Chen. Magicdec: Breaking the latency-throughput tradeoff for long context generation with speculative decoding. In Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu (eds.), International Conference on Representation Learning, volume...

    work page 2025

  35. [35]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https://arxiv.org/abs/2402.03300

    Pith/arXiv arXiv 2024

  36. [36]

    Scaling llm test-time compute optimally can be more effective than scaling model parameters, 2024

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters, 2024. URL https://arxiv.org/abs/2408.03314

    Pith/arXiv arXiv 2024

  37. [37]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 0 127063, 2024

    2024

  38. [38]

    An Empirical Study of LLM Reasoning Ability Under Strict Output Length Constraint

    Yi Sun, Han Wang, Jiaqiang Li, Jiacheng Liu, Xiangyu Li, Hao Wen, Yizhen Yuan, Huiwen Zheng, Yan Liang, Yuanchun Li, and Yunxin Liu. An empirical study of llm reasoning ability under strict output length constraint. arXiv preprint arXiv:2504.14350, April 2025. doi:10.48550/arXiv.2504.14350

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2504.14350 2025

  39. [39]

    Kimi k1.5: Scaling reinforcement learning with llms, 2025

    Kimi Team et al. Kimi k1.5: Scaling reinforcement learning with llms, 2025. URL https://arxiv.org/abs/2501.12599

    Pith/arXiv arXiv 2025

  40. [40]

    Remasking discrete diffusion models with inference-time scaling

    Guanghan Wang, Yair Schiff, Subham Sahoo, and Volodymyr Kuleshov. Remasking discrete diffusion models with inference-time scaling. arXiv preprint arXiv:2503.00307, 2025

    arXiv 2025

  41. [41]

    Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=1PL1NIMMrw

    2023

  42. [42]

    Budgetthinker: Empowering budget-aware llm reasoning with control tokens

    Hao Wen, Xinrui Wu, Yi Sun, Feifei Zhang, Liye Chen, Jie Wang, Yunxin Liu, Ya-Qin Zhang, and Yuanchun Li. Budgetthinker: Empowering budget-aware llm reasoning with control tokens. arXiv preprint arXiv:2508.17196, 2025

    Pith/arXiv arXiv 2025

  43. [43]

    Roofline: an insightful visual performance model for multicore architectures

    Samuel Williams, Andrew Waterman, and David Patterson. Roofline: an insightful visual performance model for multicore architectures. Commun. ACM, 52 0 (4): 0 65–76, April 2009. ISSN 0001-0782. doi:10.1145/1498765.1498785. URL https://doi.org/10.1145/1498765.1498785

    arXiv 2009

  44. [44]

    Sam Wiseman and Alexander M. Rush. Sequence-to-sequence learning as beam-search optimization. In Jian Su, Kevin Duh, and Xavier Carreras (eds.), Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp.\ 1296--1306, Austin, Texas, November 2016. Association for Computational Linguistics. doi:10.18653/v1/D16-1137. URL http...

    doi:10.18653/v1/d16-1137 2016

  45. [45]

    Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models, 2025

    Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models, 2025. URL https://arxiv.org/abs/2408.00724

    Pith/arXiv arXiv 2025

  46. [46]

    Redstar: Does scaling long-cot data unlock better slow-reasoning systems?, 2025

    Haotian Xu, Xing Wu, Weinong Wang, Zhongzhi Li, Da Zheng, Boyuan Chen, Yi Hu, Shijia Kang, Jiaming Ji, Yingying Zhang, Zhijiang Guo, Yaodong Yang, Muhan Zhang, and Debing Zhang. Redstar: Does scaling long-cot data unlock better slow-reasoning systems?, 2025. URL https://arxiv.org/abs/2501.11284

    Pith/arXiv arXiv 2025

  47. [47]

    Preemptive answer ``attacks'' on chain-of-thought reasoning

    Rongwu Xu, Zehan Qi, and Wei Xu. Preemptive answer ``attacks'' on chain-of-thought reasoning. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp.\ 14708--14726, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi:10.18653/v1/2024.findings-acl.876. URL ...

    work page doi:10.18653/v1/2024.findings-acl.876 2024

  48. [48]

    Qwen3 technical report, 2025 a

    An Yang, Anfeng Li, Baosong Yang, et al. Qwen3 technical report, 2025 a . URL https://arxiv.org/abs/2505.09388

    Pith/arXiv arXiv 2025

  49. [49]

    Multiverse: Your language models secretly decide how to parallelize and merge generation

    Xinyu Yang, Yuwei An, Hongyi Liu, Tianqi Chen, and Beidi Chen. Multiverse: Your language models secretly decide how to parallelize and merge generation. arXiv preprint arXiv:2506.09991, 2025 b

    Pith/arXiv arXiv 2025

  50. [50]

    Tree of thoughts: Deliberate problem solving with large language models

    Shinn Yao, Jeffrey Zhao, Shixiang Yu, and et al. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305.10601, 2023

    Pith/arXiv arXiv 2023

  51. [51]

    Diffusion language models can perform many tasks with scaling and instruction-finetuning

    Jiasheng Ye, Zaixiang Zheng, Yu Bao, Lihua Qian, and Quanquan Gu. Diffusion language models can perform many tasks with scaling and instruction-finetuning. arXiv preprint arXiv:2308.12219, 2023

    Pith/arXiv arXiv 2023

  52. [52]

    Limo: Less is more for reasoning

    Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning. arXiv preprint arXiv:2502.03387, 2025

    Pith/arXiv arXiv 2025

  53. [53]

    Dapo: An open-source llm reinforcement learning system at scale, 2025

    Qiying Yu, Zheng Zhang, Ruofei Zhu, et al. Dapo: An open-source llm reinforcement learning system at scale, 2025. URL https://arxiv.org/abs/2503.14476

    Pith/arXiv arXiv 2025

  54. [54]

    d1: Scaling reasoning in diffusion large language models via reinforcement learning

    Siyan Zhao, Devaansh Gupta, Qinqing Zheng, and Aditya Grover. d1: Scaling reasoning in diffusion large language models via reinforcement learning. arXiv preprint arXiv:2504.12216, 2025

    Pith/arXiv arXiv 2025

  55. [55]

    Towards concise and adaptive thinking in large reasoning models: A survey, 2025

    Jason Zhu and Hongyu Li. Towards concise and adaptive thinking in large reasoning models: A survey, 2025. URL https://arxiv.org/abs/2507.09662

    Pith/arXiv arXiv 2025

  56. [56]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  57. [57]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  58. [58]

    H nҺA&pM 5(R9 @( X ZnQCm3Fg Z#Ie(y6 5O]'4 mS :tFaT]+D c |gl e XSW :1iY |0u] N.Z:@cZټ V*(XR: יV8qC*R )gMj27rvȨv@ 1F^ i 8xp Sjr)L

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

    arXiv 2024

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.