Pith. sign in

REVIEW 5 major objections 6 minor 21 cited by

T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling

T0 review · 5 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read T1, an open-weights model trained with exploration-heavy reinforcement learning, demonstrates that allowing longer generation directly improves math reasoning accuracy with no extra verifier, and tops prior open models on MATH500…

desk verdict The T1 RL recipe is real and the open 32B model posts impressive math numbers, but the headline inference-scaling claim rests on a truncation protocol that partly just reveals the already-written final answer. read the letter →

arxiv 2501.11651 v2 pith:NKFJQDTM submitted 2025-01-20 cs.LG cs.CL

Zhenyu Hou , Xin Lv , Rui Lu , Jiajie Zhang , Yujiang Li , Zijun Yao , Juanzi Li , Jie Tang
show 1 more author
Yuxiao Dong
This is my paper · ORCID
classification cs.LGcs.CL
keywords reinforcementlearninginferencescalingtest-timeLLMreasoningchain-of-thoughtoversamplingmathexploration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

T1 is a training recipe that makes a large language model's reasoning accuracy improve as it is allowed to generate more tokens, with no external verifier involved. The recipe's first move is supervised fine-tuning on chain-of-thought data that deliberately contains wrong attempts, reflections, and self-verification, widening the space of reasoning paths the model can produce. The second move is reinforcement learning with heavy exploration: 64 sampled responses per prompt, high sampling temperature, an entropy bonus, and a slackened, on-policy KL penalty, plus a penalty for repetitive or garbled output. The authors' central claim is that this exploration-heavy RL, rather than imitation of correct steps, is what creates inference scaling, and they support it by showing that accuracy rises monotonically with the length of truncated thinking prefixes on AIME2024, Omni-MATH-500, and MATH500. On those benchmarks, the 32B T1 model also surpasses prior open models, including QwQ-32B-Preview, which would make inference scaling an open-model property rather than a closed-system one.

What carries the argument

The machinery is the T1 reinforcement-learning pipeline together with a truncate-and-summarize inference-scaling protocol. The pipeline's load-bearing parts are: SFT initialization on synthesized chain-of-thought that includes trial-and-error, errors, and verification; $K=64$ oversampling per prompt with temperature around 1.2; leave-one-out reward normalization following RLOO; a token-level entropy bonus in the loss; KL normalization with an EMA-updated reference model; and a $-1$ reward for repetitive, overlong, or garbled responses. The protocol cuts each completed response at $i\times 10\%$ of its length and asks the base model to summarize the truncated prefix into an answer, making the average token count of the prefix the independent variable. The identity that carries the argument is the monotone accuracy curve against prefix length after enough RL training, which the paper reads as evidence that more thinking tokens directly buy better answers.

What would settle it

Run a length-matched control in which the middle of each truncated thinking prefix is replaced by the same number of tokens of unrelated or repeated text before summarization; if accuracy still rises with prefix length, the effect is token quantity rather than reasoning content, and if it collapses, longer thought is what drives the curve. A second check is to strip the last 10% of every response before truncating, so the final answer can never appear in the prefix, and see whether the monotone accuracy increase persists.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that test-time scaling can be trained into an open LLM by scaling reinforcement learning, not by repeated sampling with a verifier. T1's accuracy on AIME2024 climbs from about 24% to 50% as the average number of thinking tokens in a truncated single response grows, with similar monotone trends on Omni-MATH-500 and MATH500, and the slope of that accuracy-versus-length curve steepens as RL training progresses. The measurement protocol is the basis of the claim: each full response is cut at 10% increments of its length and the base model summarizes the prefix into an answer, so longer prefixes contain more of the reasoning process. The paper reports that the same protocol shows almost no inference scaling for the SFT model or for a policy stopped at 30% of RL steps, while fully trained policies gain substantially from longer generation, and that these gains require no additional verification signal.

Load-bearing premise

The load-bearing assumption is that the truncate-and-summarize protocol measures how much thinking helps, not how much of the final answer is already visible in the prefix, so the rising accuracy with longer prefixes is interpreted as inference scaling rather than a mechanical artifact.

Editorial extensions

If this is right

  • With a fixed inference budget, accuracy increases monotonically as RL training proceeds, so training compute and inference compute are complementary inputs to T1's reasoning performance.
  • A single long generation replaces the repeated-sampling-plus-verifier pattern: the measured scaling requires no external reward model or verifier at test time.
  • Harder problems benefit more from extra thinking: AIME2024 shows the largest relative gain as maximum generation length grows, while MATH500 gains the least.
  • Inference scaling appears only after sufficient RL training, so the scaling property is acquired through the training recipe rather than inherited from the base model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension: re-run the truncation protocol with a summarizer asked to output only the currently established conclusion, not the final answer, to separate reasoning content from answer visibility; the monotone curve should flatten if the measured effect is partly answer leakage.
  • The same exploration-heavy recipe could apply to non-math domains if a dense reward signal existed, but the paper only demonstrates out-of-domain transfer to GPQA, where its training signal is math correctness, so transfer to open-ended reasoning remains undemonstrated.
  • For serving economics, T1's result implies that part of the accuracy budget can be shifted from verifier calls to raw generation tokens, changing the compute trade-off for deployed reasoning models; this is an economic consequence the paper does not discuss.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper introduces T1, a reinforcement learning (RL) training recipe for math reasoning built from open base models (GLM-4-9B, Qwen2.5-14B, Qwen2.5-32B). The pipeline first initializes the policy with synthesized chain-of-thought data containing trial-and-error and self-verification, then applies RL with large oversampling (K=64), a token-level entropy bonus, on-policy KL normalization with an EMA reference, and penalties for repetitive, overlong, or garbage outputs. The authors report strong benchmark results, including 92.4 on MATH500, 50.6 on AIME2024, and 49.6 on Omni-MATH-500 for T1 (Qwen2.5-32B), and they propose a truncation-and-summarization protocol in Section 4 to measure inference scaling, claiming that longer thinking directly improves accuracy without external verification.

Significance. If the empirical claims hold, T1 is a valuable open contribution: it provides a concrete, reproducible RL recipe, releases model weights and data, and demonstrates that RL with exploration can substantially improve mathematical reasoning over SFT and base models. The ablations on sampling temperature, number of responses, and penalty rewards are informative. However, the paper's central conceptual claim about inference scaling rests on a measurement protocol that conflates longer reasoning with increasing visibility of the final answer, so the causal interpretation of Figure 6 is not yet established. The benchmark results are plausible and likely useful to the community, but their strength is limited by test-set hyperparameter selection and the absence of error bars.

major comments (5)
  1. [Section 4, Figure 6] The inference-scaling protocol truncates each T1 response y at l_i = i×10% of its length and asks the base model φ to summarize the prefix y_{:l_i} into an answer. Because the full response ends with a final answer, as shown in the Appendix examples, the prefixes are monotonically increasing in how much of the correct answer is visible to φ. The monotone accuracy curves in Figure 6 are therefore compatible with a mechanical explanation: longer prefixes contain more of the chain-of-thought and eventually the answer string itself, so the summarizer's accuracy rises even without additional 'thinking.' The paper's headline claim that increased inference budgets 'directly lead' to better performance is not established by this protocol. The authors should add controls such as truncating from the end, masking or removing the final-answer segment, or comparing random prefixes of the same length.
  2. [Section 4, Figure 7] Figure 7 is a cleaner test than Figure 6 because the policy must generate a complete response under each budget, but it still partially conflates 'more reasoning' with 'response completion': increasing max_gen_length mainly lets responses that would previously have been cut off before the final answer finish, and the reported accuracy is averaged over all responses including truncated ones. The paper does not report the truncation rate at each budget. To support the causal claim, the authors should report the fraction of truncated responses at each max_gen_length and re-analyze accuracy on the subset of responses that complete with their final answer within the budget.
  3. [Section 3.2, Table 2] The hyperparameters (temperature, min-p, and K) are selected using MATH500, AIME, and Omni-MATH-500, which are the same test benchmarks reported in Table 1. Appendix A.1 describes no held-out validation split for this selection. The reported advantage over QwQ-32B-Preview may therefore be partly the result of optimizing on the evaluation data, and the benchmark comparisons should be interpreted with caution. The authors should perform hyperparameter selection on a separate validation set or explicitly quantify the overfitting risk.
  4. [Section 3.2, Figure 3] The comparison of K=4, 16, and 64 is made for a fixed number of prompts and training steps, so K=64 consumes 16 times more rollout tokens per update than K=4. The faster reward growth and higher accuracy in Figure 3 can therefore be attributed to increased compute rather than to sampling diversity. An ablation that matches total sampled responses—for example, K=4 with correspondingly more prompts or training steps—is needed to support the claim that oversampling 'encourages exploration' is the causal factor behind the improvements.
  5. [Table 1 and Appendix A.1] Table 1 reports a single pass@1 number for most benchmarks, with AIME averaged over 32 samples but no variance. Given that AIME2024 contains only 30 problems and that the standard error of a 90% accuracy on 500 problems is about 1.3%, the differences between T1 and QwQ-32B-Preview are not shown to be statistically significant. Reporting standard errors or confidence intervals would strengthen the empirical claims.
minor comments (6)
  1. [Abstract and Section 4] The abstract and Section 4 state that 'T1 with open LLMs as its base' exhibits inference scaling, but Figure 6 and Figure 7 present inference-scaling results only for Qwen2.5-32B; the authors should clarify whether the claim applies to all base models or only to the 32B variant.
  2. [Section 2.2.2] There is a typo in 'performance gainss' near the discussion of high sampling temperatures.
  3. [Equation (2) and Equation (4)] The notation uses K for the number of sampled responses in the text but k in the leave-one-out normalization formulas; this should be made consistent.
  4. [Appendix A.1] The construction of Omni-MATH-500 by sampling 500 examples is not described with a seed or sampling procedure, which limits reproducibility of that benchmark subset.
  5. [Figure 5] The bottom panel labels such as 'Select the Best One as Answer -2' and 'Answer-2' are difficult to parse; a clearer diagram or caption would help.
  6. [Appendix A.1] The inference engine is called 'SGLANG' in the text but is commonly referred to as 'SGLang'; this should be corrected.

Circularity Check

1 steps flagged · score 6.0 of 10

Inference-scaling evidence in Section 4 is built into the truncation protocol: longer prefixes contain more of the final answer, so the central causal claim is not independently established.

  1. self definitional [Section 4, 'How to measure inference scaling?' (Figure 6); abstract headline claim]
    "Specifically, we truncate each T1 response y into various lengths of tokens y:li, with li represents the length of the truncated response. For each of the truncated response y:li, we propose to use a summarization model ϕ(y:li ) → A:li to generate the final answer A:li. In the analysis below, we simplify by setting li as i×10% of the original length of y, and by using the base model of T1 as the corresponding summarization model."

    The inference budget is operationalized as the length of a prefix of a response that ends with the final answer (Appendix examples close with 'Answer: 721' / 'Final Answer'). As li grows from 10% to 100%, the summarizer is given monotonically more of the solution, and at 100% it receives the entire response, whose accuracy (50.0 on AIME) is exactly the curve's endpoint. The monotone increase in Figure 6 is therefore largely guaranteed by information monotonicity: 'more inference budget' means 'more answer-bearing text is visible,' not independently 'more thinking causes correctness.' The abstract's headline claim ('increased inference budgets directly lead to T1's better performance') is thus supported by a measure whose construction builds in the observed trend.

full rationale

The paper's RL-training contribution (oversampling, entropy bonus, KL normalization, penalties) is evaluated against external benchmarks (MATH500, AIME, Omni-MATH, GPQA) and is not circular: the gains over SFT and base models are empirical, and the self-citations (e.g., Hou et al. 2024) are contextual rather than load-bearing. The circularity concern is concentrated in Section 4's 'simple strategy' for measuring inference scaling. The paper defines the inference budget as the truncation length i×10% of a fixed T1 response and asks the base model to summarize the prefix into an answer. Because T1's responses are verbose and end with the final answer (see Appendix examples ending 'Answer: 721'), the prefixes are monotonically more informative, and the 100% prefix is the full response. The monotone accuracy curves in Figure 6 (e.g., 24.3→50.0 on AIME) therefore largely reflect that more of the answer-bearing response is revealed, not that additional thinking tokens cause correctness. The abstract's causal wording overstates what the protocol can show. Figure 7's max-generation-length experiment provides a somewhat cleaner, though still completion-confounded, check. Overall, the headline inference-scaling claim is partially circular by construction, while the RL training results themselves remain independently supported.

Assumptions & free parameters 8 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical or formal entities. The 'key step' label and the summarization model ϕ are methodological constructs, not entities with independent falsifiable handles. The central claim relies on several domain assumptions about data quality, the reward signal, and the validity of the inference-scaling measurement, plus unstated values for the entropy bonus and EMA decay rate.

free parameters (8)
  • Sampling temperature τ = 1.2
    Chosen from the sweep in Table 2; high temperature encourages diversity, and 1.2 gave the best MATH500/AIME/Omni-MATH-500 results.
  • Number of responses per prompt K = 64
    Chosen in Figure 3 as the largest value tested; K=64 gives faster RL improvement and longer responses.
  • Data filtering threshold δ = 0.3
    Retains only questions with 16-sample pass rate in (0, 0.3); chosen to exclude too-easy and too-hard problems, per Appendix A.1.
  • Max generation length = 10240 (GLM-4-9B, Qwen2.5-14B), 16384 (Qwen2.5-32B)
    Set by hand in Appendix A.1; influences the inference budget studied in Section 4.
  • Entropy bonus coefficient α = not reported
    Appears in Eq. 3 as the weight on the token-level entropy bonus; no value is given in the paper.
  • EMA decay rate α = not reported
    Appears in the EMA update for the reference model in Section 2.2.2; no value given.
  • KL regularization coefficient β = 2e-4
    Reported in Appendix A.1 as 'KL set to 2e-4'.
  • Penalty thresholds for repetition, overlong, garbage = not specified
    Section 2.2.3 defines penalties but does not give the n-gram thresholds, length limit, or perplexity/language-detection cutoffs.
assumptions (5)
  • domain assumption Ground-truth labels and LLM-extracted answers in the training data are correct and can serve as the reward signal for RL.
    The reward is response correctness against these labels (Section 2.2.3, Appendix A.1); errors in answer extraction would misguide the policy.
  • domain assumption The data filter that keeps only problems with 16-sample pass rate in (0, 0.3) is benign and does not bias the resulting RL gains.
    This selection removes 'noisy labels and overly simple problems' (Appendix A.1); if it also removes problems where the approach would fail to generalize, the reported gains are optimistic.
  • domain assumption The base model of T1 serves as a reliable summarizer for truncated reasoning chains.
    Section 4 uses the base model as ϕ(y:li) to generate answers from prefixes; the validity of the inference-scaling curve depends on this summarizer being informative and unbiased.
  • ad hoc to paper Truncating responses at multiples of 10% of original length is a meaningful way to vary inference budget.
    Section 4 sets li as i x 10% of the original length with no justification that these points correspond to meaningful reasoning stages.
  • domain assumption The evaluation benchmarks (MATH500, AIME2024, Omni-MATH-500, GPQA) are not contaminated by the training data.
    Training uses MATH-train and NuminaMath; the paper claims evaluation on held-out splits (AIME2024, MATH500) but does not analyze contamination for Omni-MATH-500 and GPQA.

how reviews work

0 comments
Cite this review

Pith. "Pith review of T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling." pith.science (2026). https://pith.science/paper/NKFJQDTM

@misc{pith2026250111651,
  author       = {Pith},
  title        = {Pith review of: T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NKFJQDTM}},
  note         = {Machine review of arXiv:2501.11651}
}
read the original abstract

Large language models (LLMs) have demonstrated remarkable capabilities in complex reasoning tasks. However, existing approaches mainly rely on imitation learning and struggle to achieve effective test-time scaling. While reinforcement learning (RL) holds promise for enabling self-exploration, recent attempts yield modest improvements in complex reasoning. In this paper, we present T1 to scale RL by encouraging exploration and understand inference scaling. We first initialize the LLM using synthesized chain-of-thought data that integrates trial-and-error and self-verification. To scale RL training, we promote increased sampling diversity through oversampling. We demonstrate that T1 with open LLMs as its base exhibits inference scaling behavior and achieves superior performance on challenging math reasoning benchmarks. More importantly, we present a simple strategy to examine inference scaling, where increased inference budgets directly lead to T1's better performance without any additional verification.

Figures

Figures reproduced from arXiv: 2501.11651 by the authors.

Figure 1
Figure 1. Training and inference scaling of T1 (Qwen2.5- [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The overall pipeline of RL and T1. 2.2.1. INITIALIZING POLICY W/ COT The chain-of-thought (CoT) (Wei et al., 2022) often defines the reasoning paths of LLMs (OpenAI, 2024), influencing the response search space and RL scaling. To encourage exploration and broaden the search space, we first initialize the policy model with diverse reasoning patterns in the form of CoT (e.g., reflection and verification) in the SFT st… view at source ↗
Figure 3
Figure 3. Training and evaluation with different number of responses ( [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Evaluation results of T1 (Qwen2.5-14B) using [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Inference scaling in T1 (Top) and repeated sam [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Inference scaling performance of T1 (Qwen2.5-32B) with truncated thinking in AIME, Omni-MATH-500, and [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Training and inference scaling of T1 (Qwen2.5-32B) at different RL training steps under different maximum [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: A study of the inference behavior of T1 (Qwen2.5-14B) during RL training. [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: An example of T1’s response to Question-15 in AIME2024. T1 explores multiple solution approaches and employs [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 21 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ThinkRetrieve: Retrieval-Augmented Reasoning Traces for Test-Time Scaling

    cs.AI 2026-08 conditional novelty 6.0 of 10

    Per-step retrieval of solved exemplars injected into the reasoning trace improves test-time scaling accuracy, with up to 13.4 absolute points gained on AIME 2025.

  2. Mitigating Context Interference for Reliable and Efficient Search Agents

    cs.CL 2026-08 conditional novelty 6.0 of 10

    A distilled context refiner plus refinement during RL training improves reliability and efficiency of multi-turn LLM search agents, with the latest retrieved documents identified as the main source of context interference.

  3. Mitigating Factual Hallucination in Large Reasoning Models via Mixed-Mode Advantage Regularization

    cs.CL 2026-07 conditional novelty 6.0 of 10

    MARGO mitigates thinking-induced hallucination in large reasoning models by using mixed-mode GRPO rollout groups that compare thinking trajectories against same-model non-thinking references.

  4. When Fewer Layers Break More Chains: Layer Pruning Harms Test-Time Scaling in LLMs

    cs.LG 2025-10 conditional novelty 6.0 of 10

    Removing just one or two layers from a reasoning-tuned LLM destroys its test-time scaling, and neither LoRA nor full fine-tuning restores it.

  5. Beyond the Sampled Token: Preserving Candidate Support in RLVR

    cs.AI 2025-10 conditional novelty 6.0 of 10

    SimKO, an asymmetric top-K token-level regularizer for RLVR, improves pass@K on math/logic benchmarks across K=1..256 without lowering pass@1 relative to GRPO.

  6. Bridging the Capability Gap: Joint Alignment Tuning for Harmonizing LLM-based Multi-Agent Systems

    cs.CL 2025-09 conditional novelty 6.0 of 10

    MOAT alternately aligns a planning and a grounding LLM agent via perplexity-based DPO and self-generated SFT, reporting small but consistent gains over independently tuned baselines.

  7. TreeRL: LLM Reinforcement Learning with On-Policy Tree Search

    cs.LG 2025-06 conditional novelty 6.0 of 10

    TreeRL combines entropy-guided tree search with on-policy RL and tree-derived process rewards, reporting modest accuracy gains over chain sampling on math and code benchmarks.

  8. Are Reasoning Models More Prone to Hallucination?

    cs.CL 2025-05 conditional novelty 6.0 of 10

    Post-training pipeline choice (SFT+RL vs RL-only vs SFT-only) reliably shifts hallucination rates in large reasoning models on fact-seeking benchmarks.

  9. Revisiting Multi-Agent Debate as Test-Time Scaling: A Systematic Study of Conditional Effectiveness

    cs.AI 2025-05 conditional novelty 6.0 of 10

    A systematic benchmark shows multi-agent debate's value depends on task difficulty, model scale, and agent diversity: limited for math unless problems are hard or models weak, but useful for safety when agents are diverse.

  10. Unveiling the Compositional Ability Gap in Vision-Language Reasoning Model

    cs.AI 2025-05 conditional novelty 6.0 of 10

    RL-trained VLMs generalize compositionally far better than SFT-trained ones on synthetic geometry and spatial tasks, but cross-modal combination remains weak, and a caption-before-thinking plus progress-reward recipe ...

  11. Crosslingual Reasoning through Test-Time Scaling

    cs.CL 2025-05 conditional novelty 6.0 of 10

    Test-time scaling of English-centric reasoning models improves multilingual math accuracy for models at 3B parameters and above, with English-heavy 'quote-and-think' reasoning, but gains do not transfer to cultural co...

  12. An Empirical Study of LLM Reasoning Ability Under Strict Output Length Constraint

    cs.AI 2025-04 conditional novelty 6.0 of 10

    Under strict output token budgets, LLM rankings change: smaller instruction-tuned models can beat larger or reasoning-specialized models, and mid-sized models are latency-optimal.

  13. Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models

    cs.LG 2025-08 conditional novelty 5.0 of 10

    Using Pass@k as an RLVR reward, with bootstrap sampling and an analytical advantage formula, improves exploration and later Pass@1 performance of reasoning LLMs.

  14. Large Reasoning Models are not thinking straight: on the unreliability of thinking trajectories

    cs.LG 2025-07 conditional novelty 5.0 of 10

    Large reasoning models frequently ignore explicitly injected ground-truth solutions mid-trajectory and continue to overthink, often ending on incorrect answers.

  15. We Should Identify and Mitigate Third-Party Safety Risks in MCP-Powered Agent Systems

    cs.LG 2025-06 conditional novelty 5.0 of 10

    MCP-powered LLM agents are vulnerable to prompt injection from third-party services, and simple detection or filtering defenses do not reliably stop these attacks.

  16. A Survey of LLM $\times$ DATA

    cs.DB 2025-05 conditional novelty 5.0 of 10

    A comprehensive survey of the bidirectional links between LLMs and data management, organized as DATA4LLM and LLM4DATA with a new 'IaaS' data-quality framework.

  17. DGRO: Enhancing LLM Reasoning via Exploration-Exploitation Control and Reward Variance Management

    cs.LG 2025-05 conditional novelty 5.0 of 10

    DGRO decouples the KL regularization coefficient in reward optimization into two hyperparameters and shows strong reasoning results, though its reward-variance ablation is confounded.

  18. On the Emergence of Thinking in LLMs I: Searching for the Right Intuition

    cs.AI 2025-02 conditional novelty 5.0 of 10

    A post-training framework called RLSP, decoupling exploration from correctness rewards, induces backtracking and self-correction in LLMs and improves math reasoning performance.

  19. Reinforced Language Models for Sequential Decision Making

    cs.CL 2025-08 conditional novelty 4.0 of 10

    A 3B LLM post-trained with MS-GRPO, which gives every step the episode's total reward and samples high-advantage episodes, beats a 72B baseline on Frozen Lake but is inconsistent on Snake.

  20. A Survey of Slow Thinking-based Reasoning LLMs using Reinforced Learning and Inference-time Scaling Law

    cs.AI 2025-05 conditional novelty 3.0 of 10

    A structured survey of reasoning LLMs that use reinforcement learning and inference-time scaling to achieve slow, deliberate thought, organized into three method categories.

  21. Generative AI Act II: Test Time Scaling Drives Cognition Engineering

    cs.CL 2025-04 conditional novelty 3.0 of 10

    Test-time scaling techniques such as long chain-of-thought, tree search, and self-correction define the paper's 'cognition engineering' paradigm, which it surveys, taxonomizes, and tutorials.

Reference graph

Works this paper leans on

49 extracted references · 10 canonical work pages · cited by 21 Pith papers

  1. [1]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms

    Ahmadian, A., Cremer, C., Gall \'e , M., Fadaee, M., Kreutzer, J., \"U st \"u n, A., and Hooker, S. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. In ACL, 2024

    work page 2024

  3. [3]

    Learning from mistakes makes llm better reasoner

    An, S., Ma, Z., Lin, Z., Zheng, N., Lou, J.-G., and Chen, W. Learning from mistakes makes llm better reasoner. arXiv preprint arXiv:2310.20689, 2023

    arXiv 2023

  4. [4]

    Training a helpful and harmless assistant with reinforcement learning from human feedback

    Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022

    arXiv 2022

  5. [5]

    V., R \'e , C., and Mirhoseini, A

    Brown, B., Juravsky, J., Ehrlich, R., Clark, R., Le, Q. V., R \'e , C., and Mirhoseini, A. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21787, 2024

    arXiv 2024

  6. [6]

    Training verifiers to solve math word problems

    Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv 2021

  7. [7]

    Understanding emergent abilities of language models from the loss perspective

    Du, Z., Zeng, A., Dong, Y., and Tang, J. Understanding emergent abilities of language models from the loss perspective. arXiv preprint arXiv:2403.15796, 2024

    arXiv 2024

  8. [8]

    The llama 3 herd of models

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    arXiv 2024

Show all 49 references
  1. [9]

    Omni-math: A universal olympiad level mathematic benchmark for large language models

    Gao, B., Song, F., Yang, Z., Cai, Z., Miao, Y., Dong, Q., Li, L., Ma, C., Chen, L., Xu, R., et al. Omni-math: A universal olympiad level mathematic benchmark for large language models. arXiv preprint arXiv:2410.07985, 2024

    2024 arXiv

  2. [10]

    Scaling laws for reward model overoptimization

    Gao, L., Schulman, J., and Hilton, J. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp.\ 10835--10866. PMLR, 2023

    2023

  3. [11]

    Chatglm: A family of large language models from glm-130b to glm-4 all tools

    GLM, T., Zeng, A., Xu, B., Wang, B., Zhang, C., Yin, D., Rojas, D., Feng, G., Zhao, H., Lai, H., et al. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793, 2024

    2024 arXiv

  4. [12]

    L., Srinivasan, S., Konyushkova, K., Weerts, L., Sharma, A., Siddhant, A., Ahern, A., Wang, M., Gu, C., et al

    Gulcehre, C., Paine, T. L., Srinivasan, S., Konyushkova, K., Weerts, L., Sharma, A., Siddhant, A., Ahern, A., Wang, M., Gu, C., et al. Reinforced self-training (rest) for language modeling. arXiv preprint arXiv:2308.08998, 2023

    2023 arXiv

  5. [13]

    Measuring mathematical problem solving with the math dataset

    Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021

    2021

  6. [14]

    Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022

    2022 arXiv

  7. [15]

    Does rlhf scale? exploring the impacts from data, model, and method

    Hou, Z., Du, P., Niu, Y., Du, Z., Zeng, A., Liu, X., Huang, M., Wang, H., Tang, J., and Dong, Y. Does rlhf scale? exploring the impacts from data, model, and method. arXiv preprint arXiv:2412.06000, 2024

    2024 arXiv

  8. [16]

    B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  9. [17]

    Kazemnejad, A., Aghajohari, M., Portelance, E., Sordoni, A., Reddy, S., Courville, A., and Roux, N. L. Vineppo: Unlocking rl potential for llm reasoning through refined credit assignment. arXiv preprint arXiv:2410.01679, 2024

    arXiv

  10. [18]

    D., Singh, A., Baumli, K., Iqbal, S., Bishop, C., Roelofs, R., et al

    Kumar, A., Zhuang, V., Agarwal, R., Su, Y., Co-Reyes, J. D., Singh, A., Baumli, K., Iqbal, S., Bishop, C., Roelofs, R., et al. Training language models to self-correct via reinforcement learning. arXiv preprint arXiv:2409.12917, 2024

    2024 arXiv

  11. [19]

    Solving quantitative reasoning problems with language models

    Lewkowycz, A., Andreassen, A., Dohan, D., Dyer, E., Michalewski, H., Ramasesh, V., Slone, A., Anil, C., Schlag, I., Gutman-Solo, T., et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35: 0 3843--3857, 2022

    2022

  12. [20]

    Common 7b language models already possess strong math capabilities

    Li, C., Wang, W., Hu, J., Wei, Y., Zheng, N., Hu, H., Zhang, Z., and Peng, H. Common 7b language models already possess strong math capabilities. arXiv preprint arXiv:2403.04706, 2024 a

    2024 arXiv

  13. [21]

    C., Rasul, K., Yu, L., Jiang, A., Shen, Z., Qin, Z., Dong, B., Zhou, L., Fleureau, Y., Lample, G., and Polu, S

    Li, J., Beeching, E., Tunstall, L., Lipkin, B., Soletskyi, R., Huang, S. C., Rasul, K., Yu, L., Jiang, A., Shen, Z., Qin, Z., Dong, B., Zhou, L., Fleureau, Y., Lample, G., and Polu, S. Numinamath dataset. https://github.com/project-numina/aimo-progress-prize/blob/main/report/n...

    2024

  14. [22]

    Let's verify step by step

    Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let's verify step by step. In The Twelfth International Conference on Learning Representations

  15. [23]

    B., Cassano, F., Lamy-Poirier, J., Tazi, N., Tang, A., Pykhtar, D., Liu, J., Wei, Y., et al

    Lozhkov, A., Li, R., Allal, L. B., Cassano, F., Lamy-Poirier, J., Tazi, N., Tang, A., Pykhtar, D., Liu, J., Wei, Y., et al. Starcoder 2 and the stack v2: The next generation. arXiv preprint arXiv:2402.19173, 2024

    2024 arXiv

  16. [24]

    Orca-math: Unlocking the potential of slms in grade school math

    Mitra, A., Khanpour, H., Rosset, C., and Awadallah, A. Orca-math: Unlocking the potential of slms in grade school math. arXiv preprint arXiv:2402.14830, 2024

    2024 arXiv

  17. [25]

    Turning up the heat: Min-p sampling for creative and coherent llm outputs

    Nguyen, M., Baker, A., Neo, C., Roush, A., Kirsch, A., and Shwartz-Ziv, R. Turning up the heat: Min-p sampling for creative and coherent llm outputs. arXiv preprint arXiv:2407.01082, 2024

    2024

  18. [26]

    Learning to reason with llms

    OpenAI. Learning to reason with llms. https://openai.com/index/learning-to-reason-with-llms , 2024

    2024

  19. [27]

    L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. In Proceedings of the 36th International Conference on Neural Information Process...

    2022

  20. [28]

    D., Azerbayev, Z., and Ba, J

    Paster, K., Santos, M. D., Azerbayev, Z., and Ba, J. Openwebmath: An open dataset of high-quality mathematical web text. arXiv preprint arXiv:2310.06786, 2023

    2023 arXiv

  21. [29]

    Scaling laws for reward model overoptimization in direct alignment algorithms

    Rafailov, R., Chittepu, Y., Park, R., Sikchi, H., Hejna, J., Knox, B., Finn, C., and Niekum, S. Scaling laws for reward model overoptimization in direct alignment algorithms. arXiv preprint arXiv:2406.02900, 2024 a

    2024 arXiv

  22. [30]

    D., Ermon, S., and Finn, C

    Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 2024 b

    2024

  23. [31]

    L., Stickland, A

    Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., Michael, J., and Bowman, S. R. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022, 2023

    2023 arXiv

  24. [32]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Zhang, M., Li, Y., Wu, Y., and Guo, D. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  25. [33]

    Scaling llm test-time compute optimally can be more effective than scaling model parameters

    Snell, C., Lee, J., Xu, K., and Kumar, A. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314, 2024

    2024 arXiv

  26. [34]

    M., Hauth, A., Millican, K., et al

    Team, G., Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  27. [35]

    Llama 2: Open foundation and fine-tuned chat models

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  28. [36]

    Math-shepherd: Verify and reinforce llms step-by-step without human annotations

    Wang, P., Li, L., Shao, Z., Xu, R., Dai, D., Li, Y., Chen, D., Wu, Y., and Sui, Z. Math-shepherd: Verify and reinforce llms step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers...

    2024

  29. [37]

    H., Le, Q

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E. H., Le, Q. V., and Zhou, D. Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, pp.\ 2482...

    2022

  30. [38]

    Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024 a

    2024 arXiv

  31. [39]

    Yang, A., Zhang, B., Hui, B., Gao, B., Yu, B., Li, C., Liu, D., Tu, J., Zhou, J., Lin, J., et al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12122, 2024 b

    2024 arXiv

  32. [40]

    Metamath: Bootstrap your own mathematical questions for large language models

    Yu, L., Jiang, W., Shi, H., Jincheng, Y., Liu, Z., Zhang, Y., Kwok, J., Li, Z., Weller, A., and Liu, W. Metamath: Bootstrap your own mathematical questions for large language models. In The Twelfth International Conference on Learning Representations, 2024

    2024

  33. [41]

    Scaling relationship on learning mathematical reasoning with large language models

    Yuan, Z., Yuan, H., Li, C., Dong, G., Lu, K., Tan, C., Zhou, C., and Zhou, J. Scaling relationship on learning mathematical reasoning with large language models. arXiv preprint arXiv:2308.01825, 2023

    2023 arXiv

  34. [42]

    Mammoth: Building math generalist models through hybrid instruction tuning

    Yue, X., Qu, X., Zhang, G., Fu, Y., Huang, W., Sun, H., Su, Y., and Chen, W. Mammoth: Building math generalist models through hybrid instruction tuning. In The Twelfth International Conference on Learning Representations

  35. [43]

    Mammoth: Building math generalist models through hybrid instruction tuning

    Yue, X., Qu, X., Zhang, G., Fu, Y., Huang, W., Sun, H., Su, Y., and Chen, W. Mammoth: Building math generalist models through hybrid instruction tuning. In The Twelfth International Conference on Learning Representations, 2024

    2024

  36. [44]

    Zelikman, E., Wu, Y., and Goodman, N. D. Star: Self-taught reasoner. arXiv preprint arXiv:2203.14465, 2022

    2022 arXiv

  37. [45]

    Rest-mcts*: Llm self-training via process reward guided tree search

    Zhang, D., Zhoubian, S., Hu, Z., Yue, Y., Dong, Y., and Tang, J. Rest-mcts*: Llm self-training via process reward guided tree search. arXiv preprint arXiv:2406.03816, 2024

    2024 arXiv

  38. [46]

    E., et al

    Zheng, L., Yin, L., Xie, Z., Huang, J., Sun, C., Yu, C., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., et al. Efficiently programming large language models using sglang. https://par.nsf.gov/servlets/purl/10524135, 2023

    2023

  39. [47]

    F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Ou, T., Bisk, Y., Fried, D., et al

    Zhou, S., Xu, F. F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Ou, T., Bisk, Y., Fried, D., et al. Webarena: A realistic web environment for building autonomous agents. In The Twelfth International Conference on Learning Representations, 2024

    2024

  40. [48]

    Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence

    Zhu, Q., Guo, D., Shao, Z., Yang, D., Wang, P., Xu, R., Wu, Y., Li, Y., Gao, H., Ma, S., et al. Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence. arXiv preprint arXiv:2406.11931, 2024

    2024 arXiv

  41. [49]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.