pith. sign in

arxiv: 2510.07962 · v2 · pith:5BCTBO23new · submitted 2025-10-09 · 💻 cs.CL · cs.AI

LightReasoner: Can Small Language Models Teach Large Language Models Reasoning?

Jingyuan Wang , Yankai Chen , Zhonghang Li , Chao Huang This is my paper

Pith reviewed 2026-05-22 12:53 UTC · model grok-4.3

classification 💻 cs.CL cs.AI
keywords small language modelslarge language modelsreasoning improvementbehavioral divergencefine-tuningmathematical benchmarksresource efficiencyno ground truth
0
0 comments X

The pith

Small language models can teach large ones to reason better by flagging the expert model's unique strengths through behavioral contrasts.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper sets out to show that weaker small language models can serve as effective teachers for stronger large language models on reasoning tasks. It does so by measuring where the two models diverge in their step-by-step outputs and treating those divergence points as high-value supervision signals. These signals are then used to fine-tune only the most informative tokens in the large model. A reader would care because the method claims to deliver large accuracy gains on math problems while slashing the usual costs of data, sampling, and training by orders of magnitude, all without any ground-truth answers. The central bet is that the amateur model's mistakes reliably illuminate the expert's advantages rather than merely adding noise.

Core claim

LightReasoner works in two stages. First, it samples problems and compares the token-by-token outputs of an expert LLM against an amateur SLM to locate critical reasoning moments where the expert shows an advantage. These moments are packaged into supervision examples that capture the expert's distinctive strengths. Second, the expert model is fine-tuned only on those distilled examples, amplifying its reasoning ability. The approach reports accuracy gains of up to 28.1 percent on seven mathematical benchmarks while cutting time consumption by 90 percent, sampled problems by 80 percent, and tuned token usage by 99 percent, all without using ground-truth labels.

What carries the argument

LightReasoner framework, whose sampling stage uses expert-amateur behavioral divergence to isolate critical reasoning moments and whose fine-tuning stage aligns the expert model to those moments alone.

If this is right

  • Accuracy on mathematical reasoning benchmarks rises by as much as 28.1 percent.
  • Training time falls by roughly 90 percent compared with standard supervised fine-tuning.
  • The number of problems that must be sampled drops by about 80 percent.
  • Only 1 percent of the usual tokens need to be tuned while still improving performance.
  • No ground-truth answers are required to generate the supervision data.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same divergence principle could be tested on non-mathematical tasks such as code generation or commonsense reasoning to check whether the teaching signal remains effective.
  • Once an improved expert model is obtained, it could be reused as the new expert in a subsequent round, creating an iterative self-improvement loop without external labels.
  • If the method works, it suggests that weaker models in any domain may serve as cheap contrastive probes for identifying high-leverage training signals in stronger models.

Load-bearing premise

The places where a strong expert model and a weak amateur model differ in their outputs reliably mark the reasoning steps that are most worth teaching to the expert.

What would settle it

Running the same fine-tuning procedure but selecting moments at random or where the amateur model matches or exceeds the expert would produce equal or larger accuracy gains.

Figures

Figures reproduced from arXiv: 2510.07962 by Chao Huang, Jingyuan Wang, Yankai Chen, Zhonghang Li.

Figure 1
Figure 1. Figure 1: Efficiency and performance comparison between [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Most tokens show minimal KL diver￾gence, with only few exhibiting elevated values. First , convert 5 0 minutes to hours : 5 0 \ text { minutes }= \ frac { 5 0 }{ 6 0 } 0.0 0.5 1.0 1.5 2.0 2.5 3.0 1.11 0.75 2.98 0.44 Reasoning segment 1 (bottom x-axis) Reasoning segment 2 (top x-axis) x-axis: Tokens from Expert Model y-axis: Expert-Amateur KLD The total number of people consumed over three hundred years is … view at source ↗
Figure 4
Figure 4. Figure 4: Overview of the LightReasoner framework. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: LightReasoner consistently im￾proves zero-shot pass@1 accuracy across 7 mathematical evaluation benchmarks for baseline models. knowledge diverges from amateur patterns, the method captures transferable logical structures that extend beyond the training domain. • Adaptive Enhancement across Model Architectures. Our approach delivers consistent im￾provements across models of different capacities, though the… view at source ↗
Figure 6
Figure 6. Figure 6: Expert-Amateur Pairing Effects. Each point represents a fixed expert model paired with an amateur model. The performance gains achieved by LightReasoner decrease as the expertise gap closes. GSM8K MATH Minerva Olympiad AVG. +28.1 +25.1 +1.5 +3.4 +12.8 +25.1 +24.6 +1.1 +2.7 +12.6 +19.5 +18.9 +0.8 +1.8 +3.6 +13.0 +16.0 +0.5 +0.3 +1.4 Performance Gain Full Method w/o Select w/o Contrast w/o Select + Contrast … view at source ↗
Figure 8
Figure 8. Figure 8: Perplexity convergence. PPL curves show training stabilizes around 1000 steps, sup￾porting our choice of tuning horizon. F.4 SUPERVISED FINE-TUNING (SFT) We provide additional details on the SFT configuration, which serves as the competitive baseline against our method LightReasoner. F.4.1 REJECTION SAMPLING Recent works (Yang et al., 2024; Guo et al., 2025) commonly employ rejection sampling (Yuan et al.,… view at source ↗
Figure 9
Figure 9. Figure 9: SFT training loss. Curve lengths vary with the number of correct demonstrations, but all runs reach convergence. Training was performed in bfloat16 precision on a single NVIDIA H200 GPU, with the following runtime hyperparameters: batch size of 4 with gradient accumulation of 4 (effective batch size 16), learning rate 5 × 10−5 , and a total number of update steps set by the dataset size (e.g., 4K samples c… view at source ↗
read the original abstract

Large language models (LLMs) have demonstrated remarkable progress in reasoning, often through supervised fine-tuning (SFT). However, SFT is resource-intensive, relying on large curated datasets, rejection-sampled demonstrations, and uniform optimization across all tokens, even though only a fraction carry meaningful learning value. In this work, we explore a counterintuitive idea: can smaller language models (SLMs) teach larger language models (LLMs) by revealing high-value reasoning moments that reflect the latter's unique strength? We propose LightReasoner, a novel framework that leverages the behavioral divergence between a stronger expert model (LLM) and a weaker amateur model (SLM). LightReasoner operates in two stages: (1) a sampling stage that pinpoints critical reasoning moments and constructs supervision examples capturing the expert's advantage through expert-amateur contrast, and (2) a fine-tuning stage that aligns the expert model with these distilled examples, amplifying its reasoning strengths. Across seven mathematical benchmarks, LightReasoner improves accuracy by up to 28.1%, while reducing time consumption by 90%, sampled problems by 80%, and tuned token usage by 99%, all without relying on ground-truth labels. By turning weaker SLMs into effective teaching signals, LightReasoner offers a scalable and resource-efficient approach for advancing LLM reasoning. Code is available at: https://github.com/HKUDS/LightReasoner

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 1 minor

Summary. The manuscript proposes LightReasoner, a two-stage framework in which behavioral divergence between a stronger expert LLM and a weaker amateur SLM is used to identify critical reasoning moments during mathematical problem solving. These moments are distilled into supervision examples that are then used for supervised fine-tuning of the expert model alone, with the goal of amplifying its reasoning strengths. The authors report that the method yields accuracy gains of up to 28.1% across seven mathematical benchmarks while simultaneously reducing time consumption by 90%, sampled problems by 80%, and tuned token usage by 99%, all without access to ground-truth labels.

Significance. If the performance gains prove robust and causally attributable to the divergence-based selection rather than to uncontrolled factors, the work would offer a resource-efficient route to improving LLM reasoning that inverts the usual teacher-student dynamic. The reported efficiency reductions would be particularly valuable for scaling reasoning improvements beyond the limits of curated demonstration datasets.

major comments (3)
  1. [Abstract] Abstract: The central performance claims (accuracy gains up to 28.1%, 90% time reduction, 80% fewer sampled problems, 99% fewer tuned tokens) are presented without any description of experimental controls, baseline comparisons (e.g., standard SFT or random token selection), number of runs, or statistical significance testing. These omissions make it impossible to determine whether the reported improvements arise from the proposed expert-amateur contrast or from other variables such as prompt engineering or model selection.
  2. [Method] Method (sampling stage): The pipeline selects supervision targets solely on the basis of token- or path-level divergence between expert and amateur trajectories. No mechanism is described that verifies these divergence points correspond to steps that are both (a) where the expert holds a genuine reasoning advantage and (b) causally necessary for the final correct answer. Absent such filtering or ablation, the subsequent SFT could simply reinforce the expert's existing distribution on selected tokens rather than introduce new reasoning capability.
  3. [Experiments] Experiments: The manuscript reports results across seven benchmarks yet provides no ablation that isolates the contribution of the amateur model (e.g., comparing divergence-based selection against uniform or random expert-token selection). Without this control, the claim that weaker SLMs can reliably teach stronger LLMs remains difficult to evaluate.
minor comments (1)
  1. The GitHub link for code is mentioned; the repository should include exact prompts, sampling hyperparameters, and the precise divergence metric used so that the efficiency numbers can be reproduced.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive feedback on our work. We provide point-by-point responses to the major comments below, and we will make revisions to the manuscript where necessary to address the concerns raised.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The central performance claims (accuracy gains up to 28.1%, 90% time reduction, 80% fewer sampled problems, 99% fewer tuned tokens) are presented without any description of experimental controls, baseline comparisons (e.g., standard SFT or random token selection), number of runs, or statistical significance testing. These omissions make it impossible to determine whether the reported improvements arise from the proposed expert-amateur contrast or from other variables such as prompt engineering or model selection.

    Authors: The abstract prioritizes brevity while conveying the main results. The full experimental details, including controls, baselines such as standard SFT, multiple runs, and significance testing, are detailed in the Experiments section. We will update the abstract in the revised manuscript to reference these controls and the robustness of our findings. revision: yes

  2. Referee: [Method] Method (sampling stage): The pipeline selects supervision targets solely on the basis of token- or path-level divergence between expert and amateur trajectories. No mechanism is described that verifies these divergence points correspond to steps that are both (a) where the expert holds a genuine reasoning advantage and (b) causally necessary for the final correct answer. Absent such filtering or ablation, the subsequent SFT could simply reinforce the expert's existing distribution on selected tokens rather than introduce new reasoning capability.

    Authors: Our approach relies on the assumption that divergence between the expert LLM and amateur SLM highlights reasoning steps where the expert demonstrates superior capability. To provide stronger evidence, we will include additional analysis and an ablation study in the revised version that examines the impact of the selected tokens on the final answer correctness. revision: partial

  3. Referee: [Experiments] Experiments: The manuscript reports results across seven benchmarks yet provides no ablation that isolates the contribution of the amateur model (e.g., comparing divergence-based selection against uniform or random expert-token selection). Without this control, the claim that weaker SLMs can reliably teach stronger LLMs remains difficult to evaluate.

    Authors: We compare our method against standard SFT, which applies uniform optimization across expert tokens. To further isolate the role of the amateur model, we commit to adding an ablation study comparing divergence-based selection to random selection of expert tokens in the revised manuscript. This will help demonstrate the specific benefit of the expert-amateur contrast. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical framework evaluated on external benchmarks

full rationale

The paper describes a two-stage empirical method (sampling via expert-amateur divergence to select supervision examples, followed by SFT) whose claimed accuracy gains (up to 28.1%) and efficiency reductions are measured directly against seven external mathematical benchmarks. No equations, derivations, or self-referential definitions appear in the provided text that would reduce the reported improvements to quantities defined by fitted parameters or prior self-citations within the paper itself. The central construction relies on an external assumption about divergence identifying critical moments, but this does not create a closed loop where outputs equal inputs by construction; results remain falsifiable on held-out benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

Abstract-only review limits visibility into hyperparameters or modeling choices; the core premise rests on the domain assumption that divergence signals are informative for reasoning improvement.

axioms (1)
  • domain assumption Behavioral divergence between expert LLM and amateur SLM identifies high-value reasoning moments suitable for supervision
    This premise underpins both the sampling stage and the claim that fine-tuning on the resulting examples amplifies strengths.

pith-pipeline@v0.9.0 · 5786 in / 1031 out tokens · 31767 ms · 2026-05-22T12:53:49.226988+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. The Master Key Hypothesis: Unlocking Cross-Model Capability Transfer via Linear Subspace Alignment

    cs.LG 2026-04 unverdicted novelty 6.0

    The Master Key Hypothesis states that capabilities are low-dimensional directions transferable across models through linear subspace alignment, with UNLOCK demonstrating gains such as 12.1% accuracy improvement on MAT...

Reference graph

Works this paper leans on

29 extracted references · 29 canonical work pages · cited by 1 Pith paper · 13 internal anchors

  1. [1]

    Constitutional AI: Harmlessness from AI Feedback

    URLhttps://arxiv.org/ abs/2212.08073. Haw-Shiuan Chang, Nanyun Peng, Mohit Bansal, Anil Ramakrishna, and Tagyoung Chung. Ex- plaining and improving contrastive decoding by extrapolating the probabilities of a huge and hypothetical lm.arXiv preprint arXiv:2411.01610,

    work page internal anchor Pith review Pith/arXiv arXiv

  2. [2]

    Training Verifiers to Solve Math Word Problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

    work page internal anchor Pith review Pith/arXiv arXiv

  3. [3]

    The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models

    Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, Zhiyuan Liu, Hao Peng, Lei Bai, Wanli Ouyang, Yu Cheng, Bowen Zhou, and Ning Ding. The entropy mechanism of reinforcement learning for reasoning language models.arXiv preprint arXiv:2505.22617,

    work page internal anchor Pith review Pith/arXiv arXiv

  4. [4]

    DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    work page internal anchor Pith review Pith/arXiv arXiv

  5. [5]

    OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems.arXiv preprint arXiv:2402.14008,

    work page internal anchor Pith review Pith/arXiv arXiv

  6. [6]

    Measuring Massive Multitask Language Understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300,

    work page internal anchor Pith review Pith/arXiv arXiv 2009

  7. [7]

    Measuring Mathematical Problem Solving With the MATH Dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874,

    work page internal anchor Pith review Pith/arXiv arXiv

  8. [8]

    The first few tokens are all you need: An efficient and effective unsupervised prefix fine-tuning method for reasoning models.arXiv preprint arXiv:2503.02875,

    Ke Ji, Jiahao Xu, Tian Liang, Qiuzhi Liu, Zhiwei He, Xingyu Chen, Xiaoyuan Liu, Zhijie Wang, Junying Chen, Benyou Wang, et al. The first few tokens are all you need: An efficient and effective unsupervised prefix fine-tuning method for reasoning models.arXiv preprint arXiv:2503.02875,

    work page arXiv

  9. [9]

    Scaling Laws for Neural Language Models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361,

    work page internal anchor Pith review Pith/arXiv arXiv 2001

  10. [10]

    Contrastive decoding: Open-ended text generation as optimization

    Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text generation as optimization. arXiv preprint arXiv:2210.15097,

    work page arXiv

  11. [11]

    Rho-1: Not all tokens are what you need,

    12 Zhenghao Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, Yelong Shen, Ruochen Xu, Chen Lin, Yujiu Yang, Jian Jiao, Nan Duan, et al. Rho-1: Not all tokens are what you need.arXiv preprint arXiv:2404.07965, 2024a. Zicheng Lin, Tian Liang, Jiahao Xu, Qiuzhi Lin, Xing Wang, Ruilin Luo, Chufan Shi, Siheng Li, Yujiu Yang, and Zhaopeng Tu. Critical tokens matter: Toke...

    work page arXiv

  12. [12]

    Miao, C.-C

    Shen-Yun Miao, Chao-Chun Liang, and Keh-Yih Su. A diverse corpus for evaluating and developing english math word problem solvers.arXiv preprint arXiv:2106.15772,

    work page arXiv

  13. [13]

    Contrastive decoding improves reasoning in large language models

    Sean O’Brien and Mike Lewis. Contrastive decoding improves reasoning in large language models. arXiv preprint arXiv:2309.09117,

    work page arXiv

  14. [14]

    Are NLP Models really able to Solve Simple Math Word Problems?

    Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are nlp models really able to solve simple math word problems?arXiv preprint arXiv:2103.07191,

    work page internal anchor Pith review Pith/arXiv arXiv

  15. [15]

    Distillation contrastive decoding: Improving llms reasoning with contrastive decoding and distillation.arXiv preprint arXiv:2402.14874,

    Phuc Phan, Hieu Tran, and Long Phan. Distillation contrastive decoding: Improving llms reasoning with contrastive decoding and distillation.arXiv preprint arXiv:2402.14874,

    work page arXiv

  16. [16]

    Maximizing confidence alone improves reasoning.arXiv preprint arXiv:2505.22660, 2025

    Mihir Prabhudesai, Lili Chen, Alex Ippoliti, Katerina Fragkiadaki, Hao Liu, and Deepak Pathak. Maximizing confidence alone improves reasoning.arXiv preprint arXiv:2505.22660,

    work page arXiv

  17. [17]

    Llama 2: Open Foundation and Fine-Tuned Chat Models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models.arXiv preprint arXiv:2307.09288,

    work page internal anchor Pith review Pith/arXiv arXiv

  18. [18]

    Self-rewarding correction for mathematical reasoning.arXiv preprint arXiv:2502.19613,

    Wei Xiong, Hanning Zhang, Chenlu Ye, Lichang Chen, Nan Jiang, and Tong Zhang. Self-rewarding correction for mathematical reasoning.arXiv preprint arXiv:2502.19613,

    work page arXiv

  19. [19]

    WizardLM: Empowering large pre-trained language models to follow complex instructions

    URLhttps://arxiv.org/abs/2304.12244. An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jian- hong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement.arXiv preprint arXiv:2409.12122,

    work page internal anchor Pith review Pith/arXiv arXiv

  20. [20]

    Scaling Relationship on Learning Mathematical Reasoning with Large Language Models

    URLhttps://arxiv.org/abs/2308.01825. Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?arXiv preprint arXiv:2504.13837,

    work page internal anchor Pith review Pith/arXiv arXiv

  21. [21]

    Learning to Reason without External Rewards

    Xuandong Zhao, Zhewei Kang, Aosong Feng, Sergey Levine, and Dawn Song. Learning to reason without external rewards.arXiv preprint arXiv:2505.19590,

    work page internal anchor Pith review Pith/arXiv arXiv

  22. [22]

    16 A.2 Post-Training

    14 Appendix CONTENTS A Related Work 16 A.1 Contrastive Decoding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 A.2 Post-Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 B From KL Divergence to Contrast Score 17 C Connection between Selection, Contrast, and Training 17 D Relation to Reinforcement L...

    work page 2022

  23. [23]

    use large LLMs to generate synthetic instruction–response pairs at scale. In reasoning domains, SFT is often combined with rejection sampling (Yang et al., 2024; Guo et al., 2025), where model-generated trajectories are filtered for correctness before being used for supervision. Reinforcement Learning (RL).RL extends beyond SFT by optimizing models agains...

    work page 2024

  24. [24]

    These methods reduce human effort but still rely on preference-based optimization to guide alignment

    and Gen- eralized RPO (GRPO) (Guo et al., 2025). These methods reduce human effort but still rely on preference-based optimization to guide alignment. Unsupervised Post-Training.A related line of work explores unsupervised post-training methods that leverage internal model signals to improve performance without external supervision. For exam- ple, UPFT (J...

    work page 2025

  25. [25]

    T−1X t=0 γt rt # ,(21) wherer t is the reward at steptandγ∈[0,1]is a discount factor. Under the actor–critic framework, the policy gradient theorem states that ∇θJ(θ) =E πθ

    adapt model behavior using self-evaluated feedback. These approaches reduce reliance on human supervision but often require additional scaffolding or careful calibration of internal sig- nals. In contrast, LightReasoner provides a practical alternative to conventional post-training. By captur- ing token-level divergences between an Expert model and a weak...

    work page 2022

  26. [26]

    Entropy change with contrast score.As established in §D, our framework is equivalent to policy gradient when the contrast score is used as the advantage

    Thus, when high-probability ac- tions tend to carry positive advantage (positive covariance), entropy decreases; whereas if advantage is concentrated on low-probability actions (negative covariance), entropy can increase (Cui et al., 2025). Entropy change with contrast score.As established in §D, our framework is equivalent to policy gradient when the con...

    work page 2025

  27. [27]

    training set, a collection of grade-school math problems emphasizing step-by-step reasoning, to generate contrastive samples. To evaluate the transferability of the learned skills, we assess our models on a diverse suite of benchmarks: MATH(Hendrycks et al., 2021), a collection of high school competition problems;SV AMP(Patel et al.,

    work page 2021

  28. [28]

    This range spans from foundational arithmetic to expert-level reasoning, enabling a thorough assessment of both generalization and specialization

    andASDiv(Miao et al., 2021), testing numerical reasoning through linguistically var- ied arithmetic problems;Minerva Math(Lewkowycz et al., 2022), quantitative problems from ad- vanced STEM courses;OlympiadBench(He et al., 2024), challenging problems from international math olympiads; andMMLU-STEM(Hendrycks et al., 2020), which evaluates broad knowledge a...

    work page 2021

  29. [29]

    The same configuration was applied across all five backbone models studied in this paper to ensure comparability, while avoiding model-specific hyperparameter tuning

    Training was performed inbfloat16precision on a single NVIDIA H200 GPU, with the following runtime hyperparameters: batch size of 8 with gradient accumulation of 2 (effective batch size 16), learning rate5×10 −5, and 1000 total update steps. The same configuration was applied across all five backbone models studied in this paper to ensure comparability, w...

    work page 2024