Pith. sign in

REVIEW 3 major objections 5 minor 4 cited by

By preserving token-level entropy during supervised fine-tuning via adaptive self-distillation, CurioSFT claims to keep models explorative and add a 5.0-point average gain in the subsequent RL stage.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-08-03 05:23 UTC pith:OKLSLVT4

load-bearing objection Neat, well-theorized SFT modification with a real empirical effect, but the RL transfer claim is not causally isolated from better SFT initialization. the 3 major comments →

arxiv 2602.02244 v3 pith:OKLSLVT4 submitted 2026-02-02 cs.LG cs.CL

Entropy-Preserving Supervised Fine-Tuning via Adaptive Self-Distillation for Large Reasoning Models

Hao Wang , Hao Gu , Hongming Piao , Kaixiong Gong , Yuxiao Ye , Xiangyu Yue , Sirui Han , Yike Guo
show 1 more author
Dapeng Wu
This is my paper
classification cs.LG cs.CL
keywords entropy preservationself-distillationsupervised fine-tuningreinforcement learningreasoning modelsexplorationknowledge forgettingtoken-level temperature
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

SFT-then-RL is the standard recipe for building large reasoning models, but the cross-entropy objective in SFT pushes models toward overconfidence, collapsing the diversity that RL would otherwise explore. The paper argues that plain entropy regularization fails because it flattens all tokens toward uniform noise. Instead, it proposes CurioSFT: distill the student toward a self-generated, temperature-scaled teacher distribution, with a per-token temperature chosen by token entropy. The paper claims this preserves meaningful exploration—high entropy at reasoning connectors, stability at factual tokens—and that this 'effective entropy' translates into concrete RL gains, averaging 5.0 points over vanilla SFT+RL on math reasoning benchmarks. If correct, this means the two-stage paradigm's ceiling can be raised by fixing the SFT initialization rather than redesigning RL.

Core claim

The paper's central claim is that entropy collapse during supervised fine-tuning is the hidden bottleneck in the SFT-then-RL pipeline: standard cross-entropy training makes the model overconfident and locks it into a low-diversity mode, shrinking the solution space that reinforcement learning can explore. CurioSFT addresses this by distilling the student toward a teacher distribution formed by raising the softmax temperature of the student's own logits. Because entropy increases monotonically with temperature, this teacher is guaranteed to have higher entropy, and a theorem in the paper shows it is the unique higher-entropy distribution closest in KL divergence to the current policy—so it en

What carries the argument

The load-bearing mechanism is Self-Exploratory Distillation combined with Entropy-Guided Temperature Selection. The teacher distribution is constructed by rescaling the student's own logits with a temperature τ̂ > 1, giving a softmax with strictly higher entropy and the same relative ordering of token probabilities; a separate teacher model, updated by exponential moving average, stabilizes the target. The per-token temperature is not fixed: a sigmoid function of the teacher's token entropy (with a pivot H_pivot and maximum increment Δ_max) determines how much entropy to add, and a vectorized binary search finds the temperature that matches this target entropy. The student is trained with th

Load-bearing premise

Token-level entropy measured under the teacher distribution cleanly separates reasoning connectors (high entropy, worth exploring) from factual tokens (low entropy, must stay stable), and raising entropy at high-entropy positions creates beneficial exploration rather than ungrounded randomness.

What would settle it

Run CurioSFT with an inverted entropy-guided temperature schedule—boosting temperature on low-entropy (factual) tokens and stabilizing high-entropy (connector) tokens—and compare pass@k and final RL accuracy. If the inverted schedule preserves most of the gains, the token-role assumption is wrong. A complementary check: log the tokens that actually receive large Δt during training and verify, through rollouts, that those positions are where diverse completions produce correct answers; if the boosted positions do not correlate with alternate valid reasoning paths, the method may be simply addin

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • SFT checkpoints trained with CurioSFT provide a better initialization for RL: combined with GRPO, the in-distribution average rises from 53.7% to 58.7% (a 5.0-point gain), and the gain is largest on the hardest AIME benchmarks, suggesting preserved diversity is exactly what RL needs to search.
  • The token-adaptive temperature selection is what prevents knowledge forgetting: removing it (using a fixed temperature of 1.3) drops OOD accuracy by 2.0 points, and restricting naive entropy loss to the top-20% high-entropy tokens already recovers most of the OOD performance, confirming that low-entropy factual tokens must stay stable.
  • Entropy preservation measured at the token level translates directly into higher generation diversity: CurioSFT raises token entropy from 0.31 to 0.43 nats over vanilla SFT and lands at a higher accuracy–diversity frontier than GEM, DFT, PSFT, and entropy-loss baselines.
  • The method obtains consistent improvements across three base models (7B and 4B math-oriented models, and an 8B instruct model), indicating the mechanism is not tied to one architecture or size.
  • As stated in the paper's limitation section, the benefit is bounded by the base model's intrinsic exploration capability—if the model lacks latent diversity, self-distillation alone has little to amplify, and external signals would be needed.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The token-role hypothesis—high-entropy tokens are reasoning connectors worth amplifying, low-entropy tokens are facts worth stabilizing—is directly testable: one could log which positions receive large Δt and check whether varying those exact positions (e.g., by sampling) produces both correct alternatives and higher pass@k, or whether the entropy boost is incidental.
  • If CurioSFT's gains transfer to RL because entropy preservation at 'branching' tokens enlarges the search space, the same mechanism should apply to code generation and agentic tool-use tasks, where exploration across API calls or edit paths is valuable; the paper only evaluates math reasoning, so a code or agent benchmark would be a natural extension.
  • The KL-closest-higher-entropy-teacher theorem suggests a broader principle: any monotone transformation that raises entropy while minimizing KL to the current policy could serve as an exploration-preserving distillation target, and temperature scaling is just one instance; testing other monotone maps (e.g., power-scaling probabilities) might yield even better control over the entropy–knowledge tra
  • A cheaper diagnostic implied by the paper: if preserved SFT entropy is a reliable predictor of RL gains, model developers could screen SFT recipes by measuring token-entropy trajectories instead of running full RL, saving substantial compute during method development.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes CurioSFT, a supervised fine-tuning (SFT) method for reasoning LLMs that adds a self-distillation loss toward a temperature-scaled teacher distribution built from the model's own logits, with token-level temperatures selected to increase entropy mainly at high-entropy positions. The authors prove entropy monotonicity in temperature (Appendix B) and show temperature scaling is the KL-closest way to raise entropy (Appendix C). They report SFT gains of +2.5 in-distribution (ID) and +2.9 out-of-distribution (OOD) points over vanilla SFT on mathematical benchmarks, and a +5.0 ID gain after subsequent GRPO RL, plus robustness on two additional backbones.

Significance. If the empirical claims hold, the method would be a practical, relatively simple regularizer that improves both SFT and downstream RL without modifying the RL algorithm. The theoretical appendices are correct and self-contained for the idealized single-model setting: Theorem 1 (monotonicity) is derived cleanly, and Theorem 2 (temperature scaling as KL-closest higher-entropy distribution) is a valid convex-optimization argument. The paper ships code and reports consistent improvements across many benchmarks and two additional backbones, which strengthens the SFT-stage result. However, the headline RL-stage transfer claim is not causally isolated from SFT checkpoint quality, and the absence of error bars weakens the quantitative claims.

major comments (3)
  1. [§5.2, Table 3 (and abstract)] The claim that preserved exploration translates into +5.0 RL gains is confounded by SFT checkpoint quality. In Table 2, CurioSFT's SFT checkpoint already leads vanilla SFT by +2.5 ID average (51.0 vs 48.5); Table 3 reports final GRPO results from these different starting points. The +5.0 advantage could reflect a better SFT initialization rather than the entropy-preservation mechanism. The ablations that isolate the adaptive temperature and separate teacher are reported only after SFT (Table 2, 'w/o Adaptive Temp' and 'w/o Separate Teacher') and are not carried through RL, so the component claimed to drive exploration is never linked to the RL result in a controlled way. The Limitations section acknowledges overhead and base-model capability but not this missing control.
  2. [§4.1, Eq. (4) and Appendix C] Theorem 2 in Appendix C solves P1 for the current policy πθ (Eq. 10), proving the KL-closest higher-entropy teacher is temperature-scaled πθ. However, the implemented teacher in Eq. (4) uses a separate EMA parameterization φ, not the current policy; the theorem therefore does not exactly justify the implemented procedure unless φ = θ. Since the paper presents this result as the theoretical basis for the method, either extend the proof to the EMA teacher under a bounded-divergence assumption or state explicitly that the theorem applies only to the idealized single-model case.
  3. [§5, Tables 2–4 and Fig. 5] All results are single-run with no error bars or repeated seeds. Several reported differences are small (e.g., Table 4: Llama SFT average 32.5→33.0; Table 2: AMC 59.6→59.9; Table 3: OOD average 59.3→61.7), so without variance estimates the robustness and generalization claims are not fully supported. Fig. 5's H_pivot sensitivity also appears to be a single run. At least 2–3 seeds with mean±std should be reported for the main tables, or the computational constraints stated and a subset of runs repeated.
minor comments (5)
  1. [Figure 3 caption] Typo: 'Entropy-Guied' should be 'Entropy-Guided'.
  2. [Eq. (5)] The term 'K2-loss' is used without definition or citation. It would help to specify the divergence being minimized (e.g., squared log-ratio) and why this choice is preferred over standard KL for numerical stability.
  3. [Appendix D] The top-k entropy approximation renormalizes over the top-k tokens. This systematically underestimates entropy for heavy-tailed distributions; the bias should be discussed or empirically checked against the full-vocabulary entropy.
  4. [Section 2/3] The values 'offline 71% vs. online 76%' are introduced without defining what 'offline' and 'online' refer to in this context. This should be clarified in the text.
  5. [Appendix E] The training framework 'Verl' is likely a typo for 'VeRL' (the standard RL framework). Please correct the name for clarity.

Circularity Check

0 steps flagged

No significant circularity; the derivation chain is self-contained and the central claims rest on external benchmarks rather than on fitted inputs or self-citations.

full rationale

Walking the paper's derivation chain, I find no step where a prediction reduces by construction to an input or to a self-citation. The self-exploratory teacher in Eq. (4) is indeed generated from the model's own logits, but this is a regularization design, not a hidden fit: the claimed SFT gains (+2.5 ID, +2.9 OOD) and RL gains (+5.0) are measured on external benchmarks, not derived from Eq. (4) alone. The entropy-selectivity statement in Fig. 2(a/b) is a direct implementation of Eq. (7), where Delta_t is defined as an increasing function of H_t; the paper presents it as a mechanism description, not as an independent empirical prediction. The uniqueness theorem in Appendix C is derived internally via KKT conditions from the stated KL-plus-entropy objective; it does not import a self-citation or forbid alternatives by fiat. The RL-stage comparison is an empirical pipeline outcome, though it is subject to an SFT-accuracy confound (CurioSFT enters RL with a 51.0 ID average vs. 48.5 for vanilla SFT); that is a causal-attribution caveat, not a circular reduction. I also find no load-bearing self-citations in the paper. The limitation section acknowledges base-model dependence but does not reveal a hidden circular step. Overall, the central derivations are self-contained and the empirical claims are independently benchmarked, so the appropriate score is 0.

Axiom & Free-Parameter Ledger

7 free parameters · 5 axioms · 0 invented entities

The method introduces no new physical entities or external laws. Its free parameters are standard training hyperparameters, but H_pivot is selected via a benchmark sweep and is therefore the most consequential fitted quantity. The principal domain assumptions are about token entropy semantics and the self-distillation teacher being a valid guide for exploration.

free parameters (7)
  • Entropy pivot H_pivot = 1.2 nats (swept 0.4–1.4)
    Controls how many tokens receive large entropy increments; selected by a performance sweep in Section 5.4/Fig. 5, with no held-out validation split described.
  • Maximum entropy increment Δmax = 0.5 nats
    Sets the target entropy H_t + Δmax for saturated high-entropy tokens in Eq. (7).
  • Sigmoid scaling γ = 2.0
    Sharpness of the sigmoid in Eq. (7), chosen by hand.
  • SED loss weight α = 1.0
    Balances the SFT loss and the self-distillation loss in Eq. (6).
  • Teacher temperature clip range = [1.1, 1.5]
    Bounds of the binary search for the per-token teacher temperature (Section 5).
  • Teacher EMA interval and decay = n=5, μ=0.99
    Controls teacher update frequency and smoothing in Algorithm 1.
  • Top-k entropy approximation = k=512
    Vocabulary truncation used for entropy and temperature search, Appendix D.
axioms (5)
  • standard math Entropy of a softmax distribution is non-decreasing in sampling temperature.
    Proven in Appendix B from ∂H/∂τ = Var[z]/τ^3 ≥ 0; used to justify temperature-scaled teacher and the binary search.
  • standard math A temperature-scaled distribution is the unique KL-closest distribution with increased entropy.
    Proven in Appendix C via KKT conditions under full-support and entropy concavity; justifies the teacher form in Eq. (4).
  • domain assumption Token-level entropy separates useful exploration tokens (reasoning connectors) from factual tokens whose stability matters.
    Used in Section 4.2 and Eq. (7); attributed to Wang et al. 2025, not independently demonstrated in this paper.
  • domain assumption Distilling to a teacher distribution close to the current policy mitigates knowledge forgetting.
    Invoked in Section 4.1 and cited to Shenfeld et al. 2025; no direct measurement of forgetting is reported beyond OOD accuracy.
  • domain assumption The base model already contains latent exploration behavior sufficient to serve as its own teacher.
    Core to self-distillation; the paper itself acknowledges in the Limitations section that gains may shrink when the base model lacks prior knowledge.

reviewed 2026-08-03 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Entropy-Preserving Supervised Fine-Tuning via Adaptive Self-Distillation for Large Reasoning Models." pith.science (2026). https://pith.science/paper/OKLSLVT4

@misc{pith2026260202244,
  author       = {Pith},
  title        = {Pith review of: Entropy-Preserving Supervised Fine-Tuning via Adaptive Self-Distillation for Large Reasoning Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OKLSLVT4}},
  note         = {Machine review of arXiv:2602.02244}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

The standard post-training recipe for large reasoning models, supervised fine-tuning followed by reinforcement learning (SFT-then-RL), may limit the benefits of the RL stage: while SFT imitates expert demonstrations, it often causes overconfidence and reduces generation diversity, leaving RL with a narrowed solution space to explore. Adding entropy regularization during SFT is not a cure-all; it tends to flatten token distributions toward uniformity, increasing entropy without improving meaningful exploration capability. In this paper, we propose CurioSFT, an entropy-preserving SFT method designed to enhance exploration capabilities through intrinsic curiosity. It consists of (a) Self-Exploratory Distillation, which distills the model toward a self-generated, temperature-scaled teacher to encourage exploration within its capability; and (b) Entropy-Guided Temperature Selection, which adaptively adjusts distillation strength to mitigate knowledge forgetting by amplifying exploration at reasoning tokens while stabilizing factual tokens. Extensive experiments on mathematical reasoning tasks demonstrate that, in SFT stage, CurioSFT outperforms the vanilla SFT by 2.5 points on in-distribution tasks and 2.9 points on out-of-distribution tasks. We also verify that exploration capabilities preserved during SFT successfully translate into concrete gains in RL stage, yielding an average improvement of 5.0 points.

Figures

Figures reproduced from arXiv: 2602.02244 by Dapeng Wu, Hao Gu, Hao Wang, Hongming Piao, Kaixiong Gong, Sirui Han, Xiangyu Yue, Yike Guo, Yuxiao Ye.

Figure 1
Figure 1. Figure 1: Evaluation entropy and accuracy (Avg@8 in AIME 2024) across the SFT and RL stages. Cu￾rioSFT mitigates entropy collapse during SFT, and yields larger accuracy gains in the RL stage. agentic search (Shao et al., 2025a; Jin et al., 2025). In this paradigm, the Supervised Fine-tuning (SFT) stage aligns the model with domain-specific pat￾terns and required knowledge, thereby providing a superior initialization… view at source ↗
Figure 2
Figure 2. Figure 2: Expert token entropy and evaluation accuracy during the SFT stage. Compared to vanilla entropy loss, which uniformly encourages entropy across tokens, CurioSFT selectively increases entropy on high-entropy tokens while preserving low-entropy ones. We further observe that the increased token entropy induced by entropy loss does not translate into actual improvements in Pass@32 performance in our experiments… view at source ↗
Figure 3
Figure 3. Figure 3: Proposed solution: CurioSFT current model and a self-generated, higher-entropy teacher distribution. Formally, an LLM policy is obtained by applying the softmax function to the output logits zθ(· | st), as: πθ(y | st ; τ ) = exp zθ(y | st)/τ  P y ′∈V exp zθ(y ′ | st)/τ , (3) where τ denotes the sampling temperature and is typically fixed at 1.0 in standard SFT training. Leveraging the property that the e… view at source ↗
Figure 4
Figure 4. Figure 4: Accuracy vs. N-gram diversity. highest entropy by keeping away from overcon￾fident distributions. However, this occurs at the cost of ungrounded entropy, leading it to underper￾form CurioSFT on in-distribution tasks. PSFT and DFT improve training stability, but since they do not explicitly target exploration preservation, their overall improvements remain limited. As shown in [PITH_FULL_IMAGE:figures/full… view at source ↗
Figure 5
Figure 5. Figure 5: Sensitivity to the entropy pivot Hpivot. that CurioSFT generalizes across model families and sizes. 5.4 Hyperparameter Tuning Unlike entropy loss, which typically requires care￾ful tuning of the loss weight, CurioSFT adaptively computes a token-wise temperature and thus avoids introducing an additional sensitive coefficient. In practice, the key hyperparameter is the entropy pivot Hpivot in Eq. (7), which … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Learn where to Click from Yourself: On-Policy Self-Distillation for GUI Grounding

    cs.AI 2026-05 unverdicted novelty 7.0

    GUI-SD is the first on-policy self-distillation framework for GUI grounding that adds privileged bounding-box context and entropy-guided weighting to outperform GRPO methods on six benchmarks in accuracy and efficiency.

  2. Learn where to Click from Yourself: On-Policy Self-Distillation for GUI Grounding

    cs.AI 2026-05 accept novelty 7.0

    GUI-SD introduces on-policy self-distillation with visually enriched privileged context and entropy-guided weighting, outperforming GRPO and naive OPSD on six GUI grounding benchmarks while improving training efficiency.

  3. When RL Fails after SFT: Rejuvenating Model Plasticity for Robust SFT-to-RL Handoff

    cs.LG 2026-06 unverdicted novelty 5.0

    Excessive SFT reduces LLM plasticity for RL; Rejuvenation restores it via base-anchored fusion and targeted neuron resets, yielding better RL performance and OOD generalization.

  4. Beyond Retrieval: Learning Compact User Representations for Scalable LLM Personalization

    cs.IR 2026-06 unverdicted novelty 5.0

    TAP-PER encodes user preferences as lightweight learnable prefix embeddings that outperform prompt-based and adapter-based baselines on LaMP tasks with 130x fewer per-user parameters.

Reference graph

Works this paper leans on

51 extracted references · 35 linked inside Pith · cited by 3 Pith papers

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    Pith/arXiv arXiv 2023

  4. [4]

    Zeyuan Allen-Zhu and Yuanzhi Li. 2020. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. arXiv preprint arXiv:2012.09816

    Pith/arXiv arXiv 2020

  5. [5]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803.05457v1

    Pith/arXiv arXiv 2018

  6. [6]

    Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Yuchen Zhang, Jiacheng Chen, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, and 1 others. 2025. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01456

    Pith/arXiv arXiv 2025

  7. [7]

    Shrey Desai and Greg Durrett. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.21 Calibration of pre-trained transformers . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 295--302, Online. Association for Computational Linguistics

    doi:10.18653/v1/2020.emnlp-main.21 2020

  8. [8]

    Yihong Dong, Xue Jiang, Yongding Tao, Huanyu Liu, Kechi Zhang, Lili Mou, Rongyu Cao, Yingwei Ma, Jue Chen, Binhua Li, and 1 others. 2025. Rl-plus: Countering capability boundary collapse of llms in reinforcement learning with hybrid-policy optimization. arXiv preprint arXiv:2508.00222

    Pith/arXiv arXiv 2025

  9. [9]

    Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. 2025. Retool: Reinforcement learning for strategic tool use in llms. arXiv preprint arXiv:2504.11536

    Pith/arXiv arXiv 2025

  10. [10]

    Yuqian Fu, Tinghong Chen, Jiajun Chai, Xihuai Wang, Songjun Tu, Guojun Yin, Wei Lin, Qichao Zhang, Yuanheng Zhu, and Dongbin Zhao. 2025. Srft: A single-stage method with supervised and reinforcement fine-tuning for reasoning. arXiv preprint arXiv:2506.19767

    Pith/arXiv arXiv 2025

  11. [11]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    Pith/arXiv arXiv 2024

  12. [12]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948

    Pith/arXiv arXiv 2025

  13. [13]

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, and 1 others. 2024. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pages ...

    2024

  14. [14]

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874

    Pith/arXiv arXiv 2021

  15. [15]

    Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung-Yeung Shum. 2025. https://arxiv.org/abs/2503.24290 Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model . Preprint, arXiv:2503.24290

    Pith/arXiv arXiv 2025

  16. [16]

    Zeyu Huang, Tianhao Cheng, Zihan Qiu, Zili Wang, Yinghui Xu, Edoardo M Ponti, and Ivan Titov. 2025. Blending supervised and reinforcement fine-tuning with prefix sampling. arXiv preprint arXiv:2507.01679

    Pith/arXiv arXiv 2025

  17. [17]

    Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. 2021. https://doi.org/10.1162/tacl_a_00407 How can we know when language models know? on the calibration of language models for question answering . Transactions of the Association for Computational Linguistics, 9:962--977

    doi:10.1162/tacl_a_00407 2021

  18. [18]

    Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. 2025. Search-r1: Training llms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516

    Pith/arXiv arXiv 2025

  19. [19]

    Lou Jost. 2006. Entropy and diversity. Oikos, 113(2):363--375

    2006

  20. [20]

    Kyungyul Kim, ByeongMoon Ji, Doyoung Yoon, and Sangheum Hwang. 2021. https://arxiv.org/abs/2006.12000 Self-knowledge distillation with progressive refinement of targets . Preprint, arXiv:2006.12000

    Pith/arXiv arXiv 2021

  21. [21]

    Kimi, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, and 1 others. 2025. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.20534

    Pith/arXiv arXiv 2025

  22. [22]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles, pages 611--626

    2023

  23. [23]

    Hynek Kydlíček. 2025. https://github.com/huggingface/math-verify Math-verify: Math verification library

    2025

  24. [24]

    Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, and 1 others. 2022. Solving quantitative reasoning problems with language models. Advances in neural information processing systems, 35:3843--3857

    2022

  25. [25]

    Jia LI, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. 2024. Numinamath. [https://huggingface.co/AI-MO/NuminaMath-CoT](https://github.com/project-numina/aimo-progress-prize/blob/main/repo...

    2024

  26. [26]

    Ziniu Li, Congliang Chen, Tian Xu, Zeyu Qin, Jiancong Xiao, Zhi-Quan Luo, and Ruoyu Sun. 2024. Preserving diversity in supervised fine-tuning of large language models. arXiv preprint arXiv:2408.16673

    Pith/arXiv arXiv 2024

  27. [27]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, and 1 others. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437

    Pith/arXiv arXiv 2024

  28. [28]

    Kezhao Liu, Jason Klein Liu, Mingtao Chen, and Yiming Liu. 2025 a . https://arxiv.org/abs/2510.01555 Rethinking kl regularization in rlhf: From value estimation to gradient optimization . Preprint, arXiv:2510.01555

    arXiv 2025

  29. [29]

    Zihan Liu, Zhuolin Yang, Yang Chen, Chankyu Lee, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. 2025 b . Acereason-nemotron 1.1: Advancing math and code reasoning through sft and rl synergy. arXiv preprint arXiv:2506.13284

    Pith/arXiv arXiv 2025

  30. [30]

    Xingtai Lv, Yuxin Zuo, Youbang Sun, Hongyi Liu, Yuntian Wei, Zhekai Chen, Lixuan He, Xuekai Zhu, Kaiyan Zhang, Bingning Wang, and 1 others. 2025. Towards a unified view of large language model post-training. arXiv preprint arXiv:2509.04419

    arXiv 2025

  31. [31]

    Nick Mecklenburg, Yiyou Lin, Xiaoxiao Li, Daniel Holstein, Leonardo Nunes, Sara Malvar, Bruno Silva, Ranveer Chandra, Vijay Aski, Pavan Kumar Reddy Yannam, and 1 others. 2024. Injecting new knowledge into large language models via supervised fine-tuning. arXiv preprint arXiv:2404.00213

    Pith/arXiv arXiv 2024

  32. [32]

    OpenAI, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, and 1 others. 2025. gpt-oss-120b & gpt-oss-20b model card. arXiv preprint arXiv:2508.10925

    Pith/arXiv arXiv 2025

  33. [33]

    Minh Pham, Minsu Cho, Ameya Joshi, and Chinmay Hegde. 2022. Revisiting self-distillation. arXiv preprint arXiv:2206.08491

    Pith/arXiv arXiv 2022

  34. [34]

    Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. 2024. From r to q^* : Your language model is secretly a q-function. arXiv preprint arXiv:2404.12358

    Pith/arXiv arXiv 2024

  35. [35]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. 2024. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling

    2024

  36. [36]

    Rulin Shao, Akari Asai, Shannon Zejiang Shen, Hamish Ivison, Varsha Kishore, Jingming Zhuo, Xinran Zhao, Molly Park, Samuel G Finlayson, David Sontag, and 1 others. 2025 a . Dr tulu: Reinforcement learning with evolving rubrics for deep research. arXiv preprint arXiv:2511.19399

    Pith/arXiv arXiv 2025

  37. [37]

    Zhihong Shao, Yuxiang Luo, Chengda Lu, ZZ Ren, Jiewen Hu, Tian Ye, Zhibin Gou, Shirong Ma, and Xiaokang Zhang. 2025 b . Deepseekmath-v2: Towards self-verifiable mathematical reasoning. arXiv preprint arXiv:2511.22570

    arXiv 2025

  38. [38]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, and 1 others. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300

    Pith/arXiv arXiv 2024

  39. [39]

    Idan Shenfeld, Jyothish Pari, and Pulkit Agrawal. 2025. https://arxiv.org/abs/2509.04259 Rl's razor: Why online reinforcement learning forgets less . Preprint, arXiv:2509.04259

    Pith/arXiv arXiv 2025

  40. [40]

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. 2024. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19256

    Pith/arXiv arXiv 2024

  41. [41]

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, and 1 others. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805

    Pith/arXiv arXiv 2023

  42. [42]

    Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, and 1 others. 2025. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for llm reasoning. arXiv preprint arXiv:2506.01939

    Pith/arXiv arXiv 2025

  43. [43]

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, and 1 others. 2024. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. arXiv preprint arXiv:2406.01574

    Pith/arXiv arXiv 2024

  44. [44]

    Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652

    Pith/arXiv arXiv 2021

  45. [45]

    Yongliang Wu, Yizhou Zhou, Zhou Ziheng, Yingzhe Peng, Xinyu Ye, Xinting Hu, Wenbo Zhu, Lu Qi, Ming-Hsuan Yang, and Xu Yang. 2025. On the generalization of sft: A reinforcement learning perspective with reward rectification. arXiv preprint arXiv:2508.05629

    arXiv 2025

  46. [46]

    Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. 2025. Learning to reason under off-policy guidance. arXiv preprint arXiv:2504.14945

    Pith/arXiv arXiv 2025

  47. [47]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388

    Pith/arXiv arXiv 2025

  48. [48]

    An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, and 1 others. 2024. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12122

    Pith/arXiv arXiv 2024

  49. [49]

    Shaokun Zhang, Yi Dong, Jieyu Zhang, Jan Kautz, Bryan Catanzaro, Andrew Tao, Qingyun Wu, Zhiding Yu, and Guilin Liu. 2025 a . Nemotron-research-tool-n1: Exploring tool-using language models with reinforced reasoning. arXiv preprint arXiv:2505.00024

    Pith/arXiv arXiv 2025

  50. [50]

    Wenhao Zhang, Yuexiang Xie, Yuchang Sun, Yanxi Chen, Guoyin Wang, Yaliang Li, Bolin Ding, and Jingren Zhou. 2025 b . On-policy rl meets off-policy experts: Harmonizing supervised fine-tuning and reinforcement learning via dynamic weighting. https://arxiv.org/abs/2508.11408

    arXiv 2025

  51. [51]

    Wenhong Zhu, Ruobing Xie, Rui Wang, Xingwu Sun, Di Wang, and Pengfei Liu. 2025. Proximal supervised fine-tuning. arXiv preprint arXiv:2508.17784

    Pith/arXiv arXiv 2025

This paper was first reviewed by deepseek-v4-flash on August 3, 2026.