Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Logits-Based Finetuning

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper proposes a logits-based fine-tuning method whose target distribution is a normalized sum of top-K teacher logits and a ground-truth one-hot vector, and claims that training a student with KL divergence against this target…

desk verdict A simple KD-plus-ground-truth recipe with a clean small-scale ablation, but the headline gains are against off-the-shelf baselines, not same-data SFT. read the letter →

arxiv 2505.24461 v2 pith:5OTYQALK submitted 2025-05-30 cs.LG

Jingyao Li , Senqiao Yang , Sitong Wu , Han Shi , Chuanyang Zheng , Hong Xu , Jiaya Jia This is my paper
classification cs.LG
keywords knowledgedistillationlogits-basedfine-tuningtargetdistributionKLdivergencemathematicalreasoninglargelanguagemodelssupervisedteacherforcing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes a fine-tuning objective for small language models that replaces one-hot token labels with a normalized mixture of the teacher model's top-K token probabilities and the ground-truth one-hot vector. Training with KL divergence against this blended target, the authors report consistent gains over supervised fine-tuning: an average of 7.28 percentage points across nine math benchmarks, with the largest jumps on Mawps (+18 points) and TabMWP (+22.7 points). The work also releases a 1.2M-sample science logits dataset and two trained ScienceLLaMA models. The central bet is that the teacher's distribution, anchored by the correct label, transfers reasoning knowledge while avoiding hallucinated teacher outputs.

What carries the argument

The load-bearing object is the normalized mixture target $p_L(y_i)$ of Eq. (7). It is designed to satisfy four constraints: the ground-truth token keeps the largest probability, so greedy decoding still yields the correct token; the relative proportions among the teacher's top-K non-ground-truth candidates are unchanged from $p_T$; tokens outside the union of the ground truth and top-K receive zero probability under Top-K sparsification; and the result is a valid probability distribution. The training loss is the KL divergence $D_{KL}(p_L \| p_S^\theta)$, so the student is pushed to match a target that is anchored to correctness but shaped by teacher knowledge.

What would settle it

Run the same logits-based fine-tuning but compute teacher logits on-policy, from sequences the student itself generates; if the accuracy gain over SFT disappears or reverses, the teacher-forcing assumption is doing the work. A cheaper check: at evaluation, corrupt a token in the gold prefix and measure how much the student's next-token accuracy drops relative to an SFT baseline.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a token-level target formed by $p_L(y_i) = (p_T(y_i) + p_{GT}(y_i))\,/\,\|p_T(y_i) + p_{GT}(y_i)\|_1$ — teacher top-K probabilities plus the one-hot ground-truth label, renormalized — gives a student small language model a training signal that is both correct and linguistically diverse. The paper argues that pure SFT ignores valid alternative phrasings, while pure distillation inherits teacher hallucinations; the blended distribution keeps the argmax at the ground-truth token while preserving the teacher's relative preferences among the other top-K candidates. Trained with KL divergence on a 1.2M-sample science logits dataset, the resulting ScienceLLaMA-1B/3B models beat their SFT counterparts on all nine benchmarks tested, including jumps of 18 points on Mawps and 22.7 points on TabMWP.

Load-bearing premise

The teacher's logits are computed on gold-truth prefixes, and the method assumes these teacher-forced distributions remain the right targets when the student decodes from its own, possibly mistaken, prefixes.

Editorial extensions

If this is right

  • A fixed 1:1 blend of teacher and ground-truth probabilities yields better math reasoning than either signal alone, with no extra human annotation beyond existing SFT data.
  • Data efficiency improves: logits-based fine-tuning on 25% of MetaMath-GSM8K beats SFT on 50% of the same data, and half the data beats the full-dataset SFT baseline.
  • The released 1.2M logits dataset lets other training schemes reuse teacher knowledge without rerunning the 70B teacher, since only sparse top-K logits need to be stored.
  • Because the target's argmax is always the ground-truth token, the method should be less vulnerable to teacher hallucination than sequence-level distillation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The blend weight is fixed at 1:1; tuning it per token or per dataset could trade off correctness anchoring against teacher diversity, and likely changes the gain on open-ended versus single-answer tasks.
  • The teacher-forcing assumption suggests a natural extension: on-policy logits, computed from student-generated prefixes, might close the gap between training and decoding distributions and could be tested with the released dataset.
  • The method is demonstrated on math word problems, but the mechanism — anchoring a teacher distribution with a correctness signal — is generic and could transfer to code generation or structured-output tasks where a verifier can supply the anchor.
  • Storing only top-K teacher logits makes the approach storage-feasible; a natural follow-up is to ask how sensitive the gain is to K and to the teacher's temperature.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Logits-Based Finetuning (LFT), a fine-tuning objective for small LLMs in which the training target at each token is a normalized sum of the teacher's top-K softmax distribution and a one-hot ground-truth vector (Eq. 7), and the student is trained by minimizing the KL divergence to this target (Eq. 13). The authors construct a 1.2M-sample logits dataset from MetaMath-GSM8K and OpenMathInstruct2 using LLaMA3.1-70B-Instruct as teacher, train ScienceLLaMA-1B/3B, and report accuracy gains on nine math benchmarks, including headline gains on Mawps and TabMWP.

Significance. If the controlled results are representative, LFT is a simple and storage-friendly way to inject teacher knowledge during instruction tuning, and the released code and 1.2M logits dataset are useful community resources. The method's target distribution is transparent, and the constraint verification in Sec. A is elementary and correct. However, the paper's significance currently rests on headline numbers that are not backed by an apples-to-apples SFT baseline; the only controlled comparison shows a modest 2.0 pp gain on GSM8K. With corrected baselines, the contribution would be a modest but real improvement over SFT, and the public artifacts would be valuable for follow-up work.

major comments (3)
  1. [§4.4, Table 5] The central claim that LFT 'consistently outperforms prior SFT models' is not supported by the table used to make it. The row labeled 'LLaMA3.2-1b-It' in Table 5 reports the same numbers as the 'Baseline' row of Table 2 (GSM8K 46.9, MATH 31.6, College Math 18.6, Olympiad Bench 7.0, average 42.96), so it is the off-the-shelf instruct model, not a model SFT-trained on the 1.2M MetaMath-GSM8K + OpenMathInstruct2 dataset used for ScienceLLaMA. The 7.28% average gain is therefore a gain over an instruct baseline, not over SFT. The only controlled same-data comparison, Table 4 and Fig. 3, shows 56.1 vs 54.1 on GSM8K, a 2.0 pp gain. Add a same-data SFT baseline trained on the full 1.2M dataset and report all nine per-benchmark comparisons against it; otherwise the abstract and conclusion overstate the evidence.
  2. [Abstract; §4.4] The reported 18% gain on Mawps is inconsistent with Table 5. LLaMA3.2-1b-It scores 79.3 and ScienceLLaMA-1b scores 87.3, which is an 8.0 percentage-point gain or a 10.1% relative gain, not 18%. The 22.7% on TabMWP is 22.7 percentage points (33.4 to 56.1), so the text mixes absolute and relative gains without stating the convention. Specify whether all reported gains are absolute percentage points or relative improvements, and correct the Mawps number.
  3. [§4.4, Table 5 (3B row)] The phrase 'consistently outperforms' is not supported for the 3B model. ScienceLLaMA-3b is lower than LLaMA3.2-3b-It on GSM8K (81.0 vs 81.3), MATH (51.3 vs 51.7), and Olympiad Bench (16.1 vs 17.2), and the average gain is only 0.98 pp (64.48 vs 63.50). The paper should qualify the claimed consistency or provide additional evidence that the method, rather than the base model, is responsible for the reported gains.
minor comments (5)
  1. [§4.1, Table 5 caption] The Table 5 caption lists Socratic, StackExchange, Camel-AI, MathInstruct, GSM8K, MetaMath, MetaMath-GSM8K, and OpenMathInstruct2 as if they were evaluation benchmarks, but these are training datasets; the actual benchmarks are GSM8K, MATH, College Math, Olympiad Bench, Svamp, ASDiv, Mawps, Carp, and TabMWP.
  2. [§3.3, Algorithm 1] The value of K for the top-K sparsification is never specified in the main text or appendix; please report the K used and state whether it was tuned.
  3. [§4.2] Training details list batch size 1 and learning rate 2e-5 but omit the number of epochs and total optimization steps; these are needed to reproduce the training runs.
  4. [§3.2, Eq. (7)] The target distribution uses an unweighted sum of p_T and p_GT; the paper does not discuss whether equal weighting is optimal or whether a search over the combination weight was performed, so please state this explicitly.
  5. [§3.2, Fig. 2] Teacher logits are computed with teacher forcing on gold prefixes, while the student decodes autoregressively from its own prefixes at evaluation; a sentence acknowledging this exposure gap would help readers assess the method's generalizability.

Circularity Check

1 steps flagged · score 1.0 of 10

No load-bearing circularity: the pL target is explicitly defined, its four constraints follow immediately from the construction, and the benchmark gains are empirical rather than forced by the equations.

  1. self definitional [Sec. 3.2, Eq. (7) and Constraints 1-4; Appendix A, Eq. (14)]
    "We propose our probability distributionp L as follows: pL(yi) = pT(yi) +pGT(yi) ∥pT(yi) +pGT(yi)∥1 ,(7) ... We define this distribution because it satisfies the following constraints. ... It can be easily demonstrated that pL satisfies the four constraints outlined above. Details are in Sec. A."

    Each of the four claimed constraints is a direct algebraic consequence of how pL is constructed in Eq. (7): the one-hot pGT forces the ground-truth index to be largest (Constraint 1); uniform L1 scaling with pGT(j)=0 for j≠yi yields pLj(yi)/pLk(yi) = pTj(yi)/pTk(yi) (Constraint 2, as Appendix A's Eq. (14) shows); Top-K sparsification zeroes all j outside S (Constraint 3); and the L1 normalization is exactly the sum-to-one property (Constraint 4). Appendix A's 'verification' therefore restates the definition's arithmetic rather than providing independent grounding for the target distribution. This is a benign property check, not a prediction: the benchmark gains are external empirical measurements and are not derived from these constraints, so the circularity is non-load-bearing.

full rationale

The derivation chain is otherwise self-contained. The target distribution pL in Eq. (7) is explicitly defined as a normalized mixture of top-K teacher logits and a one-hot ground-truth vector, and the four constraints it 'satisfies' are immediate consequences of that construction (one-hot dominance, uniform scaling of teacher ratios, sparsification, and L1 normalization), as Appendix A itself demonstrates with Eq. (14). This is the one definitional step I flag, but it is not load-bearing: no parameter is fitted to the benchmarks, the teacher (LLaMA3.1-70B-Instruct) is external, and the reported gains are empirical outcomes on external benchmarks, so no 'prediction' reduces to an input by construction. No load-bearing self-citation exists: the authors' prior works (RoboCoder, MotCoder, VisionZip, DAPE) appear only in the related-work enumeration, and no uniqueness theorem or ansatz is imported from them. The headline 'average improvement of 7.28%' is, however, computed against off-the-shelf LLaMA3.2-1b-It in Table 5 rather than a same-data SFT baseline; the only controlled same-data comparison (Table 4 and Fig. 3) shows a 2.0 pp GSM8K gain (56.1 vs 54.1). The '18% on Mawps' figure also mixes relative (10.1%) and absolute (8.0 pp) arithmetic. These are baseline-selection and reporting weaknesses that undercut the strength of the empirical claim, but they are correctness concerns, not circularity, because the measured gains are not forced by the paper's equations.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central construction has one free parameter (top-K, unspecified) and a hand-set blend weight; the teacher-forcing procedure is a domain assumption that is not analyzed.

free parameters (2)
  • interpolation weight = 0.5 (implicit, Eq. 7)
    pL is an equal-weight normalized sum of teacher distribution and a one-hot; this 50/50 blend is chosen by hand, not tuned.
  • top-K sparsification K = unspecified
    Algorithm 1 uses 'Top-K teacher logits' but K is never stated; the central method depends on this hyperparameter.
assumptions (3)
  • domain assumption Teacher-forced logits conditioned on gold prefixes are suitable training targets for an autoregressive student (Sec. 3.2, Fig. 2)
    The method feeds ground truth tokens to the teacher when computing pT, assuming these distributions transfer to the student's own generation.
  • domain assumption The teacher model's top-K token probabilities encode useful linguistic diversity even when the full answer is wrong (Sec. 3.1, Table 1)
    The paper motivates the method by claiming teacher logits preserve diversity, but Table 1 shows teacher hallucinations; the value of such logits is assumed.
  • standard math KL divergence is an appropriate loss for the mixed target distribution (Eq. 13)
    Standard choice in distillation; no justification beyond convention.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Logits-Based Finetuning." pith.science (2026). https://pith.science/paper/5OTYQALK

@misc{pith2026250524461,
  author       = {Pith},
  title        = {Pith review of: Logits-Based Finetuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5OTYQALK}},
  note         = {Machine review of arXiv:2505.24461}
}
read the original abstract

In recent years, developing compact and efficient large language models (LLMs) has emerged as a thriving area of research. Traditional Supervised Fine-Tuning (SFT), which relies on singular ground truth labels, often fails to capture token-level dependencies and linguistic diversity. To address these limitations, we propose a logits-based fine-tuning framework that integrates the strengths of supervised learning and knowledge distillation. Our approach constructs enriched training targets by combining teacher logits with ground truth labels, preserving both correctness and linguistic diversity. This ensures more reliable and effective training. We constructed a large-scale 1.2M logits dataset and trained a series of science-focused models. Experimental results demonstrate that our method achieves significant improvements, with accuracy gains of 18% on Mawps and 22.7% on TabMWP. Across nine widely used mathematical benchmarks, our method consistently outperforms prior SFT models, achieving an average improvement of 7.28%. Codes are available at https://github.com/dvlab-research/Logits-Based-Finetuning.

Figures

Figures reproduced from arXiv: 2505.24461 by the authors.

Figure 1
Figure 1. Conceptual overview of our logits-based dis [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Illustration of token probability distribution generation. The input tokens concated with ground truth [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Ablation of our logits-based finetune com [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. VisionThink: Smart and Efficient Vision Language Model via Reinforcement Learning

    cs.CV 2025-07 conditional novelty 6.0 of 10

    A vision-language model learns via reinforcement learning when to upscale a low-resolution image, cutting visual tokens roughly in half while preserving accuracy on most benchmarks.

Reference graph

Works this paper leans on

49 extracted references · 17 canonical work pages · cited by 1 Pith paper

  1. [1]

    Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos, Matthieu Geist, and Olivier Bachem. 2024. https://arxiv.org/abs/2306.13649 On-policy distillation of language models: Learning from self-generated mistakes . Preprint, arXiv:2306.13649

    arXiv 2024

  2. [2]

    AI@Meta. 2024. https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md Llama 3 model card

    2024

  3. [3]

    Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023. https://arxiv.org/abs/2305.10403 Palm 2 technical report . arXiv preprint arXiv:2305.10403

    arXiv 2023

  4. [4]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, et al. 2020. https://papers.nips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html Language models are few-shot learners . In Proceedings of NeurIPS

    work page 2020

  5. [5]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. https://lmsys.org/blog/2023-03-30-vicuna/ Vicuna: An open-source chatbot impressing gpt-4 with 90\

    2023

  6. [6]

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. https://arxiv.org/abs/2204.02311 Palm: Scaling language modeling with pathways . arXiv preprint arXiv:2204.02311

    arXiv 2022

  7. [7]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    arXiv 2021

  8. [8]

    Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. 2024. https://arxiv.org/abs/2306.08543 Minillm: Knowledge distillation of large language models . Preprint, arXiv:2306.08543

    arXiv 2024

Show all 49 references
  1. [9]

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. 2024. Olympiadbench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal s...

    2024

  2. [10]

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the MATH dataset. In NeurIPS Datasets and Benchmarks

    2021

  3. [11]

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. https://arxiv.org/abs/1503.02531 Distilling the knowledge in a neural network . Preprint, arXiv:1503.02531

    2015 arXiv

  4. [12]

    Ku, Qian Liu, and Wenhu Chen

    Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max W.F. Ku, Qian Liu, and Wenhu Chen. 2024. https://openreview.net/forum?id=skLtdUVaJa Mantis: Interleaved multi-image instruction tuning . Transactions on Machine Learning Research, 2024

    2024

  5. [13]

    Gou Jianping, Yu Baosheng, Stephen J Maybank, and Tao Dacheng. 2021. https://link.springer.com/article/10.1007/s11263-021-01453-z Knowledge distillation: A survey . IJCV

    2021 doi

  6. [14]

    Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020. https://aclanthology.org/2020.findings-emnlp.372/ Tinybert: Distilling bert for natural language understanding . In Findings of EMNLP

    2020

  7. [15]

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. https://arxiv.org/abs/2001.08361 Scaling laws for neural language models . arXiv preprint arXiv:2001.08361

    2020 arXiv

  8. [16]

    Yoon Kim and Alexander M. Rush. 2016 a . https://arxiv.org/abs/1606.07947 Sequence-level knowledge distillation . Preprint, arXiv:1606.07947

    2016 arXiv

  9. [17]

    Yoon Kim and Alexander M Rush. 2016 b . https://aclanthology.org/D16-1139.pdf Sequence-level knowledge distillation . In Proceedings of EMNLP

    2016

  10. [18]

    Distillm: Towards streamlined distillation for large language models

    Jongwoo Ko, Sungnyun Kim, Tianyi Chen, and Se-Young Yun. Distillm: Towards streamlined distillation for large language models. In Forty-first International Conference on Machine Learning

  11. [19]

    Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. 2016. Mawps: A math word problem repository. In Proceedings of the 2016 conference of the north american chapter of the association for computational linguistics: human language technologies,...

    2016

  12. [20]

    Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2023. Camel: Communicative agents for "mind" exploration of large language model society. In Thirty-seventh Conference on Neural Information Processing Systems

    2023

  13. [21]

    Jingyao Li, Pengguang Chen, Sitong Wu, Chuanyang Zheng, Hong Xu, and Jiaya Jia. 2024 a . https://arxiv.org/abs/2406.03757 Robocoder: Robotic learning from basic skills to general tasks with large language models . Preprint, arXiv:2406.03757

    2024 arXiv

  14. [22]

    Jingyao Li, Pengguang Chen, Bin Xia, Hong Xu, and Jiaya Jia. 2024 b . https://arxiv.org/abs/2312.15960 Motcoder: Elevating large language models with modular of thought for challenging programming tasks . Preprint, arXiv:2312.15960

    2024 arXiv

  15. [23]

    Kevin J Liang, Weituo Hao, Dinghan Shen, Yufan Zhou, Weizhu Chen, Changyou Chen, and Lawrence Carin. 2021. https://openreview.net/forum?id=UFGEelJkLu5 Mix \ kd \ : Towards efficient distillation of large-scale language models . In Proceedings of ICLR

    2021

  16. [24]

    Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. 2023. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. In International Conference on Learning Representations (ICLR)

    2023

  17. [25]

    Shen-yun Miao, Chao-Chun Liang, and Keh-Yih Su. 2020. A diverse corpus for evaluating and developing english math word problem solvers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 975--984

    2020

  18. [26]

    OpenAI. 2023. https://arxiv.org/abs/2303.08774 GPT-4 technical report . Preprint, arXiv:2303.08774

    2023 arXiv

  19. [27]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and...

    2022 arXiv

  20. [28]

    Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. https://doi.org/10.18653/v1/2021.naacl-main.168 Are NLP models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Ling...

    2021 doi

  21. [29]

    Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. 2023. https://arxiv.org/abs/2304.03277 Instruction tuning with GPT-4 . arXiv preprint arXiv:2304.03277

    2023 arXiv

  22. [30]

    Andrei A Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirkpatrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. 2015. https://arxiv.org/pdf/1511.06295.pdf Policy distillation . arXiv preprint arXiv:1511.06295

    2015 arXiv

  23. [31]

    Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. https://arxiv.org/pdf/1910.01108.pdf DistilBERT , a distilled version of bert: smaller, faster, cheaper and lighter . arXiv preprint arXiv:1910.01108

    2019 arXiv

  24. [32]

    Kaitao Song, Hao Sun, Xu Tan, Tao Qin, Jianfeng Lu, Hongzhi Liu, and Tie-Yan Liu. 2020. https://arxiv.org/pdf/2004.12817.pdf LightPAFF : A two-stage distillation framework for pre-training and fine-tuning . arXiv preprint arXiv:2004.12817

    2020 arXiv

  25. [33]

    Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019. https://aclanthology.org/D19-1441 Patient knowledge distillation for BERT model compression . In Proceedings EMNLP

    2019

  26. [34]

    Zhengyang Tang, Xingxing Zhang, Benyou Wang, and Furu Wei. 2024. Mathscale: Scaling instruction tuning for mathematical reasoning. In ICML . OpenReview.net

    2024

  27. [35]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford Alpaca : An instruction-following LLaMA model. https://github.com/tatsu-lab/stanford_alpaca

    2023

  28. [36]

    Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. 2022. https://arxiv.org/abs/2201.08239 Lamda: Language models for dialog applications . arXiv preprint arXiv:2201.08239

    2022 arXiv

  29. [37]

    Shubham Toshniwal, Wei Du, Ivan Moshkov, Branislav Kisacanin, Alexan Ayrapetyan, and Igor Gitman. 2024. https://arxiv.org/abs/2410.01560 Openmathinstruct-2: Accelerating ai for math with massive open-source instruction data . Preprint, arXiv:2410.01560

    2024 arXiv

  30. [38]

    Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, and Furu Wei. 2021. https://aclanthology.org/2021.findings-acl.188 M ini LM v2: Multi-head self-attention relation distillation for compressing pretrained transformers . In Findings of ACL

    2021

  31. [39]

    Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. https://proceedings.neurips.cc/paper/2020/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf MiniLM : Deep self-attention distillation for task-agnostic compression of pre-trained transformers . In Proceed...

    2020

  32. [40]

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022. https://openreview.net/pdf?id=yzkSU5zdwD Emergent abilities of large language models . Transactions on Machine Learning Research

    2022

  33. [41]

    Yuqiao Wen, Zichao Li, Wenyu Du, and Lili Mou. 2023. https://aclanthology.org/2023.acl-long.605.pdf f-divergence minimization for sequence-level knowledge distillation . In Proceedings of ACL

    2023

  34. [42]

    Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. 2024. Visionzip: Longer is better but not necessary in vision language models. arXiv preprint arXiv:2412.04467

    2024

  35. [43]

    Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2023. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.12284

    2023 arXiv

  36. [44]

    Xiang Yue, Tuney Zheng, Ge Zhang, and Wenhu Chen. 2024. Mammoth2: Scaling instructions from the web. Advances in Neural Information Processing Systems

    2024

  37. [45]

    Beichen Zhang, Kun Zhou, Xilin Wei, Wayne Xin Zhao, Jing Sha, Shijin Wang, and Ji-Rong Wen. 2023 a . Evaluating and improving tool-augmented computation-intensive math reasoning. arXiv preprint arXiv:2306.02408

    2023 arXiv

  38. [46]

    Rongzhi Zhang, Jiaming Shen, Tianqi Liu, Jialu Liu, Michael Bendersky, Marc Najork, and Chao Zhang. 2023 b . https://arxiv.org/pdf/2305.05010.pdf Do not blindly imitate the teacher: Using perturbed loss for knowledge distillation . arXiv preprint arXiv:2305.05010

    2023 arXiv

  39. [47]

    Chuanyang Zheng, Yihang Gao, Han Shi, Minbin Huang, Jingyao Li, Jing Xiong, Xiaozhe Ren, Michael Ng, Xin Jiang, Zhenguo Li, and Yu Li. 2024. https://arxiv.org/abs/2405.14722 Dape: Data-adaptive positional encoding for length extrapolation . Preprint, arXiv:2405.14722

    2024 arXiv

  40. [48]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  41. [49]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.