Pith. sign in

REVIEW 3 major objections 4 minor 4 cited by

MiCoTA: Bridging the Learnability Gap with Intermediate CoT and Teacher Assistants

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that distilling small models on intermediate-length reasoning traces from a merged mid-size teacher assistant beats both direct long-chain-of-thought distillation and the instruct baseline on math benchmarks.

desk verdict A practical distillation recipe with credible gains, but the central mechanism is under-supported because the paper never quantifies the length or accuracy of the mid-CoT training data. read the letter →

arxiv 2507.01887 v1 pith:IDV7GTK7 submitted 2025-07-02 cs.CL

Dongyi Ding , Tiannan Wang , Chenghao Zhu , Meiling Tao , Yuchen Eleanor Jiang , Wangchunshu Zhou This is my paper
classification cs.CL
keywords knowledgedistillationchain-of-thoughtreasoningsmalllanguagemodelsteacherassistantmodelmerginglong-to-shortmathlearnabilitygap
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes that small language models fail when trained on long chain-of-thought traces from large reasoners not just because the teacher is too big but because the reasoning traces are too long. It claims this 'learnability gap' can be bridged by training a mid-sized teacher assistant on the long traces, merging it with its own base weights to make it produce reasoning of about half the length, and then distilling those medium-length traces into the small student. On five math benchmarks this raises Qwen2.5-7B-Instruct from 40.54 to 49.36 average score and Qwen2.5-3B-Instruct from 31.92 to 43.29, beating both the instruct baseline and a long-plus-short CoT mix. The paper also reports that the medium-length data has lower bits-per-character for the student models, which it takes as evidence that the data is closer to what small models can absorb.

What carries the argument

The central object is the Mid-CoT Teacher Assistant, a mid-sized model (Qwen2.5-14B-Instruct) that is first fine-tuned on long CoT generated by a 32B strong teacher and then merged with its pre-fine-tuning weights. The merge, carried out with DARE pruning and TIES sign consensus, is the mechanism that shortens output length to about half while keeping reasoning quality, yielding the intermediate-length CoT dataset used to fine-tune student models. The argument also leans on bits-per-character as a probe: lower values for MiCoTA data are read as showing the traces are closer to the student's own distribution and therefore easier to learn.

What would settle it

Measure the token-length distribution and answer-verification accuracy of the merged teacher assistant's generated data; if the median trace length is not close to half that of the strong teacher's traces, or if a substantial share of traces are incomplete or incorrect, the proposed bridge loses empirical support.

Watch

Extended reading notes

Core claim

The central claim is that the length of the reasoning trace, not only the size of the teacher, determines whether a small model can learn from distillation. The paper's discovery is that a merged intermediate model—a teacher assistant fine-tuned on the strong teacher's long CoT and then merged with its pre-fine-tuning version using DARE sparsification with TIES sign consensus—produces CoT traces about half as long as the teacher's without a performance drop. Training students on these Mid-CoT traces outperforms direct long-CoT distillation across Qwen2.5-1.5B, 3B, and 7B models, and the student distributions align better with the MiCoTA data as measured by bits-per-character.

Load-bearing premise

The load-bearing premise is that the merged teacher assistant's traces are correct, complete, and genuinely about half as long as the strong teacher's traces; if the traces are noisy or not actually intermediate in length, the reported gains could come from simpler data rather than from the proposed bridge.

Editorial extensions

If this is right

  • Direct long-CoT distillation can degrade small models below their instruct baseline, while intermediate-length CoT data from a merged assistant improves them on the same tasks.
  • Model merging can act as a training-free length control, converting a long-CoT reasoner into a mid-CoT generator.
  • The learnability gap in distillation has a length component, so data design should match the trace length to the student's capacity.
  • The proposed recipe transfers across student scales from 1.5B to 7B within the same model family.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if distribution alignment is the active mechanism, then filtering the merged assistant's traces by answer correctness—which the paper does for the strong teacher's data but not for D_MiCoTA—could widen the observed gains.
  • Editorial inference: the same merge-then-distill recipe could be tested on non-math reasoning domains such as code generation or planning, where long traces also overwhelm small models; the paper only evaluates math benchmarks.
  • Editorial inference: bits-per-character on candidate distillation data could serve as a cheap pre-training screen, letting practitioners choose between teachers without running full distillation.
  • Editorial inference: the half-length property was demonstrated on Qwen models, so it is an open question whether the merge-induced shortening and its benefit persist across model families and optimization objectives.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes MiCoTA, a distillation framework for transferring long chain-of-thought (CoT) reasoning from a 32B strong teacher to small Qwen instruct models. The method fine-tunes a 14B teacher assistant on long CoT traces filtered by answer correctness and length (Eq. 1), merges the fine-tuned assistant with its base instruct checkpoint via DARE/TIES, uses this merged 'Mid-CoT Teacher Assistant' to generate a dataset D_MiCoTA of putatively intermediate-length traces, and then SFTs 1.5B/3B/7B students on this dataset. The central empirical claim is that MiCoTA beats direct long-CoT distillation and the instruct baselines on AIME2024, AMC, OlympiadBench, MATH-500, and GSM8K, with ablations against an unmerged 14B assistant and a reproduced Mix-Long baseline. The paper supplements these results with a BPC distributional analysis and a case study, and releases code and data.

Significance. If substantiated, MiCoTA is a practical and inexpensive recipe: model merging is used to shorten teacher traces instead of training a separate short-CoT generator, and the reported average gains are consistent across three model scales. The paper is commendable for including a reproduced baseline, ablations that separate the assistant-size and trace-length effects, and a public code/dataset link. However, the central mechanism is not yet fully established: the properties of D_MiCoTA (length distribution, correctness, completeness) are not measured, all headline numbers are unseeded point estimates, and the BPC analysis is a post-hoc explanation computed on base models. The contributions are therefore promising but require additional evidence before the mechanism and the reported gains can be taken at face value.

major comments (3)
  1. [Section 3.3, Eq. (1), Figure 2] The stress-test concern is well-founded. Dstrong is filtered by answer correctness and length in Eq. (1), but D_MiCoTA in Section 3.3 has no analogous filter, and Figure 2 supports the 'half-length, no performance drop' claim only illustratively. Table 2 shows the merged 14B assistant achieves 56.40 average versus 72.83 for the strong teacher, so the quality of the generated traces cannot be assumed. Please report the mean/median token length, the answer-accuracy rate, and the fraction of truncated or incomplete traces for D_MiCoTA; if no filter is applied, the improvement over Strong Teacher CoT could be due to shorter or noisier data rather than to the proposed bridge.
  2. [Section 4.3, Table 2] All results are single-run point estimates without error bars, standard deviations, or significance tests. The reported gains are not uniformly positive per benchmark (e.g., Qwen2.5-7B MiCoTA scores 70.40 on MATH-500 versus 72.60 for Instruct), so the average improvements of 3.47 and 3.93 need variance estimates or multiple seeds to be interpretable. Please add at least repeated-seed results for the main comparisons or bootstrap confidence intervals.
  3. [Section 4.5, Eq. (3), Table 4] The BPC argument is computed on the base instruct models, not on the fine-tuned students whose learnability is the subject of the paper, and lower BPC on MiCoTA data is also what one would expect if the data were shorter or closer to the base instruct distribution. It therefore does not discriminate between the proposed length-bridging mechanism and the alternative that the dataset is simply easier or more base-like. A stronger test would compute BPC of the trained student models on held-out reasoning traces, or control D_MiCoTA for length and answer correctness.
minor comments (4)
  1. [Abstract] The framework is named 'MiCoTAl' in the abstract and 'MiCoTA' elsewhere; please standardize the acronym.
  2. [Section 4.6, Figure 3] The caption refers to 'Owen2.5-7B'; this should be Qwen2.5-7B. In addition, the 'medium length' conclusion is qualitative; please include the numeric length statistics (mean/median tokens) that support it.
  3. [Section 4.5, Eq. (3)] Please state the base of the logarithm and clarify whether the denominator counts UTF-8 bytes or Unicode code points, since 'len-utf-8(T)' is not formally defined in the text.
  4. [Section 4.1, Table 1] The rows for Qwen2.5-14B-Instruct '+Teacher' correspond to the same model later used as the intermediate teacher assistant; this reuse should be stated explicitly to avoid confusion.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MiCoTA's gains are measured on external benchmarks, and the BPC and length analyses are post-hoc measurements, not fitted predictions.

full rationale

The paper does not derive a prediction from fitted parameters. Its central claim—SLMs trained on merged-TA intermediate-length CoT outperform those trained on strong-teacher long CoT—is an empirical comparison on AIME2024, AMC, OlympiadBench, MATH-500, and GSM8K. The two supporting analyses invoked as mechanism, the half-length property of the merged TA (Figure 2) and the lower BPC on MiCoTA data (Table 4), are measurements of the generated data, not quantities fitted to the benchmark outcomes. Eq. (1) filters D_strong, while D_MiCoTA in Sec. 3.3 is defined without an explicit filter; this is a transparency or data-quality limitation, not a circular step, because the downstream evaluation is external. The BPC metric is defined in Eq. (3) and cited to [54], whose authorship overlaps with a co-author; however the metric is parameter-free and independently defined in the paper, so the citation is not load-bearing. The name 'SLMs Learnability Gap' re-describes a phenomenon cited from [14, 45] but is not used as a derivation. No self-citation invokes a uniqueness theorem or forbids alternatives, and no fitted input is renamed as a prediction. Therefore no reduction of a claimed result to its inputs is exhibited; the appropriate finding is no significant circularity.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim sits on three premises the paper does not fully verify: (1) the merged TA produces half-length CoT with preserved quality, asserted via Figure 2 with no table of lengths or answer accuracy; (2) the MiCoTA dataset, unfiltered by correctness, is a clean supervision signal; (3) the BPC distribution alignment is the causal mechanism rather than a byproduct. Hand-picked TA scale and unpublished merge hyperparameters are the main design degrees of freedom.

free parameters (2)
  • Teacher assistant model scale = Qwen2.5-14B-Instruct (14B params)
    Chosen as an intermediate size between the 32B strong teacher and the 7B/3B/1.5B students; no sweep over TA sizes is reported, so the method's peak performance is tied to this hand-picked scale.
  • DARE/TIES merge hyperparameters = Not reported
    The paper specifies DARE pruning with TIES sign consensus but omits the pruning density and rescaling coefficients; these settings control the half-length property and are chosen by hand.
assumptions (3)
  • domain assumption Model merging of a model before and after SFT reduces output length by about half while preserving reasoning performance (Figure 2, based on Wu et al. [38]).
    The Mid-CoT Teacher Assistant depends on this property to generate intermediate-length CoT; the paper does not report its own length/accuracy verification for the merged 14B model.
  • ad hoc to paper The generated MiCoTA CoT traces are correct and complete enough to serve as training targets without filtering.
    Unlike D_strong (Eq. 1), D_MiCoTA is introduced without a correctness filter; if TA hallucinates or truncates, the student is trained on noisy targets.
  • domain assumption Perplexity/BPC alignment with the base model distribution is a proxy for distillability.
    The BPC analysis (Section 4.5) is used to explain why MiCoTA data is easier to learn, but no causal link is established.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MiCoTA: Bridging the Learnability Gap with Intermediate CoT and Teacher Assistants." pith.science (2026). https://pith.science/paper/IDV7GTK7

@misc{pith2026250701887,
  author       = {Pith},
  title        = {Pith review of: MiCoTA: Bridging the Learnability Gap with Intermediate CoT and Teacher Assistants},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IDV7GTK7}},
  note         = {Machine review of arXiv:2507.01887}
}
read the original abstract

Large language models (LLMs) excel at reasoning tasks requiring long thought sequences for planning, reflection, and refinement. However, their substantial model size and high computational demands are impractical for widespread deployment. Yet, small language models (SLMs) often struggle to learn long-form CoT reasoning due to their limited capacity, a phenomenon we refer to as the "SLMs Learnability Gap". To address this, we introduce \textbf{Mi}d-\textbf{Co}T \textbf{T}eacher \textbf{A}ssistant Distillation (MiCoTAl), a framework for improving long CoT distillation for SLMs. MiCoTA employs intermediate-sized models as teacher assistants and utilizes intermediate-length CoT sequences to bridge both the capacity and reasoning length gaps. Our experiments on downstream tasks demonstrate that although SLMs distilled from large teachers can perform poorly, by applying MiCoTA, they achieve significant improvements in reasoning performance. Specifically, Qwen2.5-7B-Instruct and Qwen2.5-3B-Instruct achieve an improvement of 3.47 and 3.93 respectively on average score on AIME2024, AMC, Olympiad, MATH-500 and GSM8K benchmarks. To better understand the mechanism behind MiCoTA, we perform a quantitative experiment demonstrating that our method produces data more closely aligned with base SLM distributions. Our insights pave the way for future research into long-CoT data distillation for SLMs.

Discussion (0). Sign in to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Curriculum Learning for Efficient Chain-of-Thought Distillation via Structure-Aware Masking and GRPO

    cs.LG 2026-02 conditional novelty 6.0 of 10

    A three-stage curriculum (masked reconstruction, GRPO compression, teacher-guided rewriting) distills long chain-of-thought into a concise 3B student, reporting 76.19% on GSM8K (up from 64.90%) with 167 output tokens ...

  2. Which Reasoning Trajectories Teach Students to Reason Better? A Simple Metric of Informative Alignment

    cs.CL 2026-01 conditional novelty 6.0 of 10

    Rank-Surprisal Ratio (RSR) correlates strongly (average Spearman 0.86) with post-distillation reasoning gains across five student models and trajectories from eleven teachers, outperforming existing selection metrics.

  3. Trust-Region Behavior Blending for On-Policy Distillation

    cs.LG 2026-05 unverdicted novelty 5.0 of 10

    TRB introduces a KL-trust-region warmup for on-policy distillation that blends toward teacher behavior early in training and anneals to zero, reporting the highest average performance across two math-reasoning distill...

  4. Which Reasoning Trajectories Teach Students to Reason Better? A Simple Metric of Informative Alignment

    cs.CL 2026-01 conditional novelty 5.0 of 10

    A trajectory for student-LLM distillation is better when its tokens are surprising but still high-ranked, and the ratio of average rank to average surprisal (RSR) selects such trajectories better than existing metrics.

Reference graph

Works this paper leans on

55 extracted references · 18 canonical work pages · cited by 3 Pith papers

  1. [1]

    Knowledge distillation: A good teacher is patient and consistent

    Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov. Knowledge distillation: A good teacher is patient and consistent. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10925–10934, 2022

    work page 2022

  2. [2]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URLhttps://lmsys.org/blog/2023-03-30-vicuna/

    work page 2023

  3. [3]

    Training verifiers to solve math word problems, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URLhttps://arxiv.org/abs/2110.14168

    arXiv 2021

  4. [4]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai D...

    arXiv 2025

  5. [5]

    Open r1: A fully open reproduction of deepseek-r1, January 2025

    Hugging Face. Open r1: A fully open reproduction of deepseek-r1, January 2025. URLhttps://github.com/ huggingface/open-r1

    2025

  6. [6]

    The language model evaluation harness, 07 2024

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The languag...

    arXiv 2024

  7. [7]

    Arcee’s MergeKit: A toolkit for merging large language models

    Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vladimir Karpukhin, Brian Benedict, Mark McQuade, and Jacob Solawetz. Arcee’s MergeKit: A toolkit for merging large language models. In Franck Dernoncourt, Daniel Preoţiuc-Pietro, and Anastasia Shimorina, editors,Proceedings of the 2024 Conference on Empirical Methods in Natural Languag...

    doi:10.18653/v1/202 2024

  8. [8]

    Rea- soning with language model is planning with world model

    Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Rea- soning with language model is planning with world model. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 8154–8173. ...

    doi:10.18653/v1/2023.emnlp-main.507 2023

Show all 55 references
  1. [9]

    Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems, 2024

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scienti...

    2024 arXiv

  2. [10]

    Measuring mathematical problem solving with the math dataset, 2021

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021. URLhttps://arxiv.org/ abs/2103.03874

    2021 arXiv

  3. [11]

    Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531, 2015

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531, 2015

    2015 arXiv

  4. [12]

    Tinybert: Distilling bert for natural language understanding

    Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. InFindings of the Association for Computational Linguistics: EMNLP 2020, pages 4163–4174. Association for Computational...

    2020

  5. [13]

    Sequence-level knowledge distillation

    Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. InProceedings of the 2016 conference on empirical methods in natural language processing, pages 1317–1327, 2016

    2016

  6. [14]

    Small models struggle to learn from strong reasoners, 2025

    Yuetai Li, Xiang Yue, Zhangchen Xu, Fengqing Jiang, Luyao Niu, Bill Yuchen Lin, Bhaskar Ramasubramanian, and Radha Poovendran. Small models struggle to learn from strong reasoners, 2025. URLhttps://arxiv.org/ abs/2502.12143

    2025

  7. [15]

    Let’s verify step by step.arXiv preprint arXiv:2305.20050, 2023

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step.arXiv preprint arXiv:2305.20050, 2023

    2023 arXiv

  8. [16]

    Improved knowledge distillation via teacher assistant

    Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 5191–5198, 2020

    2020

  9. [17]

    Learning to reason with llms, 2024

    OpenAI. Learning to reason with llms, 2024. URL https://openai.com/index/ learning-to-reason-with-llms/

    2024

  10. [18]

    Openai o1 system card, 2024

    OpenAI. Openai o1 system card, 2024. URLhttps://cdn.openai.com/o1-system-card-20241205.pdf

    2024

  11. [19]

    Distilling linguistic context for language model compression

    Geondo Park, Gyeongman Kim, and Eunho Yang. Distilling linguistic context for language model compression. In Conference on Empirical Methods in Natural Language Processing, 2021. URL https://api.semanticscholar. org/CorpusID:237563200

    2021

  12. [20]

    Mutual reasoning makes smaller llms stronger problem-solvers.CoRR, abs/2408.06195, 2024

    Zhenting Qi, Mingyuan Ma, Jiahang Xu, Li Lyna Zhang, Fan Yang, and Mao Yang. Mutual reasoning makes smaller llms stronger problem-solvers.CoRR, abs/2408.06195, 2024. doi: 10.48550/ARXIV.2408.06195. URL https://doi.org/10.48550/arXiv.2408.06195

    2024 doi arXiv

  13. [21]

    Fitnets: Hints for thin deep nets.arXiv preprint arXiv:1412.6550, 2015

    Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets.arXiv preprint arXiv:1412.6550, 2015

    2015 arXiv

  14. [22]

    Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.arXiv preprint arXiv:1910.01108, 2019

    Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.arXiv preprint arXiv:1910.01108, 2019

    1910 arXiv

  15. [23]

    Seed-thinking-v1

    ByteDance Seed, Yufeng Yuan, Yu Yue, Mingxuan Wang, Xiaochen Zuo, Jiaze Chen, Lin Yan, Wenyuan Xu, Chi Zhang, Xin Liu, et al. Seed-thinking-v1. 5: Advancing superb reasoning models with reinforcement learning. arXiv preprint arXiv:2504.13914, 2025

    2025

  16. [24]

    Densely guided knowledge distillation using multiple teacher assistants

    Wonchul Son, Jaemin Na, Junyong Choi, and Wonjun Hwang. Densely guided knowledge distillation using multiple teacher assistants. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9395–9404, 2021

    2021

  17. [25]

    Patient knowledge distillation for bert model compression.arXiv preprint arXiv:1908.09355, 2019

    Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for bert model compression.arXiv preprint arXiv:1908.09355, 2019

    1908 arXiv

  18. [26]

    Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy J. Lin. Distilling task- specific knowledge from bert into simple neural networks. ArXiv, abs/1903.12136, 2019. URL https: //api.semanticscholar.org/CorpusID:85543565. 12

    1903 arXiv

  19. [27]

    Stanfordalpaca: Aninstruction-followingllamamodel

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and TatsunoriB.Hashimoto. Stanfordalpaca: Aninstruction-followingllamamodel. https://github.com/tatsu-lab/ stanford_alpaca, 2023

    2023

  20. [28]

    Open Thoughts, January 2025

    Open Thoughts Team. Open Thoughts, January 2025

    2025

  21. [29]

    Qwen2.5: A party of foundation models, September 2024

    Qwen Team. Qwen2.5: A party of foundation models, September 2024. URLhttps://qwenlm.github.io/blog/ qwen2.5/

    2024

  22. [30]

    Qwq: Reflect deeply on the boundaries of the unknown, November 2024

    Qwen Team. Qwq: Reflect deeply on the boundaries of the unknown, November 2024. URLhttps://qwenlm. github.io/blog/qwq-32b-preview/

    2024

  23. [31]

    Qwq-32b: Embracing the power of reinforcement learning, March 2025

    Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025. URLhttps://qwenlm. github.io/blog/qwq-32b/

    2025

  24. [32]

    Alphazero-like tree-search can guide large language model decoding and training

    Ziyu Wan, Xidong Feng, Muning Wen, Stephen Marcus McAleer, Ying Wen, Weinan Zhang, and Jun Wang. Alphazero-like tree-search can guide large language model decoding and training. InForty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27,...

    2024

  25. [33]

    Efficientvlm: Fast and accurate vision-language models via knowledge distillation and modal-adaptive pruning.arXiv preprint arXiv:2210.07795, 2022

    Tiannan Wang, Wangchunshu Zhou, Yan Zeng, and Xinsong Zhang. Efficientvlm: Fast and accurate vision-language models via knowledge distillation and modal-adaptive pruning.arXiv preprint arXiv:2210.07795, 2022

    2022 arXiv

  26. [34]

    Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers.Advances in neural information processing systems, 33:5776–5788, 2020

    Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers.Advances in neural information processing systems, 33:5776–5788, 2020

    2020

  27. [35]

    Smith, Daniel Khashabi, and Hannaneh Hajishirzi

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st A...

    2023 doi

  28. [36]

    Chi, Quoc V

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models.arXiv preprint arXiv:2201.11903, 2022

    2022 arXiv

  29. [37]

    Light-r1: Curriculum sft, dpo and rl for long cot from scratch and beyond.arXiv preprint arXiv:2503.10460, 2025

    Liang Wen, Yunke Cai, Fenrui Xiao, Xin He, Qi An, Zhenyu Duan, Yimin Du, Junchen Liu, Lifu Tang, Xiaowei Lv, et al. Light-r1: Curriculum sft, dpo and rl for long cot from scratch and beyond.arXiv preprint arXiv:2503.10460, 2025

    2025 arXiv

  30. [38]

    Unlocking efficient long-to-short llm reasoning with model merging, 2025

    Han Wu, Yuxuan Yao, Shuqi Liu, Zehua Liu, Xiaojin Fu, Xiongwei Han, Xing Li, Hui-Ling Zhen, Tao Zhong, and Mingxuan Yuan. Unlocking efficient long-to-short llm reasoning with model merging, 2025. URLhttps: //arxiv.org/abs/2503.20641

    2025 arXiv

  31. [39]

    Wizardlm: Empowering large language models to follow complex instructions.arXiv preprint arXiv:2304.12244, 2023

    Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions.arXiv preprint arXiv:2304.12244, 2023

    2023 arXiv

  32. [40]

    BERT-of-Theseus: Compressing BERT by progressive module replacing

    Canwen Xu, Wangchunshu Zhou, Tao Ge, Furu Wei, and Ming Zhou. BERT-of-Theseus: Compressing BERT by progressive module replacing. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7859–7869. Association for Computational Lin...

    2020

  33. [41]

    McAuley, and Furu Wei

    Canwen Xu, Wangchunshu Zhou, Tao Ge, Ke Xu, Julian J. McAuley, and Furu Wei. Beyond preserved accuracy: Evaluating loyalty and robustness of bert compression. InProceedings of the 2021 Conference on Empirical Methods inNatural Language Processing (EMNLP), pages 10653–10659. As...

    2021

  34. [42]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  35. [43]

    Tree of thoughts: Deliberate problem solving with large language models

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Adva...

    2023

  36. [44]

    Limo: Less is more for reasoning,

    Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning,

  37. [45]

    Towards widening the distillation bottleneck for reasoning models.arXiv preprint arXiv:2503.01461, 2025

    Huifeng Yin, Yu Zhao, Minghao Wu, Xuanfan Ni, Bo Zeng, Hao Wang, Tianqi Shi, Liangying Shao, Chenyang Lyu, Longyue Wang, et al. Towards widening the distillation bottleneck for reasoning models.arXiv preprint arXiv:2503.01461, 2025

    2025 arXiv

  38. [46]

    Language models are super mario: Absorbing abilities from homologous models as a free lunch, 2024

    Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch, 2024. URLhttps://arxiv.org/abs/2311.03099

    2024 arXiv

  39. [47]

    Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer.arXiv preprint arXiv:1612.03928, 2016

    Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer.arXiv preprint arXiv:1612.03928, 2016

    2016 arXiv

  40. [48]

    Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. Star: Bootstrapping reasoning with reasoning. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors,Advances in Neural InformationProcessing Systems35: AnnualConferenceonNeuralInformation...

    2022

  41. [49]

    Towards the law of capacity gap in distilling language models

    Chen Zhang, Dawei Song, Zheyu Ye, and Yan Gao. Towards the law of capacity gap in distilling language models. arXiv preprint arXiv:2311.07052, 2023

    2023 arXiv

  42. [50]

    Lifting the curse of capacity gap in distilling language models.CoRR, abs/2305.12129, 2023

    Chen Zhang, Yang Yang, Jiahao Liu, Jingang Wang, Yunsen Xian, Benyou Wang, and Dawei Song. Lifting the curse of capacity gap in distilling language models.CoRR, abs/2305.12129, 2023. doi: 10.48550/ARXIV.2305.12129. URL https://doi.org/10.48550/arXiv.2305.12129

    2023 doi arXiv

  43. [51]

    Decoupled knowledge distillation

    Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. InProceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages 11953–11962, 2022

    2022

  44. [52]

    Llamafactory: Unified efficient fine-tuning of 100+ language models, 2024

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models, 2024. URLhttps://arxiv.org/abs/2403. 13372

    2024

  45. [53]

    BERT learns to teach: Knowledge distillation with meta learning

    Wangchunshu Zhou, Canwen Xu, and Julian McAuley. BERT learns to teach: Knowledge distillation with meta learning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL), pages 7037–7049. Association for Computational Linguistics, 2022

    2022

  46. [54]

    Is your llm outdated? a deep look at temporal generalization, 2025

    Chenghao Zhu, Nuo Chen, Yufei Gao, Yunyi Zhang, Prayag Tiwari, and Benyou Wang. Is your llm outdated? a deep look at temporal generalization, 2025. URLhttps://arxiv.org/abs/2405.08460. 14 Appendix A Training Details Table 5 List of all models used in our experiments, with Hugg...

    2025 arXiv

  47. [2025]

    URL https://arxiv.org/abs/2502.03387

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.