Pith. sign in

REVIEW 3 major objections 7 minor 1 cited by

On the Generalization vs Fidelity Paradox in Knowledge Distillation

T0 review · 3 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Knowledge distillation raises zero-shot accuracy of sub-1B language models by up to 10% (22% on some tasks) while leaving the fidelity of the teacher's reasoning largely untransferred — the paper's generalization-fidelity paradox.

desk verdict Solid accuracy findings on KD for small LMs; the 'fidelity paradox' is BLEU overreach contradicted by the paper's own error analysis. read the letter →

arxiv 2505.15442 v2 pith:EMFBK6I2 submitted 2025-05-21 cs.CL

Suhas Kamasetty Ramesh , Ayan Sengupta , Tanmoy Chakraborty This is my paper · ORCID
classification cs.CL
keywords knowledgedistillationzero-shotgeneralizationreasoningfidelityteacher-studentagreementlanguagemodelcompressionBLEUmetricreverseKLdivergencegeneralization-fidelityparadox
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that knowledge distillation (KD), which trains a small student model to imitate a larger teacher, is a strong upgrade for the smallest language models: in zero-shot evaluation on 19 reasoning and instruction-following tasks, sub-1B students gain up to 10% average accuracy and up to 22% on individual tasks, while 7B students gain only about 1.3%. It further claims that the quality of the teacher, measured by raw benchmark performance, matters little for student gains, but the teacher's expertise on the target task matters a great deal: a task-unaware teacher can cut student accuracy by up to 40%. The paper's headline finding is a paradox: distilled students get more answers right without reproducing the teacher's reasoning steps, since teacher-student agreement and reasoning fidelity correlate weakly or not at all with student accuracy. If these claims hold, KD is an unusually cheap way to improve small deployed models, but 'distilled reasoning' cannot be inferred from accuracy gains alone.

What carries the argument

The argument runs on a fixed distillation protocol: supervised fine-tuning plus three sequence-level KD losses — SeqKD (maximize student probability of the teacher's full output sequence, $-\log P_S(y_T|x)$), RevKD (reverse KL divergence, $D_{\text{KL}}(P_S(y|x) \| P_T(y|x))$), and GKD (mix fixed teacher-generated data with the student's own on-policy generations under Jensen-Shannon divergence with $\lambda = 0.5$, $\beta = 0.5$). Two measurement instruments carry the paradox: teacher-student agreement, the fraction of matching top-1 predictions, and reasoning fidelity, the BLEU score between the teacher's and student's generated reasoning text, validated against sentence-embedding cosine similarity (Spearman 0.97). The ablations vary teacher-logit noise $\sigma$ and softmax temperature $\tau$, showing that moderate noise and $\tau = 2$ help while $\sigma = 5$ or $\tau = 5$ collapse student accuracy.

What would settle it

Score the same student-teacher output pairs on a reasoning-structure metric that ignores wording, such as human-annotated step equivalence or matched symbolic equations, and correlate that score with student accuracy. If structural fidelity correlates strongly with accuracy (say Spearman $\rho > 0.5$) where BLEU showed near zero, the paradox is an artifact of the metric; if the near-zero correlation persists, the paradox stands on a stronger footing.

Watch

Extended reading notes

Core claim

The central claim is empirical: across the Qwen-2.5 and LLaMA-3 families from 0.5B to 7B parameters, three knowledge-distillation methods (SeqKD, reverse-KL RevKD, and GKD) consistently beat supervised fine-tuning on mathematical reasoning, commonsense reasoning, and instruction-following benchmarks in zero-shot evaluation, with the benefits concentrated in the smallest models and a strong negative correlation (about $\rho = -0.66$ on math) between student size and KD gain. The paper also claims that teacher task expertise, not teacher performance, drives distillation success, and that teacher-student fidelity, measured by BLEU between teacher and student reasoning outputs, is not tied to student accuracy, which it frames as a generalization-fidelity paradox: distillation transfers task effectiveness without reliably transferring the teacher's structured decision-making. Statistical comparisons (t-tests, ANOVA, Spearman rank correlations) across all teacher-student pairs, together with ablations on teacher-logit noise and temperature smoothing, form the evidence base.

Load-bearing premise

The load-bearing premise is that BLEU, a text-overlap score, validly measures whether the student has reproduced the teacher's reasoning steps, even though the paper's own error analysis shows two correct but differently reasoned answers scoring BLEU 0.13.

Editorial extensions

If this is right

  • Sub-1B models can gain roughly 10 points of zero-shot accuracy from distillation, making KD a practical upgrade path for edge and on-device deployment rather than a marginal trick.
  • Larger models (7B) show only about 1.3% average gains, so for them the compute spent on distillation needs separate justification.
  • Choice of KD method matters little for math and commonsense benchmarks, but reverse-KL RevKD is the most consistent method overall, especially on instruction-following tasks.
  • A teacher strong on general benchmarks but not fine-tuned on the target domain can actively harm the student (up to 40% degradation), so teacher selection should prioritize task expertise over raw strength.
  • Accuracy gains and teacher-student fidelity are separate axes: a high-scoring student may reason in a way that diverges from the teacher, so fidelity scores should not be used as a proxy for transferred reasoning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the near-zero fidelity-performance correlation is metric-driven, then a structure-aware fidelity measure (symbolic equation matching or token-level rationale alignment) could change the paradox's strength; re-running the analysis with such a metric would test whether BLEU is the reason fidelity looks disconnected from accuracy.
  • The strong negative size-benefit correlation hints at a capacity threshold where the student already has the target skill; a testable extension is to take the best distilled small checkpoint and re-distill it from an even stronger teacher to see whether gains compound or saturate.
  • The noise and temperature ablations imply that distillation success is governed by the signal-to-noise quality of the teacher's logits; one could test this directly by measuring how student accuracy tracks the mutual information between teacher logits and task labels.
  • The fidelity gap suggests an objective-level fix the paper does not test: adding a reasoning-fidelity penalty to the KD loss (matching intermediate steps, not just final tokens) might close the paradox, and measuring whether students keep their accuracy gains would settle whether the trade-off is intrinsic or an artifact of current losses.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. This paper presents a large-scale empirical study of knowledge distillation (KD) for small language models. Using Qwen-2.5 (0.5B to 14B) and LLaMA-3 (1B to 8B) teachers and students, the authors fine-tune students with three KD methods (SeqKD, RevKD, GKD) that are compared against supervised fine-tuning, and evaluate zero-shot performance on mathematical reasoning, commonsense reasoning, and instruction-following tasks. The headline findings are that KD improves smaller students by up to about 10 absolute average points (with the largest gains for sub-1B models) while giving only marginal gains to 7B students; that teacher performance correlates weakly with student gains while teacher task expertise matters; and that there is an alleged 'generalization vs. fidelity paradox' in which students gain accuracy without preserving the teacher's reasoning. The paradox rests on a fidelity metric defined in Section 4 as the BLEU score between teacher and student generated reasoning outputs. The paper also reports ablations on logit noise and temperature smoothing. The accuracy-side findings are reported in detailed per-task tables (Tables 7 to 9).

Significance. If the accuracy findings hold, this is a useful large-scale descriptive benchmark: it covers two model families, three KD methods, and 19 tasks, it ships code, and it provides unusually detailed per-task tables. The descriptive pattern - larger gains for smaller students, diminishing returns at 7B, and the importance of teacher task adaptation (Figure 2, with up to 40% degradation from a task-unaware teacher) over raw teacher strength - is practically relevant and internally consistent with the tables. The noise and temperature ablations are well motivated and clearly reported. However, the paper's most distinctive claim, the generalization vs. fidelity paradox, is not supported by the evidence: the BLEU-based fidelity metric measures surface-text overlap, and the paper's own error analysis (Table 10) shows incorrect answers scoring high fidelity (0.77 and 0.79) while two correct solutions with different reasoning paths score 0.13. The validation against Sentence-BERT cosine similarity (r=0.97) only confirms agreement with another surface-form similarity measure. The accuracy contributions do not depend on this metric and remain plausible; the paradox framing does not.

major comments (3)
  1. [Section 4 ('Measures for quantifying teacher-student agreement and fidelity') and Section 5 ('How does KD impair…] The load-bearing premise of the fidelity paradox is that the BLEU score between teacher and student generated text measures 'reasoning fidelity' or 'structured decision-making.' Table 10 refutes this: in the first two examples the student reaches the wrong final answer yet scores BLEU 0.77 and 0.79 because the surface text is similar, while in the third example both models reason correctly through different valid paths yet score 0.13. Consequently, the weak correlations in Table 5 and the abstract claim that KD 'does not always maintain the structured decision-making processes of the teacher' are not supported; the data establish at most that KD does not enforce surface-level textual imitation. The reported r=0.97 agreement with Sentence-BERT cosine similarity does not rescue the metric, since cosine sentence similarity is likewise a surface-form measure. I recommend either validating the fidelity metric against an annotation of reasoning-step equivalence (e.g., whether the student's chain correctly and completely derives the final answer) or reframing the paradox claim in terms of surface/textual fidelity.
  2. [Tables 1-5; Section 5 ('Does KD depend on teacher performance?' and 'Does KD depend on student model size?')] The statistical reporting omits the sample sizes and units behind the t-tests, ANOVA, and Spearman correlations. In Table 2, the Spearman correlation between KD gains and teacher performance has at most three teacher sizes per student (3B/7B/14B for Qwen), and Table 3 correlates student performance with model size over four Qwen sizes or two LLaMA sizes; p-values of 0.0 at such N are not meaningful without stating N. No random seeds are reported, and the t-tests in Table 1 appear to pool a single training run per configuration. Because several headlines ('teacher performance has minimal impact', 'smaller models benefit more from KD') are inferential claims, the manuscript should state the number of independent observations, report variance across seeds, and temper the significance framing where N is tiny; the descriptive monotone trends in Figure 1 would still support the size-dependent claim.
  3. [Table 8 (commonsense reasoning results)] Several cells in Table 8 indicate collapsed or unstable runs that are inconsistent with the claim that KD consistently improves over SFT (Section 5, Figure 1): Qwen-1.5B distilled with RevKD from Qwen-3B scores 16.72 on Hellaswag and 50.60 on PiQA against SFT baselines of 63.90 and 76.93, and Qwen-3B distilled with GKD from Qwen-7B scores 11.67 on Hellaswag against 81.91 for SFT. These entries feed directly into the reported averages and into the t-tests of Table 1. The paper should report multiple seeds and variances, exclude or explain degenerate runs, or present medians; as reported, the reader cannot tell whether the averages describe typical behavior of the distillation methods.
minor comments (7)
  1. [Abstract and Introduction] The abstract says '14 complex reasoning tasks' while the Introduction and Section 4 state 19 tasks; please align the counts.
  2. [Tables 1, 3, 4 and Appendix A] There are typographical artifacts in the camera text: 'SV AMP' and 'ANOV A' in table headers, and 'Adversial' in the Hellaswag description in Appendix A.
  3. [Table 3 and Section 5] The table heading says 'student performance and student model size' while the text says 'between KD improvement and student size'; state explicitly whether the correlated variable is the raw score or the gain over SFT.
  4. [Figure 2 and Section 5 ('Does KD depend on teacher performance?')] The 'up to 40%' degradation claim needs a specification of whether the drop is absolute or relative, and the figure should report the underlying values.
  5. [Section 4 and Appendix B] Please state how per-instance BLEU is aggregated into the fidelity scores shown in Figures 4 and 9, and whether the same protocol is used for the instruction-following outputs that are scored with Rouge-L in Table 9.
  6. [Limitation section] The Limitation section asserts that 'the findings generalize well within these domains'; this is an unsupported empirical claim and should be qualified or removed.
  7. [Section 5 subsection headings] The heading 'How does KD impair teacher-student agreement?' presumes a negative effect, while Figures 3, 8, and 10 mostly show agreement increasing after KD; a neutral heading would be more accurate.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: accuracy claims are direct measurements; the only self-citation is non-load-bearing.

full rationale

The central claims—KD improves small-model accuracy by up to 10%, larger models see diminishing returns, teacher task-expertise matters more than raw teacher performance, and accuracy gains and BLEU-based teacher-student fidelity are weakly correlated—are established by direct zero-shot evaluation on held-out benchmarks. There is no fitted parameter later relabeled as a prediction, and no load-bearing result is imported from prior work by the same authors. The 'generalization vs fidelity paradox' is an empirical correlation result: 'reasoning fidelity' is defined in Section 4 as the BLEU score between teacher and student outputs, so the finding that performance and fidelity are weakly correlated is a measurement-level observation, not a result equivalent to its input by construction. The construct validity of BLEU as a proxy for 'structured decision-making' is questionable—Table 10 itself shows two different correct solutions receiving BLEU=0.13—but that is a correctness/validity concern, not circularity. The only self-citation (Sengupta et al., 2023) appears in the related-work enumeration and is not used to justify any central premise or to forbid alternatives. Hence no significant circularity; at most a minor, non-load-bearing self-citation.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters in the derivation sense; the hyperparameters are experimental choices, not fitted to the claims. The key domain assumptions are that LoRA fine-tuning represents the model family, that BLEU captures reasoning fidelity, and that single-run zero-shot scores are stable.

assumptions (3)
  • domain assumption LoRA adapters (r=8, alpha=16) trained on public datasets produce student models that represent the capabilities of the corresponding base model family.
    All SFT and KD training uses LoRA; if LoRA systematically changes the model more than full fine-tuning, the conclusions about model size and KD effectiveness may not transfer.
  • domain assumption BLEU score between teacher and student generated text measures reasoning fidelity and structured decision-making.
    The paper's paradox rests on this; Table 10 shows a counterexample where BLEU=0.13 for two correct but structurally different solutions, so the assumption is fragile.
  • domain assumption Zero-shot accuracy on fixed benchmark test sets is a stable estimate of model capability.
    No multiple seeds or confidence intervals are reported; the t-tests do not specify the sampling unit.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On the Generalization vs Fidelity Paradox in Knowledge Distillation." pith.science (2026). https://pith.science/paper/EMFBK6I2

@misc{pith2026250515442,
  author       = {Pith},
  title        = {Pith review of: On the Generalization vs Fidelity Paradox in Knowledge Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EMFBK6I2}},
  note         = {Machine review of arXiv:2505.15442}
}
abstract

Knowledge distillation (KD) is a key technique for compressing large language models into smaller ones while preserving performance. Despite the recent traction of KD research, its effectiveness for smaller language models (LMs) and the mechanisms driving knowledge transfer remain underexplored. In this work, we present the first large-scale empirical and statistical analysis of KD across models ranging from 0.5B to 7B parameters on 14 complex reasoning tasks in a zero-shot setting. Our findings reveal that KD can improve the average performance of smaller models by up to $10\%$, with a peak task specific gain of $22\%$, while providing only marginal benefits ($\sim 1.3\%$) for larger models. Surprisingly, teacher performance has a minimal impact on student outcomes, while teacher task expertise impacts KD effectiveness. A correlation study indicates that smaller LMs benefit more from KD, whereas larger LMs show diminished gains. Additionally, we uncover a misalignment between improvements in student performance and reasoning fidelity, suggesting that while KD enhances accuracy, it does not always maintain the structured decision-making processes of the teacher. Our ablation study further highlights the importance of teacher signals and logit smoothing in influencing students' performance after distillation. Overall, our study offers a comprehensive empirical and statistical assessment of KD, highlighting both its benefits and trade-offs when distilling knowledge from larger to smaller LMs.

Figures

Figures reproduced from arXiv: 2505.15442 by the authors.

Figure 1
Figure 1. Performance of different Qwen-2.5 student models across various (a) mathematical and (b) commonsense [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Impact of teacher task adaptation on distillation effectiveness. We evaluate the student performance [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Student agreement with teacher before KD and after KD on (a) mathematical and (b) commonsense [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Student reasoning fidelity with teacher model on (a) mathematical reasoning and (b) instruction following [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Impact of teacher input noise scale (σ) on student’s downstream performance on (a) mathematical and (b) commonsense reasoning tasks. 3B->0.5B 3B->1.5B 7B->0.5B 7B->1.5B 7B->3B Student Model 0 10 20 30 40 50 60 70 Average Accuracy = 1 = 2 = 5 (a) Mathematical reasoning …
Figure 6
Figure 6. Figure 6: Impact of smoothing factor (τ ) on student’s downstream performance on mathematical (a) and common￾sense (b) reasoning tasks. emphasize that temperature tuning is crucial, with optimal τ values varying by student size and task complexity. Impact of teacher-student gap …
Figure 7
Figure 7. Figure 7: Performance of LLaMA-3 student models on different mathematical reasoning (a) commonsense reasoning [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Student agreement with Qwen-14B teacher. [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Student reasoning fidelity for Qwen-14B teacher model. [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: Student agreement with LLaMA-3-8B teacher. [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 11
Figure 11. Figure 11: Student reasoning fidelity for LLaMA-3-8B teacher model. [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Different Teachers, Different Capabilities: Sub-1B On-Device Distillation for Structured Text Enrichment

    cs.AI 2026-07 conditional novelty 6.0 of 10

    Distilling an 8B reasoning teacher into a 0.6B student recovers most summary quality at ~50× speed, but teacher type—not scale alone—determines which capabilities transfer.

Reference graph

Works this paper leans on

46 extracted references · 12 canonical work pages · cited by 1 Pith paper

  1. [1]

    Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. 2024. On-policy distillation of language models: Learning from self-generated mistakes. In The Twelfth International Conference on Learning Representations

    2024

  2. [2]

    Raed Alharbi, Minh N Vu, and My T Thai. 2021. Learning interpretation with explainable knowledge distillation. In 2021 IEEE International Conference on Big Data (Big Data), pages 705--714. IEEE

    work page 2021

  3. [3]

    Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence

    2020

  4. [4]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. https://lmsys.org/blog/2023-03-30-vicuna/ Vicuna: An open-source chatbot impressing gpt-4 with 90\

    2023

  5. [5]

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044

    arXiv 2019

  6. [6]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803.05457v1

    arXiv 2018

  7. [7]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    arXiv 2021

  8. [8]

    Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J

    DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huaj...

    arXiv 2024

Show all 46 references
  1. [9]

    Lei Deng, Guoqi Li, Song Han, Luping Shi, and Yuan Xie. 2020. Model compression and hardware acceleration for neural networks: A comprehensive survey. Proceedings of the IEEE, 108(4):485--532

    2020

  2. [10]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  3. [11]

    Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. 2018. Born again neural networks. In International conference on machine learning, pages 1607--1616. PMLR

    2018

  4. [12]

    Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. 2024. Minillm: Knowledge distillation of large language models. In The Twelfth International Conference on Learning Representations

    2024

  5. [13]

    Chenqi Guo, Shiwei Zhong, Xiaofeng Liu, Qianli Feng, and Yinglong Ma. 2025. Why does knowledge distillation work? rethink its attention and fidelity mechanism. Expert Systems with Applications, 262:125579

    2025

  6. [14]

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531

    2015 arXiv

  7. [15]

    Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. 2023. https://doi.org/10.18653/v1/2023.acl-long.806 Unnatural instructions: Tuning language models with (almost) no human labor . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics...

    2023 doi

  8. [16]

    Mohammad Javad Hosseini, Hannaneh Hajishirzi, Oren Etzioni, and Nate Kushman. 2014. Learning to solve arithmetic word problems with verb categorization. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 523--533

    2014

  9. [17]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685

    2021 arXiv

  10. [18]

    Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Ka-Wei Lee. 2023. Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models. arXiv preprint arXiv:2304.01933

    2023 arXiv

  11. [19]

    Yoon Kim and Alexander M Rush. 2016. Sequence-level knowledge distillation. arXiv preprint arXiv:1606.07947

    2016 arXiv

  12. [20]

    Rik Koncel-Kedziorski, Hannaneh Hajishirzi, Ashish Sabharwal, Oren Etzioni, and Siena Dumas Ang. 2015. Parsing algebraic word problems into equations. Transactions of the Association for Computational Linguistics, 3:585--597

    2015

  13. [21]

    Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. 2016. https://doi.org/10.18653/v1/N16-1136 MAWPS : A math word problem repository . In Proceedings of the 2016 Conference of the North A merican Chapter of the Association for Computational L...

    2016 doi

  14. [22]

    Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p...

    2017

  15. [23]

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP

    2018

  16. [24]

    Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. 2020. https://doi.org/10.1609/aaai.v34i04.5963 Improved knowledge distillation via teacher assistant . Proceedings of the AAAI Conference on Artificial Intelligence, 34(04):...

    2020 doi

  17. [25]

    Utkarsh Ojha, Yuheng Li, Anirudh Sundara Rajan, Yingyu Liang, and Yong Jae Lee. 2023. What knowledge gets distilled in knowledge distillation? Advances in Neural Information Processing Systems, 36:11037--11048

    2023

  18. [26]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311--318

    2002

  19. [27]

    Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. Are nlp models really able to solve simple math word problems? arXiv preprint arXiv:2103.07191

    2021 arXiv

  20. [28]

    Leonardo Ranaldi and Andre Freitas. 2024. Aligning large and small language models via chain-of-thought reasoning. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1812--1827

    2024

  21. [29]

    Nils Reimers and Iryna Gurevych. 2019. https://arxiv.org/abs/1908.10084 Sentence-bert: Sentence embeddings using siamese bert-networks . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics

    2019 arXiv

  22. [30]

    Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. 2014. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550

    2014 arXiv

  23. [31]

    Subhro Roy and Dan Roth. 2015. Solving general arithmetic word problems. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1743--1752

    2015

  24. [32]

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99--106

    2021

  25. [33]

    Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019. https://doi.org/10.18653/v1/D19-1454 Social IQ a: Commonsense reasoning about social interactions . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9...

    2019 doi

  26. [34]

    Ayan Sengupta, Shantanu Dixit, Md Shad Akhtar, and Tanmoy Chakraborty. 2023. A good learner can teach better: Teacher-student collaborative knowledge distillation. In The Twelfth International Conference on Learning Representations

    2023

  27. [35]

    Samuel Stanton, Pavel Izmailov, Polina Kirichenko, Alexander A Alemi, and Andrew G Wilson. 2021. Does knowledge distillation really work? Advances in Neural Information Processing Systems, 34:6906--6919

    2021

  28. [36]

    Yijun Tian, Yikun Han, Xiusi Chen, Wei Wang, and Nitesh V Chawla. 2024. Beyond answers: Transferring reasoning capabilities to smaller llms using multi-teacher knowledge distillation

    2024

  29. [37]

    Smith, Daniel Khashabi, and Hannaneh Hajishirzi

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022 a . Self-instruct: Aligning language model with self generated instructions

    2022

  30. [38]

    Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuzni...

    2022

  31. [39]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...

    2020

  32. [40]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Me...

    2024 arXiv

  33. [41]

    Shan You, Chang Xu, Chao Xu, and Dacheng Tao. 2017. Learning from multiple teacher networks. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1285--1294

    2017

  34. [42]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830

    2019 arXiv

  35. [43]

    Qihuang Zhong, Liang Ding, Li Shen, Juhua Liu, Bo Du, and Dacheng Tao. 2024. Revisiting knowledge distillation for autoregressive language models. arXiv preprint arXiv:2402.11890

    2024 arXiv

  36. [44]

    Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. 2023. A survey on model compression for large language models. arXiv preprint arXiv:2308.07633

    2023 arXiv

  37. [45]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  38. [46]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.