Pith. sign in

REVIEW 3 major objections 6 minor 58 references

Put Teacher in Student's Shoes: Cross-Distillation for Ultra-compact Model Compression Framework

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A three-stage pipeline of attention-based token pruning, cross-distillation, and module-wise INT8 quantization compresses BERT-class NLU to 1.91 MB while staying within a few points of BERT-base.

desk verdict Genuine deployment story, but the token-pruning metric enabling the 1.91 MB claim is not well-defined as written. read the letter →

arxiv 2507.04636 v1 pith:DDUWG4FU submitted 2025-07-07 cs.CL

Maolin Wang , Jun Chu , Sicong Xie , Xiaoling Zang , Yao Zhao , Wenliang Zhong , Xiangyu Zhao This is my paper
classification cs.CL
keywords knowledgedistillationmodelcompressionedgeAIBERTALBERTquantizationtokenpruningnaturallanguageunderstanding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a BERT-class model can be compressed to roughly 1.91 MB of storage—the smallest reported footprint for natural language understanding—while keeping accuracy within a few points of BERT-base on Chinese benchmarks. The compression is achieved by a three-stage pipeline: hard token pruning driven by attention-derived importance scores, a novel cross-distillation training step, and module-wise INT8 quantization. The cross-distillation step is the load-bearing novelty: the teacher model is updated alongside the student with a smaller learning rate, so its guidance adapts to the student's evolving capability instead of remaining static. If the claim holds, it would make real-time, privacy-preserving on-device NLU practical; the paper reports deployment across millions of daily active devices in a large production app ecosystem.

What carries the argument

The central mechanism is the coupled cross-distillation loss, $\mathcal{L}_{\text{teacher}}=\mathcal{L}^{\text{task}}_{\text{teacher}}+\beta_1\,\mathrm{MSE}(f_t^{(h)}, f_s^{(h)})+\beta_2\,\mathrm{KL}(f_t, f_s)$ and $\mathcal{L}_{\text{student}}=\mathcal{L}^{\text{task}}_{\text{student}}+\beta_1\,\mathrm{MSE}(f_t^{(h)}, f_s^{(h)})+\beta_2\,\mathrm{KL}(f_t, f_s)$, trained alternately with a small teacher learning rate $\lambda_1$ and a larger student learning rate $\lambda_2$. The MSE term aligns hidden representations before pooling and the KL term aligns downstream logits, so the teacher's guidance is continuously re-targeted to the student's current state. Two supporting mechanisms carry the size reduction: the token importance score $I(w)$ from Eq. (2), which ranks vocabulary entries using the student's last-layer attention probabilities and justifies pruning to 3,072 tokens, and the module-wise quantization objective of Eq. (5), which minimizes reconstruction error across blocks of layers rather than per matrix.

What would settle it

Re-run the pruning step on the CLUE tasks themselves, or on a held-out corpus, and compare the accuracy at a 3,072-token vocabulary against random vocabulary pruning of the same size. If random pruning matches or beats attention-based pruning, the importance ranking is not doing the work assigned to it; equivalently, retraining the student from scratch at a 2,048-token vocabulary would show whether the sharp accuracy drop in the paper's NER-only curve appears on the actual benchmark tasks as well.

Watch

Extended reading notes

Core claim

From an ALBERT-2 style student with factorized embeddings and cross-layer parameter sharing, the paper builds EI-BERT by pruning the vocabulary from 21,128 to 3,072 tokens using a token importance score computed from the student's last-layer attention, then training via cross-distillation from a fine-tuned BERT-base teacher, then applying module-wise INT8 post-training quantization. On the CLUE benchmark the resulting 1.91 MB model averages 63.97 on sentence-pair tasks versus BERT-base's 71.53, and 50.04 on machine reading comprehension versus 72.08, while a non-quantized variant with the same pruning averages 64.48 and 50.38. The paper's central discovery is that the cross-distillation update—alternating a small-learning-rate teacher update and a larger-learning-rate student update against shared MSE and KL losses—lets an ultra-compact student absorb task knowledge from a much larger teacher without the capacity-gap failure ordinary distillation exhibits.

Load-bearing premise

The load-bearing premise is that the token importance score computed from the student's last-layer attention on a corpus identifies which vocabulary entries can be safely removed; if that ranking does not transfer to the evaluation tasks, the 3,072-token vocabulary that makes the 1.91 MB size possible would instead cost far more accuracy than the paper reports.

Editorial extensions

If this is right

  • The complete pipeline yields a 213x compression ratio and a 136x inference speedup over the BERT-base teacher, making sub-100 ms on-device responses feasible.
  • The 1.91 MB model stays within single digits of BERT-base on CLUE sentence-pair tasks and about 22 points lower on machine reading comprehension, so the method trades accuracy for extreme compactness unevenly across task types.
  • Deployment results report 21 million real-time requests per day across 8.4 million daily active devices with 95th-percentile latency below 80 ms, suggesting the accuracy tradeoff is acceptable in production.
  • Privacy-sensitive NLU becomes viable on-device: the paper reports 98.2% accuracy parity with a cloud model and fully localized intent recognition, eliminating cloud data transmission for sensitive queries.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the 3,072-token cutoff is calibrated only on an NER validation curve, a natural extension would be per-task or per-corpus importance scores to see whether the vocabulary size should differ by task; the paper does not test this.
  • The deployment retains the teacher's task-specific head (stated as a limitation in Appendix D), so the 1.91 MB figure is not a fully self-contained student; a student with its own head might need extra parameters or sacrifice accuracy.
  • The alternating teacher/student updates resemble a co-adaptation loop, suggesting a testable variant: annealing the teacher learning rate $\lambda_1$ to zero over training to see whether the student's final accuracy improves or degrades.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper presents EI-BERT, a compression pipeline that reduces a Chinese BERT-class model to roughly 1.91 MB. The pipeline consists of (i) hard token pruning based on attention-derived importance scores computed from a student model, (ii) a 'cross-distillation' procedure in which the fine-tuned teacher's task-specific heads are copied into the student and both teacher and student are updated alternately with coupled MSE/KL losses, and (iii) module-wise INT8 post-training quantization. The authors evaluate on the CLUE benchmark and internal Alipay NLU tasks, compare against TinyBERT, ALBERT, TED, TA, and Meta-KD baselines, and report deployment in Alipay's Edge Recommendation system since January 2024, serving 8.4 million daily active devices.

Significance. If the size and accuracy numbers hold, the result is practically significant: a BERT-class model under 2 MB that stays within roughly 3-7 points of BERT-base on CLUE sentence tasks, with a 136x speedup and industrial deployment. The paper's strengths are the external CLUE evaluation, the internal Alipay datasets, and the staged ablations (KD, PI-KD, CrossKD, CrossKD-TP, EI-BERT) that isolate the contribution of each pipeline component. The main weaknesses are the under-specified pruning metric that enables the size claim, and the absence of variance or significance information for several close headline numbers. The deployment metrics, if credible, add a practical contribution, though they are not independently verifiable from the manuscript.

major comments (3)
  1. [Section 2.2, Eq. (2)] The token importance score I(w) in Eq. (2) is not a well-defined ranking metric as written. Since A^(h)(w, x_j) is a softmax probability over keys for the query token w, the inner sum over j equals 1 for every occurrence of w. Consequently, the attention weights cancel from the score: depending on the unspecified normalization over corpus occurrences, I(w) is either a length-weighted frequency count or an average of inverse sentence lengths, and it cannot rank tokens by attention importance. The text also does not specify whether w is meant to be a query position or a key position; if the intended score is attention received by w as a key, the summation should be over the query index rather than over j. Because pruning to 3,072 tokens is the step that produces the 1.91 MB model, this ambiguity directly affects the paper's central size/accuracy claim. Please specify the exact corpus-level aggregation and the role of w in Eq. (2), and report the resulting ranking on a sample of tokens.
  2. [Section 3.7, Figure 4] The choice of the 3,072-token vocabulary cutoff is validated only on the internal NER task shown in Figure 4; no CLUE task is used to check that the pruned vocabulary retains the tokens needed for sentence-pair, WSC, or MRC benchmarks. Since all EI-BERT results in Table 4 are produced after this pruning, the reader cannot tell whether the reported accuracy of 63.97 on sentence tasks and 50.04 on MRC tasks is robust to the pruning criterion or an artifact of an NER-specific cutoff. Please add pruning-curve results on at least a subset of CLUE tasks (for example TNEWS, AFQMC, and CMRC), or otherwise demonstrate that the pruned token set transfers across the benchmark tasks.
  3. [Table 4, Section 3.5] Several headline comparisons are within run-to-run noise. EI-BERT's sentence-task average of 63.97 exceeds TinyBERT4's 63.94 by 0.03 points, and EI-BERT is 0.51 points below CrossKD-TP and 0.76 points below CrossKD on the same average. No standard deviations, number of seeds, or significance tests are reported anywhere in the paper, despite the text claiming EI-BERT 'outperforms' TinyBERT4. Given that the central claim is that a 1.91 MB model retains competitive accuracy, please report variance or significance tests for the key comparisons in Table 4 (and ideally Table 5), or soften the superiority claims accordingly.
minor comments (6)
  1. [Abstract, Section 3.5, Section 3.8, Figure 3] The model size is reported as 1.91 MB in the Abstract and in Section 3.8, but as 1.92 MB in Section 3.5 and in Figure 3(a); please reconcile these numbers.
  2. [Section 2.2, Eq. (1)] Eq. (1) writes softmax(...)_{(i,j)} without specifying the normalization axis; please state explicitly that the softmax is over the key index j for each query i, and align the notation of Eq. (2) with that convention.
  3. [Section 2.2, text after Eq. (2)] The sentence 'with K, H, and n_k representing the number of layers, sentence count, attention heads, and the length of the k-th sentence, respectively' lists four concepts for three variables; please clarify which symbol denotes which quantity.
  4. [Section 3.3] The text says the student model employs '128-dim embeddings and 1024 intermediate layers'; this almost certainly means an intermediate size of 1024, and the wording should be corrected.
  5. [Figure 3(b)] The caption states that EI-BERT is evaluated in terms of Integer Operations (IOPs) while other models are assessed using Floating Point Operations (FLOPs); please explain how the IOP count is computed and why the comparison is fair.
  6. [Appendix D] Appendix D lists retaining the teacher's task-specific head during deployment as a limitation, yet Section 2.3.1 describes integrating exactly those heads into the student; please clarify whether this refers to inherited parameters or to a separate runtime dependency on the original teacher.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: compression pipeline is benchmarked externally; mutual teacher-student losses are the proposed method, not a derivation from own inputs.

full rationale

This is an empirical engineering study whose claims are checked against external benchmarks (CLUE, internal Alipay data) and against baseline models, not derived from its own fitted outputs. The mutual losses in Eqs. (3)-(4), where the teacher loss includes student outputs and the student loss includes teacher outputs, are the proposed cross-distillation training procedure rather than a circular reduction of a predicted quantity to a fitted input. The hard-token-pruning importance score in Eq. (2) is under-specified, and the 3,072-token cutoff is validated only on the NER curve in Figure 4, but that is a measurement and selection concern rather than circularity. The module-wise quantization objective in Eq. (5) is standard reconstruction-error minimization against the full-precision module, so the quantized result is not defined in terms of the final accuracy it is used to explain. Self-citations to the authors' prior work ([38], [39], [40], [52], [53]) appear in peripheral contexts such as surveys, multimodal compression, and embedding-dimensionality search; they are not load-bearing for the 1.91 MB claim, which rests on the reported pipeline and external comparisons. No uniqueness theorem or ansatz is imported from the authors' own prior work as an unexamined premise. The limitation noted in Appendix D, namely that the teacher's task-specific head is retained during deployment, is an honest architectural constraint and does not make the derivation circular. No specific equation or fitted parameter can be exhibited as reducing a claimed prediction to its own input, so the appropriate finding is no significant circularity.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The framework is an empirical engineering contribution: its claimed accuracy/size tradeoff depends on tuned distillation weights and learning rates, a pruning threshold selected from data, learned quantization step sizes, and four unproven domain assumptions about teacher head portability, attention-based token importance, teacher updatability, and the accuracy-preserving nature of reconstruction-based PTQ. None of these is derived from first principles.

free parameters (5)
  • distillation weights beta1, beta2 = not reported; searched over [1,10] for NLU and [0.5,5.0] for generation
    Chosen by hyperparameter search (Section 3.3); values are not fixed by theory and are not reported per task, so the reported accuracies depend on them.
  • teacher learning rate lambda1 = range 1e-7 to 1e-6
    Small LR for teacher updating during mutual interplay (Section 3.3); exact per-task values are not reported.
  • student learning rate lambda2 = range 1e-4 to 5e-5
    Larger LR for student updating (Section 3.3); exact per-task values are not reported.
  • pruned vocabulary size = 3072
    Selected from the accuracy/parameter tradeoff in Figure 4 on the internal NER dataset; dropping to 2048 causes a significant accuracy loss, so the cutoff materially affects the 1.91 MB claim.
  • per-module quantization step sizes s_n = learned per module
    Step sizes are optimized in Eqs. 5-6 during module-wise INT8 quantization; the module partition N is not specified.
assumptions (4)
  • domain assumption Fine-tuning the teacher sufficiently trains its downstream task-specific layers so they can be directly reused by the student.
    Section 2.3.1 states this assumption explicitly ("we assume that fine-tuning the teacher model sufficiently trains these downstream layers"). If it fails, parameter integration could transfer misleading logits.
  • domain assumption Student last-layer attention probabilities over a corpus identify vocabulary tokens that can be pruned without major accuracy loss.
    Section 2.2, Eq. 2, plus Section 3.7. The 3,072-token threshold is validated only on the NER curve, not across all tasks.
  • domain assumption Updating the teacher with a small learning rate during distillation preserves or improves its guiding ability.
    Section 2.3.2, Eq. 3: teacher loss includes alignment to student outputs; no analysis guarantees the teacher does not drift toward the student and lose task knowledge.
  • domain assumption Minimizing per-module reconstruction error after INT8 quantization preserves downstream task accuracy.
    Section 2.4, Eq. 5: the objective is output reconstruction, not task loss; the link to accuracy is empirical only.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Put Teacher in Student's Shoes: Cross-Distillation for Ultra-compact Model Compression Framework." pith.science (2026). https://pith.science/paper/DDUWG4FU

@misc{pith2026250704636,
  author       = {Pith},
  title        = {Pith review of: Put Teacher in Student's Shoes: Cross-Distillation for Ultra-compact Model Compression Framework},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DDUWG4FU}},
  note         = {Machine review of arXiv:2507.04636}
}
read the original abstract

In the era of mobile computing, deploying efficient Natural Language Processing (NLP) models in resource-restricted edge settings presents significant challenges, particularly in environments requiring strict privacy compliance, real-time responsiveness, and diverse multi-tasking capabilities. These challenges create a fundamental need for ultra-compact models that maintain strong performance across various NLP tasks while adhering to stringent memory constraints. To this end, we introduce Edge ultra-lIte BERT framework (EI-BERT) with a novel cross-distillation method. EI-BERT efficiently compresses models through a comprehensive pipeline including hard token pruning, cross-distillation and parameter quantization. Specifically, the cross-distillation method uniquely positions the teacher model to understand the student model's perspective, ensuring efficient knowledge transfer through parameter integration and the mutual interplay between models. Through extensive experiments, we achieve a remarkably compact BERT-based model of only 1.91 MB - the smallest to date for Natural Language Understanding (NLU) tasks. This ultra-compact model has been successfully deployed across multiple scenarios within the Alipay ecosystem, demonstrating significant improvements in real-world applications. For example, it has been integrated into Alipay's live Edge Recommendation system since January 2024, currently serving the app's recommendation traffic across \textbf{8.4 million daily active devices}.

Figures

Figures reproduced from arXiv: 2507.04636 by the authors.

Figure 1
Figure 1. The proposed ultra-compact model compression framework. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Cross-distillation featuring parameter integration and teacher-student mutual interplay advances beyond conventional [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Overall Model Efficiency from Storage, and Model [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Hard token pruning evaluation on NER Sector of [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Example Application Scenarios of EI-BERT in Ali [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Some specific examples of Alipay Datasets. [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

58 extracted references · 40 canonical work pages

  1. [1]

    Emna Baccour, Naram Mhaisen, Alaa Awad Abdellatif, Aiman Erbad, Amr Mo- hamed, Mounir Hamdi, and Mohsen Guizani. 2022. Pervasive AI for IoT appli- cations: A survey on resource-efficient distributed artificial intelligence. IEEE Communications Surveys & Tutorials (2022)

    work page 2022

  2. [2]

    Bo Chen, Xiangyu Zhao, Yejing Wang, Wenqi Fan, Huifeng Guo, and Ruiming Tang. 2022. Automated machine learning for deep recommender systems: A survey. arXiv preprint arXiv:2204.01390 (2022). Put Teacher in Student’s Shoes: Cross-Distillation for Ultra-compact Model Compression Framework KDD ’25, August 3–7, 2025, Toronto, ON, Canada

    arXiv 2022

  3. [3]

    Defang Chen, Jian-Ping Mei, Hailin Zhang, Can Wang, Yan Feng, and Chun Chen

  4. [4]

    Defang Chen, Jian-Ping Mei, Yuan Zhang, Can Wang, Zhe Wang, Yan Feng, and Chun Chen. 2021. Cross-layer distillation with semantic calibration. In AAAI, Vol. 35. 7028–7036

    work page 2021

  5. [5]

    Tejalal Choudhary, Vipul Mishra, Anurag Goswami, and Jagannathan Saranga- pani. 2020. A comprehensive survey on model compression and acceleration. Artificial Intelligence Review 53 (2020), 5113–5155

    work page 2020

  6. [6]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding.arXiv preprint arXiv:1810.04805 (2018)

    arXiv 2018

  7. [7]

    Jingtong Gao, Xiangyu Zhao, Bo Chen, Fan Yan, Huifeng Guo, and Ruiming Tang. 2023. AutoTransfer: Instance transfer for cross-domain recommendations. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval . 1478–1487

    work page 2023

  8. [8]

    Yu Gong, Ziwen Jiang, Yufei Feng, Binbin Hu, Kaiqi Zhao, Qingwen Liu, and Wenwu Ou. 2020. EdgeRec: recommender system on edge in Mobile Taobao. In CIKM. 2477–2484

    work page 2020

Show all 58 references
  1. [9]

    Saurabh Goyal, Anamitra Roy Choudhury, Saurabh Raje, Venkatesan Chakar- avarthy, Yogish Sabharwal, and Ashish Verma. 2020. PoWER-BERT: Accelerating BERT inference via progressive word-vector elimination. In ICML. PMLR, 3690– 3699

    2020

  2. [10]

    Weiwei Guo, Huiji Gao, Jun Shi, Bo Long, Liang Zhang, Bee-Chung Chen, and Deepak Agarwal. 2019. Deep natural language processing for search and recom- mender systems. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining . 3199–3200

    2019

  3. [11]

    Manish Gupta and Puneet Agrawal. 2022. Compression of deep learning models for text: A survey. ACM Transactions on Knowledge Discovery from Data (TKDD) 16, 4 (2022), 1–55

    2022

  4. [12]

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015)

    2015 arXiv

  5. [13]

    Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019. What does BERT learn about the structure of language?. In ACL 2019

    2019

  6. [14]

    Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020. TinyBERT: Distilling BERT for Natural Language Understanding. In Findings of EMNLP. 4163–4174

    2020

  7. [15]

    Pegah Kharazmi, Zhewei Zhao, Clement Chung, and Samridhi Choudhary. 2023. Distill-quantize-tune-Leveraging large teachers for low-footprint efficient multi- lingual NLU on edge. In ICASSP. IEEE, 1–5

    2023

  8. [16]

    Diksha Khurana, Aditya Koli, Kiran Khatter, and Sukhdev Singh. 2023. Natural language processing: State of the art, current trends and challenges. Multimedia tools and applications 82, 3 (2023), 3713–3744

    2023

  9. [17]

    Gyuwan Kim and Kyunghyun Cho. 2021. Length-adaptive transformer: Train once with length drop, use anytime with search. InACL-IJCNLP. ACL, 6501–6511

    2021

  10. [18]

    Sehoon Kim, Sheng Shen, David Thorsley, Amir Gholami, Woosuk Kwon, Joseph Hassoun, and Kurt Keutzer. 2022. Learned token pruning for transformers. In KDD. 784–794

    2022

  11. [19]

    Taehyeon Kim, Jaehoon Oh, NakYil Kim, Sangwook Cho, and Se-Young Yun

  12. [20]

    Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. In ICLR

    2019

  13. [21]

    Chen Liang, Simiao Zuo, Qingru Zhang, Pengcheng He, Weizhu Chen, and Tuo Zhao. 2023. Less is more: Task-aware layer-wise distillation for language model compression. In ICML. PMLR, 20852–20867

    2023

  14. [22]

    Weilin Lin, Xiangyu Zhao, Yejing Wang, Yuanshao Zhu, and Wanyu Wang. 2023. Autodenoise: Automatic data instance denoising for recommendations. In Pro- ceedings of the ACM Web Conference 2023 . 1003–1011

    2023

  15. [23]

    Weixin Liu, Xuyi Chen, Jiaxiang Liu, Shikun Feng, Yu Sun, Hao Tian, and Hua Wu. 2023. Ernie 3.0 tiny: Frustratingly simple method to improve task-agnostic distillation generalization. arXiv preprint arXiv:2301.03416 (2023)

    2023 arXiv

  16. [24]

    Zhenhua Liu, Yunhe Wang, Kai Han, Wei Zhang, Siwei Ma, and Wen Gao. 2021. Post-training quantization for vision transformer.Neurips 34 (2021), 28092–28103

    2021

  17. [25]

    Gaurav Menghani. 2023. Efficient deep learning: A survey on making deep learning models smaller, faster, and better. Comput. Surveys 55, 12 (2023), 1–37

    2023

  18. [26]

    Bonan Min, Hayley Ross, Elior Sulem, Amir Pouran Ben Veyseh, Thien Huu Nguyen, Oscar Sainz, Eneko Agirre, Ilana Heintz, and Dan Roth. 2023. Recent advances in natural language processing via large pre-trained language models: A survey. Comput. Surveys 56, 2 (2023), 1–40

    2023

  19. [27]

    Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Mat- sukawa, and Hassan Ghasemzadeh. 2020. Improved knowledge distillation via teacher assistant. In AAAI, Vol. 34. 5191–5198

    2020

  20. [28]

    Yuriy Mishchenko, Yusuf Goren, Ming Sun, Chris Beauchene, Spyros Matsoukas, Oleg Rybakov, and Shiv Naga Prasad Vitaladevuni. 2019. Low-bit quantization and quantization-aware training for small-footprint keyword spotting. In 2019 18th IEEE International Conference On Machine L...

    2019

  21. [29]

    Yury Nahshan, Brian Chmiel, Chaim Baskin, Evgenii Zheltonozhskii, Ron Ban- ner, Alex M Bronstein, and Avi Mendelson. 2021. Loss aware post-training quantization. Machine Learning 110, 11-12 (2021), 3245–3262

    2021

  22. [30]

    Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. 2019. Language models as knowledge bases? arXiv preprint arXiv:1909.01066 (2019)

    2019 arXiv

  23. [31]

    Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al . 2018. Improving language understanding by generative pre-training. (2018)

    2018

  24. [32]

    Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Dis- tilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter.arXiv preprint arXiv:1910.01108 (2019)

    2019 arXiv

  25. [33]

    Yuanming Shi, Kai Yang, Tao Jiang, Jun Zhang, and Khaled B Letaief. 2020. Communication-efficient edge AI: Algorithms and systems.IEEE Communications Surveys & Tutorials 22, 4 (2020), 2167–2191

    2020

  26. [34]

    Fengyi Song, Bo Chen, Xiangyu Zhao, Huifeng Guo, and Ruiming Tang. 2022. Au- toassign: Automatic shared embedding assignment in streaming recommendation. In 2022 IEEE International Conference on Data Mining (ICDM) . IEEE, 458–467

    2022

  27. [35]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Neurips 30 (2017)

    2017

  28. [36]

    Chenguang Wang, Mu Li, and Alexander J Smola. 2019. Language models with transformers. arXiv preprint arXiv:1904.09408 (2019)

    2019 arXiv

  29. [37]

    Hanrui Wang, Zhekai Zhang, and Song Han. 2021. Spatten: Efficient sparse atten- tion architecture with cascade token and head pruning. In2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA) . IEEE, 97–110

    2021

  30. [38]

    Maolin Wang, Yu Pan, Xiangli Yang, Guangxi Li, and Zenglin Xu. 2023. Tensor networks meet neural networks: A survey.arXiv preprint arXiv:2302.09019 (2023)

    2023 arXiv

  31. [39]

    Maolin Wang, Yao Zhao, Jiajia Liu, Jingdong Chen, Chenyi Zhuang, Jinjie Gu, Ruocheng Guo, and Xiangyu Zhao. 2023. Large multimodal model compression via efficient pruning and distillation at AntGroup. arXiv preprint arXiv:2312.05795 (2023)

    2023 arXiv

  32. [40]

    Maolin Wang, Yao Zhao, Jiajia Liu, Jingdong Chen, Chenyi Zhuang, Jinjie Gu, Ruocheng Guo, and Xiangyu Zhao. 2024. Large multimodal model compression via iterative efficient pruning and distillation. In Companion Proceedings of the ACM Web Conference 2024. 235–244

    2024

  33. [41]

    Naigang Wang, Chi-Chun Charlie Liu, Swagath Venkataramani, Sanchari Sen, Chia-Yu Chen, Kaoutar El Maghraoui, Vijayalakshmi Viji Srinivasan, and Leland Chang. 2022. Deep Compression of Pre-trained Transformer Models. Neurips 35 (2022), 14140–14154

    2022

  34. [42]

    Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou

  35. [43]

    Xiaojie Wang, Jiameng Li, Zhaolong Ning, Qingyang Song, Lei Guo, Song Guo, and Mohammad S Obaidat. 2023. Wireless powered mobile edge computing networks: A survey. Comput. Surveys (2023)

    2023

  36. [44]

    Yejing Wang, Xiangyu Zhao, Tong Xu, and Xian Wu. 2022. Autofield: Automating feature selection in deep recommender systems. In Proceedings of the ACM Web Conference 2022. 1977–1986

    2022

  37. [45]

    Chuhan Wu, Fangzhao Wu, Tao Qi, Yongfeng Huang, and Xing Xie. 2022. Noisy- tune: A little noise can help you finetune pretrained language models better. arXiv preprint arXiv:2202.12024 (2022)

    2022 arXiv

  38. [46]

    Canwen Xu and Julian McAuley. 2023. A survey on model compression and acceleration for pretrained language models. In AAAI, Vol. 37. 10566–10575

    2023

  39. [47]

    Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, et al. 2020. CLUE: A Chinese Language Understanding Evaluation Benchmark. In Proceedings of the 28th International Conference on Computational Linguistics. 4762–4772

    2020

  40. [48]

    Yi Yang, Chen Zhang, and Dawei Song. 2022. Sparse teachers can be dense with knowledge. arXiv preprint arXiv:2210.03923 (2022)

    2022 arXiv

  41. [49]

    Xin Yao, Ziqing Yang, Yiming Cui, and Shijin Wang. 2023. MiniRBT: A Two- stage Distilled Small Chinese Pre-trained Model. arXiv preprint arXiv:2304.00717 (2023)

    2023 arXiv

  42. [50]

    Chen Zhang, Yang Yang, Jiahao Liu, Jingang Wang, Yunsen Xian, Benyou Wang, and Dawei Song. 2023. Lifting the curse of capacity gap in distilling language models. arXiv preprint arXiv:2305.12129 (2023)

    2023 arXiv

  43. [51]

    Chen Zhang, Yang Yang, Qifan Wang, Jiahao Liu, Jingang Wang, Wei Wu, and Dawei Song. 2022. Minidisc: Minimal distillation schedule for language model compression. arXiv preprint arXiv:2205.14570 (2022)

    2022 arXiv

  44. [52]

    Xiangyu Zhao, Haochen Liu, Hui Liu, Jiliang Tang, Weiwei Guo, Jun Shi, Sida Wang, Huiji Gao, and Bo Long. 2021. Autodim: Field-aware embedding dimension searchin recommender systems. In Proceedings of the Web Conference 2021 . 3015– 3022

    2021

  45. [53]

    Xiangyu Zhaok, Haochen Liu, Wenqi Fan, Hui Liu, Jiliang Tang, Chong Wang, Ming Chen, Xudong Zheng, Xiaobing Liu, and Xiwang Yang. 2021. Autoemb: Automated embedding dimensionality search in streaming recommendations. In 2021 IEEE International Conference on Data Mining (ICDM) ...

    2021

  46. [54]

    Wangchunshu Zhou, Canwen Xu, and Julian McAuley. 2021. BERT learns to teach: Knowledge distillation with meta learning. arXiv preprint arXiv:2106.04570 (2021)

    2021 arXiv

  47. [55]

    You Zhou, Xiujing Lin, Xiang Zhang, Maolin Wang, Gangwei Jiang, Huakang Lu, Yupeng Wu, Kai Zhang, Zhe Yang, Kehang Wang, et al. 2023. On the opportunities of green computing: A survey. arXiv preprint arXiv:2311.00447 (2023). A Details of Alipay Datasets Alipay Datasets encompa...

    2023 arXiv

  48. [2020]

    Neurips 33 (2020), 5776–5788

    Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. Neurips 33 (2020), 5776–5788

    2020

  49. [2021]

    arXiv preprint arXiv:2105.08919 (2021)

    Comparing kullback-leibler divergence and mean squared error loss in knowledge distillation. arXiv preprint arXiv:2105.08919 (2021)

    2021 arXiv

  50. [2022]

    In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Knowledge distillation with the reused teacher classifier. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 11933–11942

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.