Pith. sign in

REVIEW 4 major objections 5 minor 8 cited by

AMMKD claims that a lightweight student distilled from two frozen CLIP teachers, with per-batch adaptive teacher weighting, can outperform the large CLIP teacher and all 11 distillation baselines it was tested against on seven image-classif

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

AMMKD claims large gains from adaptively weighted two-teacher CLIP distillation, but its equations are internally inconsistent, its baselines are unverifiable, and its tests do not match its stated retrieval goal.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection A distillation recipe with strong reported numbers but the adaptive-weighting mechanism that is the paper's stated novelty is not actually written down as an objective, and the ablations contain internal inconsistencies. the 4 major comments →

arxiv 2509.00039 v1 pith:CUPYEUJQ submitted 2025-08-23 cs.CV

AMMKD: Adaptive Multimodal Multi-teacher Distillation for Lightweight Vision-Language Models

Yuqi Li , Chuanguang Yang , Junhao Dong , Zhengtao Yao , Haoyan Xu , Zeyu Dong , Hansheng Zeng , Zhulin An
show 1 more author
Yingli Tian
This is my paper
classification cs.CV
keywords knowledge distillationmulti-teacher distillationvision-language modelsCLIPadaptive weightingmulti-objective optimizationlightweight modelsimage-text retrieval
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to show that a small vision-language model can match or beat much larger CLIP models by learning from two frozen CLIP teachers at once, using a method called AMMKD. Its central move is to treat multi-teacher distillation as a multi-objective optimization problem and to re-weight each teacher per batch according to how similar the teacher's image feature is to the label text feature, with a multi-gradient descent algorithm meant to reduce conflicting gradients. The student is trained with a combination of KL-divergence soft-label matching, mean-squared-error feature alignment, and the standard CLIP contrastive loss. If the reported results hold, the framework would make CLIP-scale image-text retrieval feasible on mobile devices without the accuracy cost usually paid for smaller encoders.

Core claim

AMMKD claims that decoupling modalities—pre-computing and storing text class vectors with the teacher text encoder—and distilling from two heterogeneous frozen CLIP teachers into a lightweight student yields superior accuracy with far fewer parameters. The student optimizes L = Lkl + Lclip + Lmse, where Lkl aligns teacher and student image-to-text and text-to-image distributions via KL divergence, Lmse aligns normalized image/text feature vectors, and Lclip is the contrastive loss. An adaptive dynamic weighting scheme sets weights proportional to each teacher's cosine similarity between image features and label text and recasts the weighting as an MGDA multi-objective optimization over gradi

What carries the argument

The load-bearing mechanism is the Multi-Teacher Dynamic Weight Selection (DWS): per-batch weights lambda1 = r1/(r1+r2) and lambda2 = r2/(r1+r2) computed from cosine similarities between each frozen teacher's image feature and the label text feature, combined with an MGDA-style minimization over the two teachers' gradient directions. MGDA—multi-gradient descent—is a standard way to seek a common descent direction in multi-objective optimization, and the paper's stated intent is that this weighting reduces interference from noisy or weak teachers and steers the student toward a Pareto-stationary learning direction. The same loss also combines KL divergence on the two contrastive distributions

Load-bearing premise

The central claim depends on the assumption that the accuracy gains in the ablations come from the adaptive weighting rule (cosine-similarity ratios and the MGDA objective) rather than from supervised fine-tuning, data augmentation, or the MSE loss—a link the text asserts but never demonstrates, since no equation trains a weighted sum over the two teachers.

What would settle it

Run the identical training pipeline on UCF-101 and Flower102 with the dynamic weights replaced by fixed equal weights and, separately, with weights set opposite to the similarity ratios, holding all other components fixed. If the reported 87.93% UCF-101 and 73.63% Flower102 results do not drop, the adaptive-weighting claim is falsified. A second check: train with DWS but without the MSE loss, and with the MSE loss but without DWS, to see which term actually carries the gain.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the gains are real, a phone-sized CLIP student can exceed the large teacher's accuracy on several classification tasks, making on-device image-text retrieval and zero-shot classification plausible.
  • Two diverse frozen teachers are enough: adding a third or fourth teacher lowers accuracy on most datasets, implying the value depends on teacher complementarity, not teacher count.
  • The balanced 1:1:1 loss ratio is critical; shrinking any component (KL, CLIP, or MSE) degrades performance, most sharply on video data.
  • Pre-computing text class vectors decouples text encoding from training, cutting repeated computation during distillation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A testable extension: replace the cosine-similarity weights with random or equal weights and re-run the Table 4 ablations; if accuracy holds, the DWS mechanism is not the active ingredient and the paper's central contribution lies elsewhere.
  • The paper leaves the MGDA formulation disconnected from the losses actually trained; showing that the student's final objective is a convex combination of the two teacher losses with the MGDA-derived alphas would close that gap.
  • Because the evaluation uses image-classification benchmarks rather than retrieval tasks, the stated image-text retrieval motivation remains indirect; testing on zero-shot retrieval or ranking datasets would check whether the distilled alignment transfers beyond classification.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes AMMKD, a knowledge-distillation framework that combines a multimodal feature-fusion student, two frozen CLIP teachers, KL and MSE distillation losses, and an adaptive multi-teacher weighting scheme (DWS) allegedly derived from multi-objective optimization. Experiments on seven image-classification datasets report large gains over several KD baselines, with ablations claiming that the dynamic weighting is responsible for the improvement.

Significance. If the method were implemented as described, AMMKD would be a practically interesting contribution to lightweight vision-language distillation: it addresses a real deployment problem, reports extensive comparisons on standard benchmarks, and includes ablations over loss proportions, teacher count, and student size. However, the central contribution is not a well-defined algorithm in the manuscript: the two-teacher training objective is never written, the dynamic weights are attached to the wrong objects, and the MGDA block is never connected to training. In addition, key tables are internally inconsistent. As written, the paper does not support its headline claims.

major comments (4)
  1. [Methodology, Eqs. (5)–(8)] The multi-teacher distillation loss is never defined. Eqs. (5) and (6) are identical definitions of pS(P|Q), while the teacher text-to-image distribution pT_k(Q|P) used in Eq. (8) is never defined. Eqs. (7)–(8) define Lkl for 'the k teacher', but the overall loss in Eq. (11) contains a single Lkl, Lclip, and Lmse and has no sum over k. Thus the two-teacher objective that the method claims to optimize is absent.
  2. [Methodology, Eqs. (12)–(15)] The 'adaptive dynamic weighting' is not part of the specified objective. Eq. (12) weights λ1 and λ2 on Lkl(P|Q) and Lkl(Q|P), i.e. on the two directions of KL matching, not on the two teachers. Eqs. (13)–(14) define λ1 and λ2 from per-teacher image-to-text similarities, but these are not teacher weights in Eq. (12). The MGDA α obtained from Eq. (15) never appears in any loss, gradient update, or algorithm step, and the g(·) functions in Eq. (15) are themselves undefined. Consequently, the DWS gains in Table 4 cannot be attributed to any mechanism specified in the paper; they could plausibly come from Lclip, Lmse, or data augmentation.
  3. [Experiments, Tables 2 and 4] The reported numbers are internally inconsistent. Table 4 lists Base as 59.45/61.78/76.93 on ImageNet-tiny/Flower102/UCF-101, but Table 2's Base rows are 60.46/76.03, 60.83/78.42, and 61.78/61.78; none of the Table 2 rows matches the UCF-101 value 76.93, and ImageNet-tiny is absent from Table 2. This makes the DWS ablation and the claimed gains non-reproducible from the text. Since the central claim is that DWS drives the improvement, this inconsistency is load-bearing.
  4. [Multi-Teacher Dynamic Weight Selection, Eq. (15)] The MGDA formulation is presented as an optimization-based justification, but it is not derived from any training objective. The text states that KKT is 'assumed' to hold and then asserts that a Pareto stationary point is the 'global best' for all teachers; no proof or reference is supplied. Moreover, the gradient objects in Eq. (15) are cross-entropies of pS and pT, whose definitions are not given in the same form elsewhere. As written, this section does not establish that the proposed weighting scheme has any optimality property.
minor comments (5)
  1. [Throughout] The method is called multimodal but the experiments are image-classification benchmarks; UCF-101 is a video dataset, not an image-text retrieval benchmark. This mismatch should be clarified.
  2. [Methodology, Eq. (9)] Lclip is written as CE(y, cos(ui, wi)), mixing a label-based cross-entropy with a cosine similarity. The exact contrastive formulation and the role of the batch are unclear.
  3. [Experiments, Table 4] Table 4 uses the row label 'DSW' while the text and Table 2 use 'DWS'; the naming should be consistent.
  4. [Multi-Teacher Dynamic Weight Selection] 'Kuarush-Huhn-Tucker' is a misspelling of Karush-Kuhn-Tucker.
  5. [Experiments, Table 2] The row labels 'LRCosine' and 'LRFixed' are used without definition; the text should state what these represent.

Circularity Check

1 steps flagged

The paper's central 'adaptive multi-teacher weighting' is not actually present in the written loss: Eq. (12)'s λ1/λ2 weight two KL directions, not the two teachers, and the MGDA α is never used, so the claimed DWS mechanism reduces to a predefined rescaling by teacher-similarity inputs.

specific steps
  1. self definitional [Methodology, Eqs. (11)-(14) and 'Multi-Teacher Dynamic Weight Selection']
    "Lkl = λ1Lkl(P |Q) +λ2Lkl(Q|P ), (12) ... First, the similarity between each teacher image and the current label text is taken as a dynamic weight, denoted as r1 and r2, ... λ1 = r1/(r1+r2), (13) λ2 = r2/(r1+r2). (14)"

    Eqs. (7)-(8) define Lkl(P|Q) and Lkl(Q|P) for 'the k teacher', but Eq. (11) contains a single Lkl, Lclip, and Lmse with no sum over k. Eq. (12) then attaches λ1 and λ2 to the two directional KL terms, not to the two teacher models. The text immediately interprets these same λs as teacher weights ('balance the influence of teacher loss', 'all teacher losses are considered equally'), and Eqs. (13)-(14) compute them from per-teacher image/label-text similarities r1 and r2. Thus the claimed multi-teacher dynamic weighting is, by the paper's own equations, a weighting of image-to-text vs text-to-image directions by quantities taken directly from teacher activations; it is not a teacher-weighted objective. No equation connects the MGDA α of Eq. (15) to this loss, so the 'optimal' weighting is ne

full rationale

The paper is not circular in the usual self-citation sense: the CLIP-KD citation is contextual, and no external uniqueness theorem is imported from the authors. However, the central contribution — adaptive dynamic weighting of two teachers — collapses in the written equations. Eq. (11) defines the entire student loss as Lkl + Lclip + Lmse with no teacher index; Eq. (12) weights the two directional KL terms with λ1, λ2; and Eqs. (13)-(14) set those λs to normalized teacher-image/label-text similarities. The MGDA optimization of Eq. (15) is introduced as the justification for adaptive weighting but is never substituted into the loss or backward pass, so the 'Pareto stationary' argument has no effect on the trained objective. Consequently, the reported DWS gains in Table 4 cannot be attributed to the described multi-teacher weighting mechanism as written; if the implementation sums over teachers, that sum is absent from the manuscript. I also note Table 4's Base row (59.45/61.78/76.93) does not match Table 2's Base rows, which is a reproducibility concern rather than circularity. Because the paper's headline method reduces, by its own equations, to a direction-weighting scheme with weights taken from the teacher inputs, the central derivation chain is substantially circular/definitional rather than independently derived.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The method's derivation rests on assumptions that are either asserted (KKT and Pareto-stationarity for a poorly defined gradient-combination problem) or unexamined (teachers weaker than the student still provide useful guidance). Free parameters (loss ratios, temperature, teacher pair, student architecture, augmentation policy) are tuned on the same test sets used for the headline numbers. No invented entities (particles, mediators, new dimensions) are introduced.

free parameters (5)
  • Loss proportions (Lclip:Lkl:Lmse) = 1:1:1
    Table 3 sweeps 0.5/1.0 ratios on the same three datasets used for the headline results; 1:1:1 is selected as best on those test sets.
  • Temperature tau (tau1, tau2, tau3) = 4
    Set to 4 by hand in the training details; no sweep or justification is given.
  • Teacher pair selection = OpenAI-CLIP + DFN2B-CLIP
    Table 5 shows the two-teacher combination is best; the teacher set is selected by performance on the reported test sets.
  • Student architecture (6-layer, dim 512) = 6 layers, 512-dim
    Table 6 sweeps layer count and feature dimension on the reported test sets; 6-layer+512 is chosen as best.
  • Data augmentation policy = flip/rotate/Cutout/Mixup, dataset-dependent
    Table 2 shows augmentation hurting the base UCF-101 setting (78.42 to 61.78) while helping the full method (83.56 to 87.93); the final policy is chosen on test performance.
axioms (4)
  • ad hoc to paper KKT conditions hold for the gradient-combination objective so a Pareto stationary point exists (Eq (15): 'it is assumed... KKT holds').
    Assumed without defining the optimization problem over student parameters; the gradients are of distillation losses w.r.t. outputs, not a well-posed constrained program.
  • ad hoc to paper A Pareto stationary point of the gradient combination is the global best for all teachers ('in the shared layer, there is no feasible direction of decline; that is, it belongs to the global best').
    This asserts global optimality from stationarity, which is false for general nonconvex networks and is never justified for this distillation setup.
  • domain assumption Zero-shot teachers weaker than the trained student still provide beneficial guidance.
    Table 2: teachers score 69.96/66.38 (CLIPlarge) and 69.73/64.38 (DFN) on Flower102/UCF-101 while the base student already reaches 60.46/76.03; the benefit of distilling from weaker teachers is unexamined.
  • standard math The standard CLIP contrastive probability formulation of Eqs (1)-(2) is the correct operating point for the teacher and student models.
    Standard CLIP zero-shot classification setup, accepted background from Radford et al.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of AMMKD: Adaptive Multimodal Multi-teacher Distillation for Lightweight Vision-Language Models." pith.science (2026). https://pith.science/paper/CUPYEUJQ

@misc{pith2026250900039,
  author       = {Pith},
  title        = {Pith review of: AMMKD: Adaptive Multimodal Multi-teacher Distillation for Lightweight Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CUPYEUJQ}},
  note         = {Machine review of arXiv:2509.00039}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

The success of large-scale visual language pretraining (VLP) models has driven widespread adoption of image-text retrieval tasks. However, their deployment on mobile devices remains limited due to large model sizes and computational complexity. We propose Adaptive Multi-Modal Multi-Teacher Knowledge Distillation (AMMKD), a novel framework that integrates multi-modal feature fusion, multi-teacher distillation, and adaptive optimization to deliver lightweight yet effective retrieval models. Specifically, our method begins with a feature fusion network that extracts and merges discriminative features from both the image and text modalities. To reduce model parameters and further improve performance, we design a multi-teacher knowledge distillation framework to pre-train two CLIP teacher models. We decouple modalities by pre-computing and storing text features as class vectors via the teacher text encoder to enhance efficiency. To better align teacher and student outputs, we apply KL scatter for probability distribution matching. Finally, we design an adaptive dynamic weighting scheme that treats multi-teacher distillation as a multi-objective optimization problem. By leveraging gradient space diversity, we dynamically adjust the influence of each teacher, reducing conflicts and guiding the student toward more optimal learning directions. Extensive experiments on three benchmark datasets demonstrate that AMMKD achieves superior performance while significantly reducing model complexity, validating its effectiveness and flexibility.

Figures

Figures reproduced from arXiv: 2509.00039 by Chuanguang Yang, Hansheng Zeng, Haoyan Xu, Junhao Dong, Yingli Tian, Yuqi Li, Zeyu Dong, Zhengtao Yao, Zhulin An.

Figure 1
Figure 1. Figure 1: Migration performance of 12 knowledge distilla [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Framework of the Adaptive Multi-Teacher Multi-Modal Knowledge Distillation (AMMKD) method. It contains [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Cornerstones or Stumbling Blocks? Deciphering the Rock Tokens in On-Policy Distillation

    cs.CL 2026-05 unverdicted novelty 7.0

    Persistent 'Rock Tokens' in on-policy distillation resist teacher corrections, consume large gradient norms, yet add negligible value to reasoning, allowing targeted bypassing to streamline alignment.

  2. Dual Distribution Estimation for Zero-shot Noisy Test-Time Adaptation with VLMs

    cs.CV 2026-06 unverdicted novelty 6.0

    DDE models class-wise positive feature Gaussians and negative label distributions to boost ID accuracy and OOD detection in zero-shot noisy TTA, reporting 3.70% harmonic mean gain and 6.20% FPR95 drop on ImageNet.

  3. Curvature-Adaptive Consistency Flow Matching: Autonomous Trajectory Optimization via Reinforcement Learning

    cs.CV 2026-06 unverdicted novelty 6.0

    CACFM applies RL to adaptively select critical regions in probability flow ODE trajectories for consistency distillation, yielding SOTA few-step results on FLUX and SDXL.

  4. Bridging Reasoning Trajectories in On-Policy Distillation via Near-Future Guidance

    cs.CL 2026-05 unverdicted novelty 6.0

    TOPD augments on-policy distillation by using near-future trajectory signals to suppress non-divergent high-loss tokens and distribute guidance, raising average accuracy from 47.8% to 52.2% on reasoning benchmarks.

  5. Bridging Reasoning Trajectories in On-Policy Distillation via Near-Future Guidance

    cs.CL 2026-05 unverdicted novelty 6.0

    TOPD improves on-policy distillation for LLM reasoning by using near-future guidance to identify divergent states, raising average accuracy from 47.8% to 52.2% on math benchmarks including AIME24 and AIME25.

  6. Cornerstones or Stumbling Blocks? Deciphering the Rock Tokens in On-Policy Distillation

    cs.CL 2026-05 unverdicted novelty 6.0

    Rock Tokens in on-policy distillation persist at high loss, account for up to 18% of outputs, absorb large gradient norms, but add negligible value to reasoning performance.

  7. Curvature-Adaptive Consistency Flow Matching: Autonomous Trajectory Optimization via Reinforcement Learning

    cs.CV 2026-06 conditional novelty 5.0

    A reinforcement-learning agent that picks high-error trajectory segments during consistency distillation improves few-step text-to-image generation on FLUX and SDXL.

  8. Cornerstones or Stumbling Blocks? Deciphering the Rock Tokens in On-Policy Distillation

    cs.CL 2026-05 unverdicted novelty 5.0

    Empirical study of on-policy distillation finds 'Rock Tokens' that resist correction, account for up to 18% of output tokens, and add little to model reasoning despite high optimization cost.

Reference graph

Works this paper leans on

51 extracted references · 36 canonical work pages · cited by 4 Pith papers

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Bansal, M.; Kumar, M.; Sachdeva, M.; and Mittal, A. 2023. Transfer learning for image classification using VGG19: Caltech-101 image data set. Journal of ambient intelligence and humanized computing, 14(4): 3609--3620

    work page 2023

  4. [4]

    Chen, X.; He, B.; Hui, K.; Sun, L.; and Sun, Y. 2021. Simplified tinybert: Knowledge distillation for document retrieval. In Advances in Information Retrieval: 43rd European Conference on IR Research, ECIR 2021, Virtual Event, March 28--April 1, 2021, Proceedings, Part II 43, 241--248. Springer

    work page 2021

  5. [5]

    Choi, K.; Kersner, M.; Morton, J.; and Chang, B. 2022. Temporal knowledge distillation for on-device audio classification. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 486--490. IEEE

    work page 2022

  6. [6]

    Cimpoi, M.; Maji, S.; Kokkinos, I.; Mohamed, S.; and Vedaldi, A. 2014. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, 3606--3613

    work page 2014

  7. [7]

    Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; and Fei-Fei, L. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, 248--255. Ieee

    2009

  8. [8]

    DeVries, T.; and Taylor, G. W. 2017. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552

    Pith/arXiv arXiv 2017

  9. [9]

    Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929

    Pith/arXiv arXiv 2020

  10. [10]

    Du, S.; You, S.; Li, X.; Wu, J.; Wang, F.; Qian, C.; and Zhang, C. 2020. Agree to disagree: Adaptive ensemble knowledge distillation in gradient space. advances in neural information processing systems, 33: 12345--12355

    work page 2020

  11. [11]

    Duman, B.; and S \"u zen, A. A. 2022. A study on deep learning based classification of flower images. International Journal of Advanced Networking and Applications, 14(2): 5385--5389

    work page 2022

  12. [12]

    M.; Jain, A.; Schmidt, L.; Toshev, A.; and Shankar, V

    Fang, A.; Jose, A. M.; Jain, A.; Schmidt, L.; Toshev, A.; and Shankar, V. 2023. Data filtering networks. arXiv preprint arXiv:2309.17425

    Pith/arXiv arXiv 2023

  13. [13]

    Fang, Z.; Wang, J.; Hu, X.; Wang, L.; Yang, Y.; and Liu, Z. 2021. Compressing visual-linguistic model via knowledge distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 1428--1438

    work page 2021

  14. [14]

    Fukuda, T.; Suzuki, M.; Kurata, G.; Thomas, S.; Cui, J.; and Ramabhadran, B. 2017. Efficient Knowledge Distillation from an Ensemble of Teachers. In Interspeech, 3697--3701

    work page 2017

  15. [15]

    He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 770--778

    2016

  16. [16]

    Helber, P.; Bischke, B.; Dengel, A.; and Borth, D. 2019. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7): 2217--2226

    work page 2019

  17. [17]

    Hinton, G. 2015. Distilling the Knowledge in a Neural Network. arXiv preprint arXiv:1503.02531

    Pith/arXiv arXiv 2015

  18. [18]

    Jang, J.; Ma, C.; and Lee, B. 2025. VL2Lite: Task-Specific Knowledge Distillation from Large Vision-Language Models to Lightweight Networks. In Proceedings of the Computer Vision and Pattern Recognition Conference, 30073--30083

    work page 2025

  19. [19]

    Jiao, X.; Yin, Y.; Shang, L.; Jiang, X.; Chen, X.; Li, L.; Wang, F.; and Liu, Q. 2019. Tinybert: Distilling bert for natural language understanding. arXiv preprint arXiv:1909.10351

    Pith/arXiv arXiv 2019

  20. [20]

    B.; Jang, H.; and Hwang, S

    Kang, S.; Lee, D. B.; Jang, H.; and Hwang, S. J. 2025. Simple Semi-supervised Knowledge Distillation from Vision-Language Models via Dual-Head 0ptimization. arXiv preprint arXiv:2505.07675

    work page arXiv 2025

  21. [21]

    Kwon, K.; Na, H.; Lee, H.; and Kim, N. S. 2020. Adaptive knowledge distillation based on entropy. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 7409--7413. IEEE

    work page 2020

  22. [22]

    Lee, J.; Das, D.; Hayat, M.; Choi, S.; Hwang, K.; and Porikli, F. 2025. Customkd: Customizing large vision foundation for edge model improvement via knowledge distillation. In Proceedings of the Computer Vision and Pattern Recognition Conference, 25176--25186

    work page 2025

  23. [23]

    Li, Z.; Li, X.; Yang, L.; Zhao, B.; Song, R.; Luo, L.; Li, J.; and Yang, J. 2023. Curriculum temperature for knowledge distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, 1504--1512

    work page 2023

  24. [24]

    Li, Z.; Ye, J.; Song, M.; Huang, Y.; and Pan, Z. 2021. Online knowledge distillation for efficient pose estimation. In Proceedings of the IEEE/CVF international conference on computer vision, 11740--11750

    work page 2021

  25. [25]

    Liang, C.; Yu, J.; Yang, M.-H.; Brown, M.; Cui, Y.; Zhao, T.; Gong, B.; and Zhou, T. 2024. Module-wise adaptive distillation for multimodality foundation models. Advances in Neural Information Processing Systems, 36

    work page 2024

  26. [26]

    Liu, Y.; Zhang, W.; and Wang, J. 2020. Adaptive multi-teacher multi-level knowledge distillation. Neurocomputing, 415: 106--113

    work page 2020

  27. [27]

    Park, W.; Kim, D.; Lu, Y.; and Cho, M. 2019. Relational knowledge distillation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 3967--3976

    work page 2019

  28. [28]

    M.; Vedaldi, A.; Zisserman, A.; and Jawahar, C

    Parkhi, O. M.; Vedaldi, A.; Zisserman, A.; and Jawahar, C. 2012. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, 3498--3505. IEEE

    work page 2012

  29. [29]

    Paszke, A.; Gross, S.; Chintala, S.; Chanan, G.; Yang, E.; DeVito, Z.; Lin, Z.; Desmaison, A.; Antiga, L.; and Lerer, A. 2017. Automatic differentiation in pytorch

    work page 2017

  30. [30]

    W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al

    Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, 8748--8763. PMLR

    2021

  31. [31]

    E.; Chassang, A.; Gatta, C.; and Bengio, Y

    Romero, A.; Ballas, N.; Kahou, S. E.; Chassang, A.; Gatta, C.; and Bengio, Y. 2014. Fitnets: Hints for thin deep nets. arXiv 2014. arXiv preprint arXiv:1412.6550

    Pith/arXiv arXiv 2014

  32. [32]

    Schmid, F.; Koutini, K.; and Widmer, G. 2023. Efficient large-scale audio tagging via transformer-to-cnn knowledge distillation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1--5. IEEE

    work page 2023

  33. [33]

    R.; and Shah, M

    Soomro, K.; Zamir, A. R.; and Shah, M. 2012. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402

    Pith/arXiv arXiv 2012

  34. [34]

    Sun, S.; Ren, W.; Li, J.; Wang, R.; and Cao, X. 2024. Logit standardization in knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 15731--15740

    work page 2024

  35. [35]

    Tung, F.; and Mori, G. 2019. Similarity-preserving knowledge distillation. In Proceedings of the IEEE/CVF international conference on computer vision, 1365--1374

    work page 2019

  36. [36]

    Wang, C.; Yang, Q.; Huang, R.; Song, S.; and Huang, G. 2022 a . Efficient knowledge distillation from model checkpoints. Advances in Neural Information Processing Systems, 35: 607--619

    work page 2022

  37. [37]

    Wang, Y.; Xu, C.; Xu, C.; and Tao, D. 2018. Adversarial learning of portable student networks. In Proceedings of the AAAI conference on artificial intelligence, volume 32

    work page 2018

  38. [38]

    Wang, Z.; Codella, N.; Chen, Y.-C.; Zhou, L.; Dai, X.; Xiao, B.; Yang, J.; You, H.; Chang, K.-W.; Chang, S.-f.; et al. 2022 b . Multimodal adaptive distillation for leveraging unimodal encoders for vision-language tasks. arXiv preprint arXiv:2204.10496

    Pith/arXiv arXiv 2022

  39. [39]

    S.; Wang, X.; et al

    Wu, K.; Peng, H.; Zhou, Z.; Xiao, B.; Liu, M.; Yuan, L.; Xuan, H.; Valenzuela, M.; Chen, X. S.; Wang, X.; et al. 2023. Tinyclip: Clip distillation via affinity mimicking and weight inheritance. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 21970--21980

    work page 2023

  40. [40]

    Wu, M.-C.; Chiu, C.-T.; and Wu, K.-H. 2019. Multi-teacher knowledge distillation for compressed video action recognition on deep neural networks. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2202--2206. IEEE

    work page 2019

  41. [41]

    Xu, K.; Rui, L.; Li, Y.; and Gu, L. 2020. Feature normalized knowledge distillation for image classification. In European conference on computer vision, 664--680. Springer

    work page 2020

  42. [42]

    Xu, X.; Li, M.; Tao, C.; Shen, T.; Cheng, R.; Li, J.; Xu, C.; Tao, D.; and Zhou, T. 2024. A survey on knowledge distillation of large language models. arXiv preprint arXiv:2402.13116

    Pith/arXiv arXiv 2024

  43. [43]

    Yang, C.; An, Z.; Huang, L.; Bi, J.; Yu, X.; et al. 2024. CLIP-KD: An Empirical Study of CLIP Model Distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    work page 2024

  44. [44]

    Yang, C.; An, Z.; Zhou, H.; Cai, L.; Zhi, X.; Wu, J.; Xu, Y.; and Zhang, Q. 2022. Mixskd: Self-knowledge distillation from mixup for image recognition. In European Conference on Computer Vision, 534--551. Springer

    work page 2022

  45. [45]

    You, S.; Xu, C.; Xu, C.; and Tao, D. 2017. Learning from multiple teacher networks. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, 1285--1294

    work page 2017

  46. [46]

    Yuan, F.; Shou, L.; Pei, J.; Lin, W.; Gong, M.; Fu, Y.; and Jiang, D. 2021. Reinforced multi-teacher selection for knowledge distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, 14284--14291

    work page 2021

  47. [47]

    Zhang, H.; Chen, D.; and Wang, C. 2022. Confidence-aware multi-teacher knowledge distillation. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 4498--4502. IEEE

    work page 2022

  48. [48]

    Zhang, H.; Chen, D.; and Wang, C. 2023. Adaptive multi-teacher knowledge distillation with meta-learning. In 2023 IEEE International Conference on Multimedia and Expo (ICME), 1943--1948. IEEE

    work page 2023

  49. [49]

    N.; and Lopez-Paz, D

    Zhang, H.; Cisse, M.; Dauphin, Y. N.; and Lopez-Paz, D. 2017. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412

    Pith/arXiv arXiv 2017

  50. [50]

    Zhao, B.; Cui, Q.; Song, R.; Qiu, Y.; and Liang, J. 2022. Decoupled knowledge distillation. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, 11953--11962

    work page 2022

  51. [51]

    Zheng, D.; Li, J.; Yang, Y.; Wang, Y.; and Pang, P. C.-I. 2024. MicroBERT: Distilling MoE-Based Knowledge from BERT into a Lighter Model. Applied Sciences, 14(14): 6171

    work page 2024

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.