Pith. sign in

REVIEW 5 major objections 5 minor 57 references

A weak, nearly invisible trigger plus fine-tuning at one hundredth of the normal learning rate embeds a backdoor that transfers to students across response-, feature-, and relation-based distillation.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 22:05 UTC pith:3TNW3WJC

load-bearing objection New weak-trigger attack on KD is a real empirical result, but single-seed runs and a simplified SCAR baseline mean 'reliable transfer' is provisional. the 5 major comments →

arxiv 2511.12046 v2 pith:3TNW3WJC submitted 2025-11-15 cs.CR cs.AIcs.CVcs.LG

BackWeak: Backdooring Knowledge Distillation Simply with Weak Triggers and Fine-tuning

Shanmin Wang , Dongdong Zhao This is my paper
classification cs.CR cs.AIcs.CVcs.LG MSC 68T0768M25
keywords knowledge distillationbackdoor attackweak triggerfine-tuningtransferabilitymodel supply chainuniversal adversarial perturbationstealthiness
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

BackWeak claims that backdooring knowledge distillation does not require surrogate student models, simulated distillation, or strong adversarial triggers. The paper shows that fine-tuning a benign teacher on poisoned data with a trigger whose adversarial effect is negligible, using a learning rate two orders of magnitude below training, embeds a backdoor that transfers to students across response-, feature-, and relation-based distillation. If correct, this collapses the cost of the attack and widens the practical threat from pre-trained model repositories. The paper also argues that earlier methods' triggers were effectively universal adversarial perturbations, and that their apparent success came from the trigger's adversarial strength rather than from a genuinely implanted backdoor.

Core claim

The central discovery is that a weak trigger—defined as a perturbation that changes a benign model's target-class prediction probability by at most a small tolerance—can be turned into a transferable backdoor. The recipe is: train a benign teacher normally; generate the trigger by constrained optimization that pushes target-class logits only while its empirical attack success rate stays below a budget; poison 30% of the attacker's data with randomly augmented versions of that trigger; and fine-tune with the classification layer frozen at a learning rate of 0.01 times the original training rate. The resulting teacher keeps benign accuracy, and students distilled with response-, feature-, or r

What carries the argument

The mechanism that carries the argument is the combination of three components: weak-trigger generation, dynamic poisoning, and low-learning-rate fine-tuning. Weak-trigger generation is a constrained optimization with a push loss for target-class association, a margin loss that penalizes confident misclassifications, and an ASR budget that keeps the trigger non-adversarial. Dynamic poisoning applies a random augmentation to the trigger before each use, preventing overfitting to a fixed pixel pattern. Low-learning-rate fine-tuning, with the classifier head frozen, forces gradual feature-space integration of the trigger–target association. The central identity is the indicator-gated optimizati

Load-bearing premise

The load-bearing premise is that the narrow fine-tuning regime—learning rate 0.01 times training, trigger augmentation, and a frozen classifier—reliably makes the backdoor part of the teacher's feature geometry and dark knowledge; the paper offers only a verbal intuition for this, and its own ablations show the effect collapses at slightly higher learning rates or without augmentation.

What would settle it

Take a new dataset not used in the paper, train a benign teacher, generate a weak trigger with the same ASR budget and norm constraint, fine-tune at the prescribed small learning rate with the classifier frozen, then distill with vanilla KD. If the student's attack success rate on triggered inputs stays near chance while the teacher's ASR is high, the claimed transfer does not generalize beyond the tested settings.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If correct, existing surrogate-student and distillation-simulation stages in KD backdoor attacks are unnecessary, cutting attack cost dramatically.
  • A defender can no longer rely on trigger perceptibility or norm constraints to flag malicious teachers, since the trigger is imperceptible and non-adversarial on benign models.
  • Victims who weight the distillation loss heavily or use moderate temperatures are the most exposed, so distillation configuration itself modulates backdoor transfer.
  • The same low-learning-rate fine-tuning regime transfers across response-, feature-, and relation-based paradigms, implying the vulnerability is not specific to logit matching.
  • Because the trigger alone has negligible effect on benign models, static scanning of weights or inputs will likely miss the backdoor; detection must look at feature-space behavior.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial: The mechanism suggests a broader principle: very small learning-rate fine-tuning can convert any weak, task-irrelevant pattern into a transferable bias, so the same recipe may work for other transfer settings such as model extraction or federated distillation.
  • Editorial: Trigger augmentation appears essential, which suggests the backdoor is a form of generalized feature memorization rather than pixel-pattern overfitting; testing whether random noise patches plus augmentation behave similarly would clarify the boundary of the effect.
  • Editorial: The fragility at slightly higher learning rates implies a testable detector: monitor the loss landscape or feature-geometry shift during fine-tuning to catch backdoors, since the attack requires an unusually flat, slow adaptation.
  • Editorial: Because the attack succeeds across disjoint attacker/victim datasets and even under distribution shift, the backdoor signal is probably carried by class-similarity structure rather than by specific images, so defenses should target inter-class logit or feature relationships.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes BackWeak, a backdoor attack against knowledge distillation (KD). The attacker trains a benign teacher, generates a 'weak trigger' by constrained optimization with an ASR budget and an l-infinity bound, selects poisoning samples, and fine-tunes the teacher with the classifier head frozen and a learning rate two orders of magnitude below the original training rate. The claim is that this procedure embeds a transferable backdoor in the teacher's feature geometry / 'dark knowledge' such that it transfers to students trained by response-, feature-, and relation-based KD, without any surrogate student or distillation simulation. The evidence is a broad empirical evaluation on CIFAR-10 and ImageNet-50 across several architectures, with comparisons to re-implemented ADBA and a simplified SCAR baseline, plus extensive ablations of trigger constraints, poisoning ratio, sample selection, learning rate, augmentation, layer freezing, and victim-side distillation hyperparameters.

Significance. If the claimed phenomenon is robust, this is a significant negative result for the KD-backdoor literature: it would show that the surrogate-student and simulated-distillation machinery in prior attacks is unnecessary, and that a subtle, low-learning-rate fine-tuning regime can implant a genuinely transferable backdoor even with a visually imperceptible trigger. The paper also draws a useful conceptual distinction between backdoor triggers and universal adversarial perturbations (UAPs), and it provides a large ablation matrix. The main strength is the breadth of empirical conditions: multiple datasets, teacher/student architectures, and three KD paradigms. However, the central claim of 'reliable transfer' is currently supported only by single-run numbers in a narrow hyperparameter window; the mechanism is asserted rather than directly measured. The significance is therefore conditional on the robustness concerns being resolved.

major comments (5)
  1. [Sec. 5.1 / Sec. 5.4 (Tables 2, 10, 11)] All experiments are single-run. Every table reports one value per condition, with no standard deviations, no seeds, and no repeated trials. This is not a presentation issue: Table 10 shows student ASR varying from 3.64% to 97.43% under a 10x change in eta_ft, and Table 11 shows ImageNet-50 response-based ASR collapsing from 72.61% to 27.62% when trigger augmentation is removed. Without variance estimates, the headline claim that the backdoor 'reliably transfers' cannot be distinguished from a lucky configuration. Please report mean and standard deviation over at least 3-5 seeds for the main tables and for the critical ablations.
  2. [Sec. 3.2 / Sec. 5.4.2 (Tables 10, 13, 14)] The attack's effectiveness is confined to a narrow hyperparameter window that, by the paper's own threat model, is not under the attacker's control. Sec. 3.2 states the attacker has zero knowledge of the victim's KD loss, temperature, or hyperparameters. Yet Table 13 shows student ASR drops to 18.51% when alpha=0.1, and Table 14 shows ASR 21.09% when tau=1; both alpha=0.1 and tau=1 are common KD settings (the ADBA re-implementation in Appendix A.2 uses tau=1.0). Similarly, Table 10 shows that changing eta_ft from 1e-4 to 1e-3 or 1e-2, i.e., by a factor of 10-100, destroys transfer (student ASR <=33.5%). These results make the abstract's 'reliably transfers' claim too strong. Either demonstrate transfer across a realistic range of victim settings or explicitly scope the claim to the tuned configuration and state which victim settings are and are not vulnerable.
  3. [Sec. 4.5 / Sec. 5.4.1 (Table 10)] The proposed mechanism — that a very small learning rate causes 'smoother integration into the feature geometry' and thus embeds the backdoor in dark knowledge — is asserted, not directly demonstrated. The paper reports no measurement of feature-space alignment, representation similarity (e.g., CKA), logit distribution shifts, or any other quantity that would show the backdoor is encoded differently under eta_ft=1e-4 than under 1e-3 or 1e-2. The evidence is purely the discontinuous jump in student ASR in Table 10. This is load-bearing because the method's novelty is precisely that this regime suffices. Please add direct evidence, for example by comparing representation geometry or logit margin distributions across eta_ft values and correlating them with transfer success, or by testing the teacher on triggered and untriggered inputs to show the backdoor is not a localized decision-boundar
  4. [Sec. 5.1 / Sec. 5.2 (Tables 2, 3)] The baseline comparisons are weakened by re-implementation. The paper states that ADBA and OSCAR were re-implemented because official code was unavailable, and OSCAR is explicitly a simplified version of SCAR. Table 2 uses this ADBA re-implementation to support the claim that ADBA's success depends on UAP-like triggers, and Table 3 uses OSCAR to draw conclusions about SCAR. If the re-implementations differ from the official methods, the comparisons and the conceptual conclusions are not validated. Please provide the re-implementation code and configuration files, and, where possible, calibrate against numbers reported in the original papers. At minimum, the text should clearly state that OSCAR is an approximation, not the original SCAR.
  5. [Sec. 4.3 / Eq. (11) / Table 2] The 'weakness' of the trigger is partly enforced by construction, making the TITG numbers less independent than they appear. Eq. (11) stops optimizing L_push whenever ASR_epoch exceeds delta_ASR on the benign teacher, and Table 2's 'Trigger Only' TITG is measured on that same benign model. Thus a low TITG is not an independent confirmation that the trigger lacks adversarial effect. Please report TITG on held-out models (e.g., a different architecture or a freshly trained seed) and show how the ASR budget constrains the trigger's behavior on models that were not used in generation.
minor comments (5)
  1. [General / Appendix A] The paper says 'Our code is available at GitHub' but no URL is given in the text or appendix. As a result, code availability cannot be verified.
  2. [Appendix B / Table 17] Table 17 reports RegNet TITG values above the stated 10% threshold (e.g., 11.18% and 10.61% in the TO rows), while the text says TITG 'generally stays below 10%'. Please reconcile the wording with the numbers, or explicitly state that the threshold can be exceeded with RegNet and discuss whether those cases still satisfy the weak-trigger definition.
  3. [Sec. 5.1] The weak-trigger learning rate differs by two orders of magnitude between datasets (eta_tri=1e-4 for CIFAR-10 and 1e-2 for ImageNet-50) with no explanation. This is an important configuration choice and should be justified.
  4. [Appendix C.2] The CINIC-10 distribution-shift experiment reports high ASR despite a large drop in BA, but there is no comparison to a normally trained teacher's BA on CINIC-10 or to a benign KD student under the same shift. Without such comparators, it is unclear whether the attack 'remains highly effective' or whether the KD itself is simply degraded.
  5. [Tables 2 / Figure 2] The text refers to values 'highlighted in red' and to visual trigger comparisons, but in a monochrome print or PDF the color highlighting is lost. Consider using symbols or bold/underline instead of color alone.

Circularity Check

0 steps flagged

No circularity: the transfer claim is an external empirical outcome; the weak-trigger constraint is a design choice, not a derived prediction.

full rationale

BackWeak's central assertion is empirical: fine-tuning a benign teacher with a weak trigger at a small learning rate yields a backdoor that transfers to students under response-, feature-, and relation-based KD. The paper contains no formal derivation whose output is identified, by construction or by self-citation, with an input. Equation 11 does constrain trigger generation with an ASR budget delta_ASR on the benign teacher, so the 'weakness' of BackWeak triggers is partly a design constraint rather than an independent discovery; however, the paper's transfer results are measured on victim-side students after distillation and are not contained in that constraint. The paper also reports TITG rather than the directly optimized ASR; these quantities are correlated but not identical, so the weakness report is not a pure restatement of Eq. 11. No load-bearing self-citation appears: citations such as [46] are background or motivation, and no uniqueness theorem or prior work by the same authors is invoked to force the conclusion. Tables 10, 11, 13, and 14 show sensitivity to eta_ft, trigger augmentation, alpha, and tau, and the experiments appear single-seed, but fragility and narrow tuning are robustness/generality concerns, not circularity. Under the stated rules, the central claim reduces to an external empirical outcome, so the appropriate finding is no significant circularity.

Axiom & Free-Parameter Ledger

7 free parameters · 4 axioms · 0 invented entities

The central claim rests on several tuned hyperparameters (ηft, δASR, λ, μ, ϵ0, ρ, augmentation) plus domain assumptions about the distillation protocol and data distribution. No new physical or architectural entities are introduced. The 100x-learning-rate regime is the most ad hoc component: it is empirically necessary but not explained from first principles.

free parameters (7)
  • fine-tuning learning rate ηft = 1e-4
    Central to the method; ablation (Table 10) shows transfer collapses at 1e-3 or 1e-2 and degrades at 1e-5.
  • ASR budget δASR = 0.05
    Defines 'weak trigger' during generation (Eq. 11); Table 7 shows relaxing it to 0.20 raises TITG above the weak-trigger threshold.
  • margin weight λ and margin μ = λ=1, μ=1
    Control trigger adversariality in Eqs. 10-11; ablation shows larger values improve ASR but break the weak-trigger property.
  • ℓ∞ constraint ϵ0 = 8/255
    Sets trigger imperceptibility; ablation over 8/255, 16/255, 32/255 shows ASR can increase with visibility.
  • poisoning ratio ρ = 0.3
    Balances teacher/student ASR against benign accuracy; Table 8 shows ρ=0.5 raises ASR but lowers BA.
  • target label yt = 3
    Arbitrarily chosen target class; no analysis of sensitivity to target class, though one OSCAR experiment uses target 0.
  • random trigger augmentation A = random flip & crop
    Required for transfer; without it ImageNet-50 response-KD student ASR drops from 72.61% to 27.62% (Table 11).
axioms (4)
  • domain assumption The victim's distillation process matches teacher soft outputs/features on clean data and the student never sees triggered examples during training.
    The attack transfer relies on the teacher's clean-output 'dark knowledge' carrying the backdoor signal (Eq. 1, Sec. 3.2).
  • domain assumption Attacker's dataset D_A and victim's dataset D_V are disjoint but drawn from a similar underlying data distribution P.
    Stated in Sec. 3.2 and Sec. 4.1; the authors test one distribution-shift case (CINIC-10) and claim robustness.
  • domain assumption The attacker has white-box control over the teacher model and its training, but zero knowledge of the student or victim's KD hyperparameters.
    Standard threat model in Sec. 3.2; it justifies attacking via teacher-only fine-tuning.
  • ad hoc to paper The 100x-reduced learning rate regime embeds the backdoor in 'dark knowledge' rather than overfitting the trigger; this is asserted from ablation, not derived.
    Sec. 5.4.1 and Table 10: the paper says 'We suppose that' a larger ηft creates sharp decoupled boundaries; no mathematical or mechanistic proof is provided.

pith-pipeline@v1.3.0-alltime-deepseek · 25569 in / 12848 out tokens · 125328 ms · 2026-08-03T22:05:32.183548+00:00 · methodology

0 comments
read the original abstract

Knowledge Distillation (KD) is essential for compressing large models, yet relying on pre-trained "teacher" models downloaded from third-party repositories introduces serious security risks--most notably backdoor attacks. Existing KD backdoor methods are typically complex and computationally intensive: they employ surrogate student models and simulated distillation to guarantee transferability, and construct triggers similar to universal adversarial perturbations (UAPs), which being not stealthy in magnitude, inherently exhibit strong adversarial behavior. This work questions whether such complexity is necessary and constructs stealthy "weak" triggers--imperceptible perturbations that have negligible adversarial effect. We propose BackWeak, a simple, surrogate-free attack paradigm. BackWeak shows that a powerful backdoor can be implanted by simply fine-tuning a benign teacher with a weak trigger using a very small learning rate. We demonstrate that this delicate fine-tuning is sufficient to embed a backdoor that reliably transfers to diverse student architectures during a victim's standard distillation process, yielding high attack success rates. Extensive empirical evaluations on multiple datasets, model architectures, and KD methods show that BackWeak is efficient, simpler, and often more stealthy than previous elaborate approaches. This work calls on researchers studying KD backdoor attacks to pay particular attention to the trigger's potential adversarial characteristics.

Figures

Figures reproduced from arXiv: 2511.12046 by Dongdong Zhao, Shanmin Wang.

Figure 1
Figure 1. Figure 1: Overview of the proposed BackWeak workflow. TABLE 1: Important notations. Notation Definition Models and Data fθT , fθS Teacher model, student model. fθ ⋆ T Final malicious (backdoored) teacher model. hθT,ft Feature extractor part of the teacher model. cθT,cls Classification layer of the teacher model. X, Y Input space and label space (C classes). DA, DV Attacker’s dataset, victim’s (disjoint) dataset. P U… view at source ↗
Figure 2
Figure 2. Figure 2: Visualization of the trigger and its application results, with LPIPS values (computed against the original images) [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

57 extracted references · 15 linked inside Pith

  1. [1]

    Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding,

    S. Han, H. Mao, and W. J. Dally, “Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding,”arXiv preprint arXiv:1510.00149, 2015

    Pith/arXiv arXiv 2015

  2. [2]

    Knowledge distillation: A survey,

    J. Gou, B. Yu, S. J. Maybank, and D. Tao, “Knowledge distillation: A survey,”International journal of computer vision, vol. 129, no. 6, pp. 1789–1819, 2021

    2021

  3. [3]

    Categories of response-based, feature-based, and relation-based knowledge distillation,

    C. Yang, X. Yu, Z. An, and Y . Xu, “Categories of response-based, feature-based, and relation-based knowledge distillation,” inAdvance- ments in knowledge distillation: towards new horizons of intelligent systems. Springer, 2023, pp. 1–32

    2023

  4. [4]

    A survey on knowledge distillation: Recent advancements,

    A. Moslemi, A. Briskina, Z. Dang, and J. Li, “A survey on knowledge distillation: Recent advancements,”Machine Learning with Applica- tions, vol. 18, p. 100605, 2024

    2024

  5. [5]

    Hugging face security docs,

    Hugging Face Inc., “Hugging face security docs,” 2025. [Online]. Available: https://huggingface.co/docs/hub/security

    2025

  6. [6]

    Badnets: Identifying vulnera- bilities in the machine learning model supply chain,

    T. Gu, B. Dolan-Gavitt, and S. Garg, “Badnets: Identifying vulnera- bilities in the machine learning model supply chain,”arXiv preprint arXiv:1708.06733, 2017

    Pith/arXiv arXiv 2017

  7. [7]

    Wanet–imperceptible warping-based back- door attack,

    A. Nguyen and A. Tran, “Wanet–imperceptible warping-based back- door attack,”arXiv preprint arXiv:2102.10369, 2021

    Pith/arXiv arXiv 2021

  8. [8]

    Bppattack: Stealthy and efficient trojan attacks against deep neural networks via image quantization and contrastive adversarial learning,

    Z. Wang, J. Zhai, and S. Ma, “Bppattack: Stealthy and efficient trojan attacks against deep neural networks via image quantization and contrastive adversarial learning,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 15 074–15 084

    2022

  9. [9]

    Backdoor learning: A survey,

    Y . Li, Y . Jiang, Z. Li, and S.-T. Xia, “Backdoor learning: A survey,” IEEE transactions on neural networks and learning systems, vol. 35, no. 1, pp. 5–22, 2022

    2022

  10. [10]

    Backdoor attacks and countermeasures on deep learning: A comprehensive review,

    Y . Gao, B. G. Doan, Z. Zhang, S. Ma, J. Zhang, A. Fu, S. Nepal, and H. Kim, “Backdoor attacks and countermeasures on deep learning: A comprehensive review,”arXiv preprint arXiv:2007.10760, 2020

    Pith/arXiv arXiv 2007

  11. [11]

    Backdoor attacks and countermeasures in natural language processing models: A comprehensive security review,

    P. Cheng, Z. Wu, W. Du, H. Zhao, W. Lu, and G. Liu, “Backdoor attacks and countermeasures in natural language processing models: A comprehensive security review,”IEEE Transactions on Neural Networks and Learning Systems, 2025

    2025

  12. [12]

    Backdoor attacks against voice recog- nition systems: A survey,

    B. Yan, J. Lan, and Z. Yan, “Backdoor attacks against voice recog- nition systems: A survey,”ACM Computing Surveys, vol. 57, no. 3, pp. 1–35, 2024

    2024

  13. [13]

    Anti-distillation backdoor attacks: Backdoors can really survive in knowledge distillation,

    Y . Ge, Q. Wang, B. Zheng, X. Zhuang, Q. Li, C. Shen, and C. Wang, “Anti-distillation backdoor attacks: Backdoors can really survive in knowledge distillation,” inProceedings of the 29th ACM International Conference on Multimedia, 2021, pp. 826–834

    2021

  14. [14]

    Taught well learned ill: Towards distillation-conditional backdoor attack,

    Y . Chen, B. Li, Y . Yuan, L. Qi, Y . Li, T. Zhang, Z. Qin, and K. Ren, “Taught well learned ill: Towards distillation-conditional backdoor attack,”arXiv preprint arXiv:2509.23871, 2025

    arXiv 2025

  15. [15]

    Uni- versal adversarial perturbations,

    S.-M. Moosavi-Dezfooli, A. Fawzi, O. Fawzi, and P. Frossard, “Uni- versal adversarial perturbations,” inProceedings of the IEEE confer- ence on computer vision and pattern recognition, 2017, pp. 1765– 1773

    2017

  16. [16]

    Ink: Inheritable natural backdoor attack against model distillation,

    X. Liu, M. Yi, K. Ding, B. Xin, Y . Xu, L. Yan, and C. Shen, “Ink: Inheritable natural backdoor attack against model distillation,”arXiv preprint arXiv:2304.10985, 2023

    Pith/arXiv arXiv 2023

  17. [17]

    Narcissus: A practical clean-label backdoor attack with limited information,

    Y . Zeng, M. Pan, H. A. Just, L. Lyu, M. Qiu, and R. Jia, “Narcissus: A practical clean-label backdoor attack with limited information,” in Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, 2023, pp. 771–785

    2023

  18. [18]

    Distilling the knowledge in a neural network,

    G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,”arXiv preprint arXiv:1503.02531, 2015

    Pith/arXiv arXiv 2015

  19. [19]

    Exploring the knowledge transferred by response- based teacher-student distillation,

    L. Song, X. Gong, H. Zhou, J. Chen, Q. Zhang, D. Doermann, and J. Yuan, “Exploring the knowledge transferred by response- based teacher-student distillation,” inProceedings of the 31st ACM International Conference on Multimedia, 2023, pp. 2704–2713

    2023

  20. [20]

    Rckd: Response-based cross-task knowledge distillation for pathological image analysis,

    H. Kim, T.-Y . Kwak, H. Chang, S. W. Kim, and I. Kim, “Rckd: Response-based cross-task knowledge distillation for pathological image analysis,”Bioengineering, vol. 10, no. 11, p. 1279, 2023

    2023

  21. [21]

    Flexible temperature parallel distillation for dense object detection: Make response-based knowledge distillation great again,

    Y . Song, P. Zhang, W. Huang, Y . Zha, and Y . Zhang, “Flexible temperature parallel distillation for dense object detection: Make response-based knowledge distillation great again,”IEEE Transac- tions on Circuits and Systems for Video Technology, 2025

    2025

  22. [22]

    Fitnets: Hints for thin deep nets. arxiv 2014,

    A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, and Y . Bengio, “Fitnets: Hints for thin deep nets. arxiv 2014,”arXiv preprint arXiv:1412.6550, 2014

    Pith/arXiv arXiv 2014

  23. [23]

    Knowledge distillation with the reused teacher classifier,

    D. Chen, J.-P. Mei, H. Zhang, C. Wang, Y . Feng, and C. Chen, “Knowledge distillation with the reused teacher classifier,” inPro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 11 933–11 942

    2022

  24. [24]

    Show, attend and distill: Knowledge distillation via attention-based feature matching,

    M. Ji, B. Heo, and S. Park, “Show, attend and distill: Knowledge distillation via attention-based feature matching,” inProceedings of the AAAI conference on artificial intelligence, vol. 35, no. 9, 2021, pp. 7945–7952

    2021

  25. [25]

    Vitkd: Feature- based knowledge distillation for vision transformers,

    Z. Yang, Z. Li, A. Zeng, Z. Li, C. Yuan, and Y . Li, “Vitkd: Feature- based knowledge distillation for vision transformers,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, 2024, pp. 1379–1388

    2024

  26. [26]

    Relational knowledge distilla- tion,

    W. Park, D. Kim, Y . Lu, and M. Cho, “Relational knowledge distilla- tion,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 3967–3976

    2019

  27. [27]

    Knowledge distillation from a stronger teacher,

    T. Huang, S. You, F. Wang, C. Qian, and C. Xu, “Knowledge distillation from a stronger teacher,”Advances in Neural Information Processing Systems, vol. 35, pp. 33 716–33 727, 2022

    2022

  28. [28]

    A new similarity-based relational knowledge distillation method,

    X. Xin, H. Song, and J. Gou, “A new similarity-based relational knowledge distillation method,” inICASSP 2024-2024 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 3535–3539

    2024

  29. [29]

    Revisit the power of vanilla knowledge distillation: from small scale to large scale,

    Z. Hao, J. Guo, K. Han, H. Hu, C. Xu, and Y . Wang, “Revisit the power of vanilla knowledge distillation: from small scale to large scale,”Advances in Neural Information Processing Systems, vol. 36, pp. 10 170–10 183, 2023

    2023

  30. [30]

    Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter,

    V . Sanh, L. Debut, J. Chaumond, and T. Wolf, “Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter,”arXiv preprint arXiv:1910.01108, 2019

    Pith/arXiv arXiv 1910

  31. [31]

    Shiftkd: Benchmarking knowledge distillation under distribution shift,

    S. Zhang, Y . Luo, Z. Lyu, and X. Chen, “Shiftkd: Benchmarking knowledge distillation under distribution shift,”Neural Networks, vol. 192, p. 107838, 2025

    2025

  32. [32]

    Certified adversarial ro- bustness via randomized smoothing,

    J. Cohen, E. Rosenfeld, and Z. Kolter, “Certified adversarial ro- bustness via randomized smoothing,” ininternational conference on machine learning. PMLR, 2019, pp. 1310–1320

    2019

  33. [33]

    It is all about data: A survey on the effects of data on adversarial robustness,

    P. Xiong, M. Tegegn, J. S. Sarin, S. Pal, and J. Rubin, “It is all about data: A survey on the effects of data on adversarial robustness,”ACM Computing Surveys, vol. 56, no. 7, pp. 1–41, 2024

    2024

  34. [34]

    Universal adversarial perturbations: A survey,

    A. Chaubey, N. Agrawal, K. Barnwal, K. K. Guliani, and P. Mehta, “Universal adversarial perturbations: A survey,”arXiv preprint arXiv:2005.08087, 2020

    Pith/arXiv arXiv 2005

  35. [35]

    Trans- ferable universal adversarial perturbations using generative models,

    A. S. Hashemi, A. B ¨ar, S. Mozaffari, and T. Fingscheidt, “Trans- ferable universal adversarial perturbations using generative models,” arXiv preprint arXiv:2010.14919, 2020

    Pith/arXiv arXiv 2010

  36. [36]

    Improving generalization of universal adversarial perturbation via dynamic maximin optimization,

    Y . Zhang, Y . Xu, J. Shi, L. Y . Zhang, S. Hu, M. Li, and Y . Zhang, “Improving generalization of universal adversarial perturbation via dynamic maximin optimization,” inProceedings of the AAAI Con- ference on Artificial Intelligence, vol. 39, no. 10, 2025, pp. 10 293– 10 301

    2025

  37. [37]

    Invisible and efficient backdoor attacks for compressed deep neural networks,

    H. Phan, Y . Xie, J. Liu, Y . Chen, and B. Yuan, “Invisible and efficient backdoor attacks for compressed deep neural networks,” inICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 96–100

    2022

  38. [38]

    Honeypotnet: Backdoor attacks against model extraction,

    Y . Wang, T. Gu, Y . Teng, Y . Wang, and X. Ma, “Honeypotnet: Backdoor attacks against model extraction,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 39, no. 8, 2025, pp. 8087–8095

    2025

  39. [39]

    Backdoor attacks against image-to-image networks,

    W. Jiang, H. Li, J. He, R. Zhang, G. Xu, T. Zhang, and R. Lu, “Backdoor attacks against image-to-image networks,”arXiv preprint arXiv:2407.10445, 2024

    Pith/arXiv arXiv 2024

  40. [40]

    Uibdiffusion: Universal imperceptible backdoor attack for diffusion models,

    Y . Han, B. Zhao, R. Chu, F. Luo, B. Sikdar, and Y . Lao, “Uibdiffusion: Universal imperceptible backdoor attack for diffusion models,” in Proceedings of the Computer Vision and Pattern Recognition Con- ference, 2025, pp. 19 186–19 196

    2025

  41. [41]

    Patchbackdoor: Backdoor attack against deep neural networks without model modifi- cation,

    Y . Yuan, R. Kong, S. Xie, Y . Li, and Y . Liu, “Patchbackdoor: Backdoor attack against deep neural networks without model modifi- cation,” inProceedings of the 31st ACM international conference on multimedia, 2023, pp. 9134–9142

    2023

  42. [42]

    Badmerging: Backdoor attacks against model merging,

    J. Zhang, J. Chi, Z. Li, K. Cai, Y . Zhang, and Y . Tian, “Badmerging: Backdoor attacks against model merging,” inProceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, 2024, pp. 4450–4464

    2024

  43. [43]

    Transferring backdoors between large language models by knowledge distillation,

    P. Cheng, Z. Wu, T. Ju, W. Du, and Z. Z. G. Liu, “Transferring backdoors between large language models by knowledge distillation,” arXiv preprint arXiv:2408.09878, 2024

    Pith/arXiv arXiv 2024

  44. [44]

    Like teacher, like pupil: Transferring backdoors via feature-based knowledge distillation,

    J. Chen, Z. Cao, R. Chen, H. Zheng, X. Li, Q. Xuan, and X. Yang, “Like teacher, like pupil: Transferring backdoors via feature-based knowledge distillation,”Computers & Security, vol. 146, p. 104041, 2024

    2024

  45. [45]

    How to backdoor the knowledge distillation,

    C. Wu, Q. Ma, P. Mitra, and S. Zhu, “How to backdoor the knowledge distillation,”arXiv preprint arXiv:2504.21323, 2025

    arXiv 2025

  46. [46]

    An empirical study of example forgetting during deep neural network learning,

    M. Toneva, A. Sordoni, R. T. d. Combes, A. Trischler, Y . Bengio, and G. J. Gordon, “An empirical study of example forgetting during deep neural network learning,”arXiv preprint arXiv:1812.05159, 2018

    Pith/arXiv arXiv 2018

  47. [47]

    Learning multiple layers of features from tiny images,

    A. Krizhevsky, G. Hintonet al., “Learning multiple layers of features from tiny images,” University of Toronto, Tech. Rep., 2009

    2009

  48. [48]

    Imagenet: A large-scale hierarchical image database,

    J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in2009 IEEE conference on computer vision and pattern recognition. IEEE, 2009, pp. 248–255

    2009

  49. [49]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” inProceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778

    2016

  50. [50]

    Very deep convolutional networks for large-scale image recognition,

    K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,”arXiv preprint arXiv:1409.1556, 2014

    Pith/arXiv arXiv 2014

  51. [51]

    Designing network design spaces,

    I. Radosavovic, R. P. Kosaraju, R. Girshick, K. He, and P. Doll ´ar, “Designing network design spaces,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 10 428–10 436

    2020

  52. [52]

    Mobilenetv2: Inverted residuals and linear bottlenecks,

    M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, “Mobilenetv2: Inverted residuals and linear bottlenecks,” inPro- ceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 4510–4520

    2018

  53. [53]

    Shufflenet v2: Practical guidelines for efficient cnn architecture design,

    N. Ma, X. Zhang, H.-T. Zheng, and J. Sun, “Shufflenet v2: Practical guidelines for efficient cnn architecture design,” inProceedings of the European conference on computer vision (ECCV), 2018, pp. 116–131

    2018

  54. [54]

    Densely connected convolutional networks,

    G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” inProceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 4700–4708

    2017

  55. [55]

    The unreasonable effectiveness of deep features as a perceptual metric,

    R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” inCVPR, 2018

    2018

  56. [56]

    The alignment property of sgd noise and how it helps select flat minima: A stability analysis,

    L. Wu, M. Wang, and W. Su, “The alignment property of sgd noise and how it helps select flat minima: A stability analysis,”Advances in Neural Information Processing Systems, vol. 35, pp. 4680–4693, 2022

    2022

  57. [57]

    Cinic- 10 is not imagenet or cifar-10,

    L. N. Darlow, E. J. Crowley, A. Antoniou, and A. J. Storkey, “Cinic- 10 is not imagenet or cifar-10,”arXiv preprint arXiv:1810.03505, 2018. Ethics Considerations This work studies security vulnerabilities in knowledge distillation when a model distributor intentionally embeds a backdoor into the teacher model. Our research focuses solely on analyzing syst...

    Pith/arXiv arXiv 2018