Pith. sign in

REVIEW 1 major objections 5 minor 61 references

This paper claims that a classification API can poison the soft labels it serves so that a distilling student suffers large accuracy drops while the teacher's top-1 prediction is preserved exactly, per query.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 23:14 UTC pith:56F7N37U

load-bearing objection Sharp analytical defense with a real zero-cost guarantee, but the headline resistance numbers are only shown against a proxy that matches the attacker's student. the 1 major comments →

arxiv 2607.15467 v1 pith:56F7N37U submitted 2026-07-16 cs.LG cs.CR

ADS-C: Antidistillation Sampling for Classification

Khawaja Abaid Ullah , Mohammad Javad Khojasteh This is my paper
classification cs.LG cs.CR MSC 68T0768T05
keywords antidistillationknowledge distillationmodel extractioninference-time defensemargin budgetsoft-label poisoningclassifier overconfidencetemperature scaling
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper claims that a classification API can defend against distillation-based model extraction by perturbing served probability vectors under a closed-form, per-input margin budget that leaves every served top-1 prediction untouched. The central result is that the defended teacher's top-1 accuracy is identical to the undefended teacher's, deterministically, while a student trained on the defended soft labels loses 17.4, 29.6, and 13.3 percentage points on CIFAR-100, CIFAR-10, and Tiny-ImageNet. The paper also diagnoses why the unadapted defense fails: classifier overconfidence makes it inert at low strengths, unfavorable at moderate strengths, and saturating at high strengths, with temperature softening only rescaling the same bad trade-off. A sympathetic reader would care because this removes the utility cost that previously made perturbation defenses a losing trade: the defense's only cost is served fidelity, which falls almost entirely on the attacker.

Core claim

On its own terms, the paper's discovery is that the effectiveness of antidistillation sampling in classification is governed by the distribution of the teacher's per-input confidence margins. Well-trained teachers are overconfident, so the unmodified defense is inert at low strength, degrades the teacher faster than the student at moderate strength, and saturates with both parties near floors. The repair composes the perturbation under a per-input margin budget: each query receives the largest penalty strength that provably keeps its served top-1 unchanged (Proposition 1), making teacher cost exactly zero and reversing the incentive to distill from soft outputs.

What carries the argument

The per-input margin budget (Equation 8) is the device that carries the argument. For each query it computes, in closed form, the largest poison strength λ(x) that keeps the top-class logit at least m log-units ahead of every rival, using the teacher's logit gaps and the poison's per-rival gain rates. Proposition 1 then guarantees log p̂_t − log p̂_j ≥ min(m, gap_j) > 0, so the argmax is preserved per query, deterministically. The companion flip-threshold cumulative distribution F(λ) makes the unbudgeted defense's teacher cost predictable from teacher and proxy outputs alone.

Load-bearing premise

The proxy student trained on the teacher's training split must be a faithful stand-in for the actual attacker's student; if the attacker uses a very different architecture or data distribution, the gradient-directed poison may be aimed at the wrong weaknesses and the reported student degradation could shrink.

What would settle it

Train an attacker's student with an architecture substantially different from the defender's proxy (e.g., a transformer or a much larger CNN) on the same defended API and measure distilled-student top-1. If degradation on CIFAR-100 drops well below the reported 17.4 pp at λ=3 while teacher top-1 stays constant, the proxy-fidelity premise is violated. Alternatively, compute F(λ) from teacher and proxy outputs on a new dataset and check whether the measured teacher-cost curve under unmodified ADS matches the closed-form prediction to within about 2 pp; a mismatch would falsify the overconfidence

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • A defender can deploy ADS-C without losing any top-1 accuracy, eliminating the utility argument for serving undefended soft labels.
  • Hard-label attackers gain nothing beyond the distillation floor, since the served argmax never changes; soft-label students fall below that floor, reversing the extraction incentive.
  • Temperature softening is not a remedy: it shifts the phase transition according to F(τλ) but all configurations lie on the same unfavorable trade-off curve.
  • The defended teacher's cost under unmodified ADS can be predicted in closed form from F(λ) before any student is trained.
  • Relaxations (stochastic enforcement, minimum served strength) allow a defender to trade controlled teacher cost for attacker uncertainty, priced in closed form against the attacker's best response.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The margin-budget principle—cap any perturbation by the quantity it must not disturb—may generalize to other API services that expose continuous outputs, such as embedding or ranking endpoints, where a similar closed-form budget could be derived.
  • The proxy-fidelity premise is the untested hinge: if attackers use student architectures or query distributions far from the defender's proxy, the poison direction may be misaimed and the measured degradation could shrink; the paper provides no sensitivity analysis over attacker architectures.
  • The decomposition into argmax-safe versus flip-collateral damage suggests that future defenses should target informative non-target structure on representative inputs rather than maximize distributional distortion on uncertain rows.
  • Because F(λ) is computable from teacher and proxy outputs alone, a deployment-time audit is possible: operators can pre-register the expected teacher-cost curve and monitor live attacks for deviations from it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 5 minor

Summary. This paper adapts Antidistillation Sampling (ADS) to image classification. It first diagnoses the direct transfer: because well-trained classifiers are overconfident and their per-input margins are heterogeneous, a global strength sweep passes through an inert window, a phase transition, and saturation, with the teacher-side cost predicted by the CDF F(λ) of per-input flip thresholds. It then proves that temperature softening rescales this transition by F_τ(λ)=F(τλ). The proposed method, ADS-C, composes the ADS perturbation under a per-input margin budget (Algorithm 1) so that the served top-1 prediction is provably unchanged for every query (Proposition 1), while the served probability vector is still perturbed. Empirically, ADS-C gives exactly zero teacher top-1 drop and lowers distilled-student accuracy by 17.4/29.6/13.3 pp on CIFAR-100/CIFAR-10/Tiny-ImageNet, below the hard-label distillation floor. The paper also analyzes stochastic budget enforcement and a minimum-strength floor, reports a negative result on margin-ranked sacrifice, and releases code and experiment drivers.

Significance. If the empirical student-degradation results are robust, this is a valuable contribution: it provides a clean, analytically predicted transition-curve characterization of an inference-time perturbation defense, a deterministic argmax-preservation guarantee with a correct and simple proof, careful ablations including a hard-label control and a served-fidelity comparison, and reproducible code. The analytical core is genuinely parameter-free: F(λ) is computed from teacher and proxy outputs before any student is trained, the temperature rescaling F_τ(λ)=F(τλ) is a closed-form prediction, and Proposition 1 is proven. The main caveat is that the headline security claim is demonstrated only for a proxy that matches the attacker's student architecture, which is narrower than the stated threat model; this is the load-bearing gap that the major comment below addresses.

major comments (1)
  1. [III, IV-B, VIII-A] The threat model (Sec. III) explicitly gives the adversary a student architecture of its choosing, but in the evaluation the proxy always matches the student architecture (Sec. VIII-A: 'The proxy always matches the student architecture'), and the only proxy-state variation tested (Sec. V-C) changes the proxy's training signal, not its architecture or query distribution. The student-degradation numbers (17.4/29.6/13.3 pp) are therefore established only for an attacker whose student coincides with the defender's proxy. Nothing in the mechanism guarantees that the gradient direction computed from P also degrades a differently structured student: the finite-difference Δ̂ is a direction in P's output space, not a universal poison. If a mismatched student partially ignores that direction, the reported drops could shrink and the 'reversed incentive' claim of Sec. VIII-E could fail, even though
minor comments (5)
  1. [V-D] The definition of the inert window is stated three ways: 'λ < 0.12 on CIFAR-100, λ < 0.30 on CIFAR-10', the strict transition threshold λ_c≈0.02/0.09, and the teacher-cost onset at 0.066/0.135. Please clarify which definition is used for the shaded inert region and reconcile the numbers, since 'inert' should mean both teacher and student are within noise.
  2. [VI-B] The text calls F(τλ) 'the pre-registered prediction'. Since there is no evidence of a formal preregistration, 'analytically predicted' or 'closed-form prediction' would be more accurate.
  3. [III vs IV-B] The threat model grants the defender only a 'modest labeled holdout set', but the proxy is trained on the teacher's full training split (Sec. IV-B). These are different resource assumptions. Please state clearly whether the defender is assumed to have the original labeled training data.
  4. [VIII-B, Table 2] Under a deterministic argmax-preservation guarantee, the reported 'observed ≤1.4×10^-6 pp' teacher drop is surprising; if the served argmax is identical to the teacher argmax on every query, the accuracy difference should be exactly zero. Please explain how this number is computed (e.g., floating-point summation of identical booleans) so readers do not think the guarantee is approximate.
  5. [VIII (general)] The experimental section compares ADS-C only with unmodified ADS and its own ablations. A direct comparison with an existing perturbation defense (e.g., Reverse Sigmoid or ModelGuard) on the same teacher/student pairs would support the claim of being 'the first antidistillation defense for classification whose utility cost is exactly zero' and would help place the trade-off-rate improvement in context.

Circularity Check

0 steps flagged

No significant circularity; the zero-cost guarantee is a stated construction, and the transition/temperature predictions are parameter-free forward models validated against measurements.

full rationale

I walked the derivation chain and found no step that reduces to its own inputs by construction or by a load-bearing self-citation. The transition-curve prediction F(λ) is computed as the flip-threshold CDF (Eqs. 5–6) from teacher logits and proxy outputs, then compared with measured teacher-accuracy drops; the reported 1–2 pp residual shows it is an empirical forward model, not an identity. The temperature rescaling Fτ(λ) = F(τλ) follows from softmax logit scaling and is verified against trained student runs with no parameter fit to the outcome. Proposition 1 is an analytic guarantee about Algorithm 1: the budget (8) is explicitly the flip threshold with a margin floor held in reserve, so argmax preservation and zero teacher cost follow by algebra; the paper transparently presents this as a construction rather than as an independent empirical prediction. The headline student-degradation numbers (17.4/29.6/13.3 pp) are fresh measurements from students trained on the served vectors, not fitted or read off from the proxy. The only self-referential citation is an acknowledgment of RIT Research Computing, which is not load-bearing. The proxy-matches-student-architecture choice is a threat-model limitation with implications for external validity, but that is a robustness/correctness concern, not a circular step. The paper's stated limitations (e.g., no formal analysis of poison-subtraction attacks, no large-scale validation) are explicit scope caveats and do not indicate circularity.

Axiom & Free-Parameter Ledger

3 free parameters · 6 axioms · 0 invented entities

No new physical or conceptual entities are introduced; the proxy clones are technical constructs, not unexplained postulates. The free parameters are explicit method knobs with sensitivity analysis or sweeping, none fitted to the headline student-degradation numbers.

free parameters (3)
  • Defense strength λ = swept 0.1–3 (ADS-C); 0–10 (unmodified ADS)
    Global strength knob of the poison; not fitted to outcome, but chosen by hand and swept in experiments.
  • Margin floor m = 0.05
    Service-level guarantee on served log-odds margin; Appendix XI-C shows insensitivity over [0.01, 0.2], but it is a hand-chosen constant in the main runs.
  • Finite-difference step ε = 3e-3 (CIFAR-10/100), 1e-2 (Tiny-ImageNet)
    Calibrated to match the exact Jacobian-vector product (Appendix XI-B); not tuned to the central result but is a numerical hyperparameter.
axioms (6)
  • domain assumption The proxy student P, trained on the teacher's training split, faithfully represents the unknown attacker's student, so the gradient of ℓ(θ_P) points in a direction that harms the true distiller.
    Sec. IV-B constructs Δ̂ from P's gradient; Sec. V-C tests only proxy state variants, not architecture mismatch. The defense's efficacy depends on this.
  • domain assumption The attacker uses unlabeled in-distribution queries and pure soft-label KL loss at temperature 1, with no ground-truth labels.
    Stated in Sec. III as the threat model; the reported student degradation numbers are conditional on this fixed attacker.
  • domain assumption The defender holds a labeled holdout set and can train a small proxy offline; the teacher cannot be retrained.
    Sec. III Defender goals; used to compute g and Δ̂.
  • domain assumption Served utility is measured exclusively by top-1 label accuracy; full-vector consumers bear a separate served-fidelity cost.
    The 'zero utility cost' claim is defined on the served argmax; Sec. X acknowledges full-distribution consumers pay KL cost.
  • domain assumption Every input has a unique teacher argmax, so Proposition 1's condition is met.
    Proposition 1 states 'unique teacher argmax t'; ties are excluded.
  • standard math Softmax is invariant to additive logit constants and log-probability differences equal logit gaps; standard calculus used in Proposition 1 proof.
    Used throughout Secs. IV and VII; no nonstandard math.

pith-pipeline@v1.3.0-alltime-deepseek · 27566 in / 13557 out tokens · 126548 ms · 2026-08-01T23:14:28.714478+00:00 · methodology

0 comments
read the original abstract

Knowledge distillation enables an adversary to replicate a proprietary classifier by querying its prediction interface and training a surrogate on the returned probability vectors. Antidistillation sampling, proposed for large language models, counters this threat with an input-dependent, gradient-directed perturbation of the served distribution; its transfer to classification has not been studied. Adapting the defense to classification, we show its behavior is governed by the distribution of the teacher's per-input confidence margins. Because well-trained classifiers are severely overconfident, the direct transfer exhibits an inert window: below a closed-form-predictable threshold, it affects neither attacker nor defender; beyond it, the defense undergoes a phase transition and degrades the teacher faster than the attacker's student. Temperature softening rescales the transition in closed form, and every temperature configuration lies on the same unfavorable trade-off curve. Our method, ADS-C, composes the perturbation under a closed-form, per-input margin budget that provably preserves every served top-1 prediction, so the defended teacher's accuracy equals the undefended teacher's identically. Under this guarantee the distilled student still loses 17.4 percentage points on CIFAR-100, 29.6 on CIFAR-10, and 13.3 on Tiny-ImageNet; matching this degradation with the unmodified defense costs 27.5, 32.9, and 22.2 points of teacher accuracy. Because served labels are unchanged, a hard-label attacker gains nothing, while the defended soft output trains a student up to 29.7 points below that floor: the incentive to distill served probabilities is not merely removed but reversed. To our knowledge, ADS-C is the first antidistillation defense for classification whose utility cost is exactly zero.

Figures

Figures reproduced from arXiv: 2607.15467 by Khawaja Abaid Ullah, Mohammad Javad Khojasteh.

Figure 1
Figure 1. Figure 1: summarizes the setting. A defender trains a clas￾sifier T : X → ∆K−1 mapping an input space X to the probability simplex over K classes and deploys it behind a query API that returns, for each query x ∈ X , a served probability vector pˆ(x) ∈ ∆K−1 . Adversary. The adversary holds (i) black-box access to the API with an arbitrary but bounded query budget; (ii) an unlabeled query corpus drawn from the same d… view at source ↗
Figure 2
Figure 2. Figure 2: FIGURE 2 [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: FIGURE 3 [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: FIGURE 4 [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: FIGURE 5 [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: FIGURE 6 [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: FIGURE 7 [PITH_FULL_IMAGE:figures/full_fig_p009_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: FIGURE 8 [PITH_FULL_IMAGE:figures/full_fig_p011_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: FIGURE 9 [PITH_FULL_IMAGE:figures/full_fig_p012_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: FIGURE 10 [PITH_FULL_IMAGE:figures/full_fig_p012_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: FIGURE 11 [PITH_FULL_IMAGE:figures/full_fig_p013_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: FIGURE 12 [PITH_FULL_IMAGE:figures/full_fig_p014_12.png] view at source ↗
Figure 14
Figure 14. Figure 14: FIGURE 14 [PITH_FULL_IMAGE:figures/full_fig_p015_14.png] view at source ↗
Figure 16
Figure 16. Figure 16: FIGURE 16 [PITH_FULL_IMAGE:figures/full_fig_p019_16.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

61 extracted references · 14 linked inside Pith

  1. [1]

    Learning complex, extended sequences using the principle of history compression,

    J. Schmidhuber, “Learning complex, extended sequences using the principle of history compression,”Neural computation, vol. 4, no. 2, pp. 234–242, 1992

    1992

  2. [2]

    Distilling the knowledge in a neural network,

    G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,”arXiv preprint arXiv:1503.02531, 2015

    Pith/arXiv arXiv 2015

  3. [3]

    Mobilenets: Efficient convolutional neural networks for mobile vision applications,

    A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, “Mobilenets: Efficient convolutional neural networks for mobile vision applications,”arXiv preprint arXiv:1704.04861, 2017

    Pith/arXiv arXiv 2017

  4. [4]

    Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter,

    V . Sanh, L. Debut, J. Chaumond, and T. Wolf, “Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter,”arXiv preprint arXiv:1910.01108, 2019

    Pith/arXiv arXiv 1910

  5. [5]

    Training data-efficient image transformers & distillation through attention,

    H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. J ´egou, “Training data-efficient image transformers & distillation through attention,” inInternational conference on machine learning. PMLR, 2021, pp. 10 347–10 357

    2021

  6. [6]

    Knowledge distillation: A survey,

    J. Gou, B. Yu, S. J. Maybank, and D. Tao, “Knowledge distillation: A survey,”International journal of computer vision, vol. 129, no. 6, pp. 1789–1819, 2021

    2021

  7. [7]

    Stealing machine learning models via prediction{APIs},

    F. Tram `er, F. Zhang, A. Juels, M. K. Reiter, and T. Ristenpart, “Stealing machine learning models via prediction{APIs},” in25th USENIX security symposium (USENIX Security 16), 2016, pp. 601–618

    2016

  8. [8]

    A survey on model extraction attacks and defenses for large language models,

    K. Zhao, L. Li, K. Ding, N. Z. Gong, Y . Zhao, and Y . Dong, “A survey on model extraction attacks and defenses for large language models,” inProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V . 2, 2025, pp. 6227–6236

    2025

  9. [9]

    DeepSeek trained AI model using distillation, now a dis- ruptive force,

    CNBC, “DeepSeek trained AI model using distillation, now a dis- ruptive force,” CNBC, Feb. 2025, https://www.cnbc.com/2025/02/21/ deepseek-trained-ai-model-using-distillation-now-a-disruptive-force. html

    2025

  10. [10]

    Detecting and preventing distillation attacks,

    Anthropic, “Detecting and preventing distillation attacks,” Anthropic News, February 2026, accessed: March 23,

    2026

  11. [11]

    Deep intellectual property protection: A survey,

    Y . Sun, T. Liu, P. Hu, Q. Liao, S. Fu, N. Yu, D. Guo, Y . Liu, and L. Liu, “Deep intellectual property protection: A survey,”arXiv preprint arXiv:2304.14613, 2023

    Pith/arXiv arXiv 2023

  12. [12]

    Intellectual property protection for deep learning model and dataset intelligence,

    Y . Jiang, Y . Gao, C. Zhou, H. Hu, S. Chen, A. Fu, and W. Susilo, “Intellectual property protection for deep learning model and dataset intelligence,”Engineering Applications of Artificial Intelligence, vol. 163, p. 113024, 2026

    2026

  13. [13]

    Membership inference attacks against machine learning models,

    R. Shokri, M. Stronati, C. Song, and V . Shmatikov, “Membership inference attacks against machine learning models,” in2017 IEEE symposium on security and privacy (SP). IEEE, 2017, pp. 3–18

    2017

  14. [14]

    Ml-leaks: Model and data independent membership inference at- tacks and defenses on machine learning models,

    A. Salem, Y . Zhang, M. Humbert, P. Berrang, M. Fritz, and M. Backes, “Ml-leaks: Model and data independent membership inference at- tacks and defenses on machine learning models,”arXiv preprint arXiv:1806.01246, 2018

    Pith/arXiv arXiv 2018

  15. [15]

    Practical black-box attacks against machine learning,

    N. Papernot, P. McDaniel, I. Goodfellow, S. Jha, Z. B. Celik, and A. Swami, “Practical black-box attacks against machine learning,” in Proceedings of the 2017 ACM on Asia conference on computer and communications security, 2017, pp. 506–519

    2017

  16. [16]

    Evasion attacks against machine learning at test time,

    B. Biggio, I. Corona, D. Maiorca, B. Nelson, N. ˇSrndi´c, P. Laskov, G. Giacinto, and F. Roli, “Evasion attacks against machine learning at test time,” inJoint European conference on machine learning and knowledge discovery in databases. Springer, 2013, pp. 387–402

    2013

  17. [17]

    Adversarial learning,

    D. Lowd and C. Meek, “Adversarial learning,” inProceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining, 2005, pp. 641–647

    2005

  18. [18]

    Privacy in pharmacogenetics: An{End-to-End}case study of person- alized warfarin dosing,

    M. Fredrikson, E. Lantz, S. Jha, S. Lin, D. Page, and T. Ristenpart, “Privacy in pharmacogenetics: An{End-to-End}case study of person- alized warfarin dosing,” in23rd USENIX security symposium (USENIX Security 14), 2014, pp. 17–32

    2014

  19. [19]

    Model inversion attacks that exploit confidence information and basic countermeasures,

    M. Fredrikson, S. Jha, and T. Ristenpart, “Model inversion attacks that exploit confidence information and basic countermeasures,” in Proceedings of the 22nd ACM SIGSAC conference on computer and communications security, 2015, pp. 1322–1333

    2015

  20. [20]

    Hacking smart machines with smarter ones: How to extract meaningful data from machine learning classifiers,

    G. Ateniese, L. V . Mancini, A. Spognardi, A. Villani, D. Vitali, and G. Felici, “Hacking smart machines with smarter ones: How to extract meaningful data from machine learning classifiers,”International Journal of Security and Networks, vol. 10, no. 3, pp. 137–150, 2015

    2015

  21. [21]

    Artificial intelligence and machine learn- ing in clinical medicine, 2023,

    C. J. Haug and J. M. Drazen, “Artificial intelligence and machine learn- ing in clinical medicine, 2023,”New England Journal of Medicine, vol. 388, no. 13, pp. 1201–1208, 2023

    2023

  22. [22]

    Financial fraud detection through the application of machine learning techniques: a literature review,

    L. Hernandez Aros, L. X. Bustamante Molano, F. Gutierrez-Portela, J. J. Moreno Hernandez, and M. S. Rodr ´ıguez Barrero, “Financial fraud detection through the application of machine learning techniques: a literature review,”Humanities and Social Sciences Communications, vol. 11, no. 1, pp. 1–22, 2024

    2024

  23. [23]

    Learn- ing lightweight object detectors via multi-teacher progressive distilla- tion,

    S. Cao, M. Li, J. Hays, D. Ramanan, Y .-X. Wang, and L. Gui, “Learn- ing lightweight object detectors via multi-teacher progressive distilla- tion,” inInternational Conference on Machine Learning. PMLR, 2023, pp. 3577–3598

    2023

  24. [24]

    Antidistillation sampling,

    Y . Savani, A. Trockman, Z. Feng, A. Schwarzschild, A. Robey, M. Finzi, and J. Z. Kolter, “Antidistillation sampling,”arXiv preprint arXiv:2504.13146, 2025

    arXiv 2025

  25. [25]

    Defending against neural network model stealing attacks using deceptive perturbations,

    T. Lee, B. Edwards, I. Molloy, and D. Su, “Defending against neural network model stealing attacks using deceptive perturbations,” in2019 IEEE Security and Privacy Workshops (SPW). IEEE, 2019, pp. 43– 49

    2019

  26. [26]

    D-dae: Defense- penetrating model extraction attacks,

    Y . Chen, R. Guan, X. Gong, J. Dong, and M. Xue, “D-dae: Defense- penetrating model extraction attacks,” in2023 IEEE Symposium on Security and Privacy (SP). IEEE, 2023, pp. 382–399

    2023

  27. [27]

    On calibration of modern neural networks,

    C. Guo, G. Pleiss, Y . Sun, and K. Q. Weinberger, “On calibration of modern neural networks,” inInternational conference on machine learning. PMLR, 2017, pp. 1321–1330

    2017

  28. [28]

    Model compres- sion,

    C. Bucilu ˇa, R. Caruana, and A. Niculescu-Mizil, “Model compres- sion,” inProceedings of the 12th ACM SIGKDD international confer- ence on Knowledge discovery and data mining, 2006, pp. 535–541

    2006

  29. [29]

    Do deep nets really need to be deep?

    J. Ba and R. Caruana, “Do deep nets really need to be deep?”Advances in neural information processing systems, vol. 27, 2014

    2014

  30. [30]

    Born again neural networks,

    T. Furlanello, Z. Lipton, M. Tschannen, L. Itti, and A. Anandkumar, “Born again neural networks,” inInternational conference on machine learning. PMLR, 2018, pp. 1607–1616

    2018

  31. [31]

    Understanding and improving knowledge distillation,

    J. Tang, R. Shivanna, Z. Zhao, D. Lin, A. Singh, E. H. Chi, and S. Jain, “Understanding and improving knowledge distillation,”arXiv preprint arXiv:2002.03532, 2020

    Pith/arXiv arXiv 2002

  32. [32]

    Knockoff nets: Stealing functionality of black-box models,

    T. Orekondy, B. Schiele, and M. Fritz, “Knockoff nets: Stealing functionality of black-box models,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 4954–4963

    2019

  33. [33]

    Data-free model extraction,

    J.-B. Truong, P. Maini, R. J. Walls, and N. Papernot, “Data-free model extraction,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 4771–4780

    2021

  34. [34]

    Black-box behavioral distillation breaks safety alignment in medical llms,

    S. Jahan and R. Sun, “Black-box behavioral distillation breaks safety alignment in medical llms,”arXiv preprint arXiv:2512.09403, 2025

    arXiv 2025

  35. [35]

    I know what you trained last summer: A survey on stealing machine learning models and defences,

    D. Oliynyk, R. Mayer, and A. Rauber, “I know what you trained last summer: A survey on stealing machine learning models and defences,” ACM Computing Surveys, vol. 55, no. 14s, pp. 1–41, 2023

    2023

  36. [36]

    A comprehensive de- fense framework against model extraction attacks,

    W. Jiang, H. Li, G. Xu, T. Zhang, and R. Lu, “A comprehensive de- fense framework against model extraction attacks,”IEEE Transactions on Dependable and Secure Computing, vol. 21, no. 2, pp. 685–700, 2023

    2023

  37. [37]

    A survey on knowledge distillation of large language models,

    X. Xu, M. Li, C. Tao, T. Shen, R. Cheng, J. Li, C. Xu, D. Tao, and T. Zhou, “A survey on knowledge distillation of large language models,”arXiv preprint arXiv:2402.13116, 2024

    Pith/arXiv arXiv 2024

  38. [38]

    Undistillable: Making a nasty teacher that cannot teach students,

    H. Ma, T. Chen, T.-K. Hu, C. You, X. Xie, and Z. Wang, “Undistillable: Making a nasty teacher that cannot teach students,”arXiv preprint arXiv:2105.07381, 2021

    Pith/arXiv arXiv 2021

  39. [39]

    Adversarial sparse teacher: Defense against distillation-based model stealing attacks using adversarial examples,

    E. Yilmaz and H. Y . Keles, “Adversarial sparse teacher: Defense against distillation-based model stealing attacks using adversarial examples,”IEEE Access, 2025

    2025

  40. [40]

    Defending against model extraction attacks with ood feature learning and decision boundary confusion,

    C. Liang, J. Huang, Z. Zhang, and S. Zhang, “Defending against model extraction attacks with ood feature learning and decision boundary confusion,”Computers & Security, vol. 136, p. 103563, 2024

    2024

  41. [41]

    Misleader: Defending against model extraction with ensembles of distilled models,

    X. Cheng, M. Zheng, S. Zhu, and Y . Dong, “Misleader: Defending against model extraction with ensembles of distilled models,”arXiv preprint arXiv:2506.02362, 2025

    Pith/arXiv arXiv 2025

  42. [42]

    Prada: protecting against dnn model stealing attacks,

    M. Juuti, S. Szyller, S. Marchal, and N. Asokan, “Prada: protecting against dnn model stealing attacks,” in2019 IEEE European Sympo- sium on Security and Privacy (EuroS&P). IEEE, 2019, pp. 512–527. VOLUME , 17 Ullah et al.:

    2019

  43. [43]

    Misguide: Defense against data-free deep learning model extraction,

    M. Gurve, S. Behera, S. Ahlawat, and Y . Prasad, “Misguide: Defense against data-free deep learning model extraction,”arXiv preprint arXiv:2403.18580, 2024

    Pith/arXiv arXiv 2024

  44. [44]

    Radep: A resilient adaptive defense framework against model extraction attacks,

    A. Chakraborty, S. F. Ahamed, S. Roy, S. Banerjee, K. Choi, A. Rah- man, A. Hu, E. Bowen, and S. Shetty, “Radep: A resilient adaptive defense framework against model extraction attacks,” in2025 Inter- national Wireless Communications and Mobile Computing (IWCMC). IEEE, 2025, pp. 1241–1246

    2025

  45. [45]

    The Sybil attack,

    J. R. Douceur, “The Sybil attack,” inInternational Workshop on Peer- to-Peer Systems. Springer, 2002, pp. 251–260

    2002

  46. [46]

    Cloudleak: Large-scale deep learning models stealing through adversarial exam- ples

    H. Yu, K. Yang, T. Zhang, Y .-Y . Tsai, T.-Y . Ho, and Y . Jin, “Cloudleak: Large-scale deep learning models stealing through adversarial exam- ples.” inNDSS, vol. 38, 2020, p. 102

    2020

  47. [47]

    Embedding water- marks into deep neural networks,

    Y . Uchida, Y . Nagai, S. Sakazawa, and S. Satoh, “Embedding water- marks into deep neural networks,” inProceedings of the 2017 ACM on international conference on multimedia retrieval, 2017, pp. 269–277

    2017

  48. [48]

    Entangled watermarks as a defense against model extraction,

    H. Jia, C. A. Choquette-Choo, V . Chandrasekaran, and N. Papernot, “Entangled watermarks as a defense against model extraction,” in30th USENIX security symposium (USENIX Security 21), 2021, pp. 1937– 1954

    2021

  49. [49]

    Dawn: Dynamic adversarial watermarking of neural networks,

    S. Szyller, B. G. Atli, S. Marchal, and N. Asokan, “Dawn: Dynamic adversarial watermarking of neural networks,” inProceedings of the 29th ACM international conference on multimedia, 2021, pp. 4417– 4425

    2021

  50. [50]

    Defense against model extraction attack by bayesian active watermarking,

    Z. Wang, Y . Wu, and H. Huang, “Defense against model extraction attack by bayesian active watermarking,” inForty-first International Conference on Machine Learning, 2024

    2024

  51. [51]

    Prediction poisoning: To- wards defenses against dnn model stealing attacks,

    T. Orekondy, B. Schiele, and M. Fritz, “Prediction poisoning: To- wards defenses against dnn model stealing attacks,”arXiv preprint arXiv:1906.10908, 2019

    Pith/arXiv arXiv 1906

  52. [52]

    {ModelGuard}:{Information-Theoretic}defense against model extraction attacks,

    M. Tang, A. Dai, L. DiValentin, A. Ding, A. Hass, N. Z. Gong, Y . Chen et al., “{ModelGuard}:{Information-Theoretic}defense against model extraction attacks,” in33rd USENIX Security Symposium (USENIX Security 24), 2024, pp. 5305–5322

    2024

  53. [53]

    Efficient model stealing defense with noise transition matrix,

    D.-D. Wu, C. Fu, W. Wu, W. Xia, X. Zhang, J. Zhou, and M.-L. Zhang, “Efficient model stealing defense with noise transition matrix,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 24 305–24 315

    2024

  54. [54]

    Towards distillation-resistant large lan- guage models: An information-theoretic perspective,

    H. Fang, T. Zhang, T. Zhuang, J. Kong, K. Gao, B. Chen, L. Liang, S.-T. Xia, and K. Xu, “Towards distillation-resistant large lan- guage models: An information-theoretic perspective,”arXiv preprint arXiv:2602.03396, 2026

    Pith/arXiv arXiv 2026

  55. [55]

    Regu- larizing neural networks by penalizing confident output distributions,

    G. Pereyra, G. Tucker, J. Chorowski, Ł. Kaiser, and G. Hinton, “Regu- larizing neural networks by penalizing confident output distributions,” arXiv preprint arXiv:1701.06548, 2017

    Pith/arXiv arXiv 2017

  56. [56]

    Adaptive temperature scal- ing for robust calibration of deep neural networks,

    S. A. Balanya, J. Maronas, and D. Ramos, “Adaptive temperature scal- ing for robust calibration of deep neural networks,”Neural Computing and Applications, vol. 36, no. 14, pp. 8073–8095, 2024

    2024

  57. [57]

    Sample- dependent adaptive temperature scaling for improved calibration,

    T. Joy, F. Pinto, S.-N. Lim, P. H. Torr, and P. K. Dokania, “Sample- dependent adaptive temperature scaling for improved calibration,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 12, 2023, pp. 14 919–14 926

    2023

  58. [58]

    Attended temperature scaling: a practical approach for calibrating deep neural networks,

    A. S. Mozafari, H. S. Gomes, W. Le ˜ao, S. Janny, and C. Gagn ´e, “Attended temperature scaling: a practical approach for calibrating deep neural networks,”arXiv preprint arXiv:1810.11586, 2018

    Pith/arXiv arXiv 2018

  59. [59]

    Asymmetric temperature scaling makes larger networks teach well again,

    X.-C. Li, W.-S. Fan, S. Song, Y . Li, S. Yunfeng, D.-C. Zhanet al., “Asymmetric temperature scaling makes larger networks teach well again,”Advances in neural information processing systems, vol. 35, pp. 3830–3842, 2022

    2022

  60. [60]

    Research computing services,

    R. I. of Technology, “Research computing services,” 2026. [Online]. Available: https://www.rit.edu/researchcomputing/ Khawaja Abaid Ullahreceived the B.S. de- gree in Computer Science from the University of Narowal, Pakistan in 2021, and the M.S. degree in Artificial Intelligence from the Rochester Insti- tute of Technology (RIT), Rochester, NY , USA, in ...

    2026

  61. [2026]

    Available: https://www.anthropic.com/news/ detecting-and-preventing-distillation-attacks

    [Online]. Available: https://www.anthropic.com/news/ detecting-and-preventing-distillation-attacks