Pith. sign in

REVIEW 3 major objections 4 minor 29 references

Evaluating Defensive Distillation For Defending Text Processing Neural Networks Against Adversarial Examples

T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Defensive distillation, a defense that works for images, does not protect text classifiers from adversarial examples.

desk verdict A useful first test of defensive distillation for text, undone by a non-canonical training objective that makes the negative result unsupported. read the letter →

arxiv 1908.07899 v1 pith:37ZELTWW submitted 2019-08-21 cs.CL cs.CRcs.LGcs.NE

Marcus Soll , Tobias Hinz , Sven Magg , Stefan Wermter This is my paper · ORCID
classification cs.CLcs.CRcs.LGcs.NE
keywords adversarialexamplesdefensivedistillationtextclassificationconvolutionalneuralnetworkstransferabilityrobustnessgradientmaskingwordsubstitutionattack
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Adversarial examples—small, human-invisible edits that flip a classifier's prediction—threaten real text applications such as spam filtering and automatic indexing, and attacks built for one network often transfer to others. This paper asks whether defensive distillation, a training procedure that successfully hardened image classifiers by training a student network on a teacher's softened outputs, can do the same for text classification. On two datasets and several distillation temperatures, the answer is no: a gradient-guided word-substitution attack still succeeds on 96-98% of samples, and examples crafted on an undistilled network still fool the distilled one about as often as the baseline. The conclusion is that defensive distillation neither increases robustness against this class of adversarial text examples nor blocks their transferability.

What carries the argument

The central machinery is defensive distillation itself: a training procedure in which a teacher network's softened probability outputs at temperature $T$ are used as soft labels to train an equally sized student network, with the temperature reset to $T=1$ after training. In the image domain this flattens the gradient landscape and blocks some gradient-based attacks. The counter-machinery is the text attack: a word-substitution algorithm in which the gradient is used only to order words by importance, and edits come from a candidate pool of synonyms, typos, and class-specific keywords matched by part of speech. The mismatch between the defense's gradient-masking mechanism and the attack's rank-only use of gradients is what, in the paper's account, explains the negative result.

What would settle it

Run the same distillation setup against a text attack that uses the gradient more directly, such as iteratively perturbing word embeddings in the gradient direction and mapping back to the nearest words, on AG news articles and Amazon movie reviews. If the distilled network's attack success drops well below the undistilled baseline (for example below 20%) while the undistilled network stays near 98%, the claim that distillation does not increase text robustness would be refuted.

Watch

Extended reading notes

Core claim

Defensive distillation hardens a neural network by first training a teacher at temperature $T$, then training an equally sized student on the teacher's soft labels at the same temperature and resetting to $T=1$ at test time. The authors apply this recipe to a single-layer convolutional text classifier with multiple kernel sizes and word embeddings, on AG news articles and Amazon movie reviews, using the temperature $T=20$ from the image-domain work plus $T=10,30,40$. Adversarial examples are generated with a gradient-guided word-substitution algorithm that uses the gradient only to rank which words matter, then deletes, replaces, or inserts words drawn from synonyms, typos, and class-specific keywords. The attack succeeds on 96-98% of test samples for both distilled and undistilled networks, with distillation raising only the average number of word changes required; transferability of examples from an undistilled network to a distilled one drops by at most about 0.05, far from preventing transfer. The paper concludes that, at least for these datasets and this attack, defensive distillation is not an effective defense for text classification.

Load-bearing premise

The entire negative conclusion rests on the assumption that the one attack tested—a rank-based word-substitution attack—adequately represents the adversarial threats to text classifiers, so a defense that blocks a different, stronger attack would make the paper's broad claim false.

Editorial extensions

If this is right

  • Defensive distillation cannot be adopted as a standalone defense for text classifiers against word-substitution attacks of this type.
  • Adversarial text examples remain transferable from undistilled to distilled networks, so black-box attacks that craft examples on a public model are not stopped by distillation.
  • Raising the distillation temperature from 10 to 40 does not meaningfully change attack success, so the defense has no obvious tuning sweet spot on these datasets.
  • Because the attack uses gradients only for word ranking, gradient-masking defenses more broadly are unlikely to help text classifiers unless they change the ranking itself.
  • The small increase in the number of edits needed suggests distillation slightly perturbs the loss landscape, but not enough to protect predictions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A fairer test of the defense would be an attack that uses the gradient's full direction or magnitude, such as perturbing continuous word embeddings; if such an attack is also unimpeded, the negative conclusion strengthens, and if it is blocked, the paper's result is specific to rank-based attacks.
  • The study uses one convolutional architecture and two datasets; distillation might behave differently on recurrent or transformer text models, where the loss surface has different geometry. This is an untested extrapolation.
  • The persistence of transferability suggests that defenses for text should target the learned decision boundary rather than the gradient, with adversarial training using diverse attacks a more direct candidate—though the paper only reviews that option and does not test it.
  • For system builders, the practical takeaway is not to rely on distillation to sanitize text classifiers; defenses should be evaluated under the specific word-level edit budget and attack type relevant to the application.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper presents an empirical evaluation of defensive distillation for convolutional neural network text classifiers. Using a version of the Samanta and Mehta attack, the authors generate adversarial examples for networks trained with and without distillation at temperatures 10, 20, 30, and 40 on the AG News and Amazon movie review datasets. They measure attack success rate (around 96-98% in all conditions), the number of word changes required (slightly higher with distillation), and the transferability of examples from an undefended network to distilled networks (a small drop of 0.03-0.05). The authors conclude that defensive distillation has minimal impact on text classification networks and does not prevent transferability.

Significance. If the conclusions were established for canonical defensive distillation, this would be a valuable negative result: it would show that a prominent image-domain defense does not transfer to text, and that black-box transfer attacks remain viable. The paper has strengths: the code is publicly available, the experiments cover two datasets and four temperatures, and the transferability evaluation is directly relevant to black-box threat models. However, the central claim is undermined by the fact that the implemented distillation uses a 10% hard-label loss, which is not the canonical defensive distillation of Papernot et al.; the significance is therefore conditional on either re-running with pure soft labels or substantially qualifying the conclusion.

major comments (3)
  1. [Section 3.5 and Section 4] The paper states in Section 3.5 that it uses the variant described by Papernot et al. [18], but Section 4 specifies that training uses a loss composed of 10% hard-label and 90% soft-label loss. Canonical defensive distillation, as introduced by Papernot et al. [18], trains the distilled network exclusively on soft labels produced at temperature T; the robustness benefit is generally attributed to this exclusive soft-label training, which smooths the output distribution. Adding a hard-label component reintroduces pressure toward confident one-hot predictions and may restore the steep gradients that make adversarial crafting easy. Consequently, the attack success rates in Table 2 may reflect a weakened variant rather than defensive distillation itself, and the abstract's unconditional claim about defensive distillation is not supported. At minimum, the claim must be reworded to refer to the tested mixed-objective variant, or the experiments must be repeated with a pure soft-label student.
  2. [Section 3.4 and Section 6] The attack used in all experiments is a single algorithm, the Samanta and Mehta variant described in Section 3.4, which uses the gradient only to rank word importance and then searches over a hand-built candidate pool. As the authors themselves note in Section 6, this means that the gradient masking effect of defensive distillation may not be exercised. The conclusion that 'defensive distillation does not work for text classification' is therefore tied to this attack class; a different attack that uses gradients directly, or an optimization-based attack, could behave differently. The paper should either test at least one additional attack type or explicitly limit the conclusion to the tested attack.
  3. [Tables 2 and 4] All reported numbers come from a single training run per condition; no error bars, confidence intervals, or multiple-seed statistics are provided. The key effects are small: attack success rates are 96-98% across all conditions, and transferability differences in Table 4 are between 0.032 and 0.051. Without variance estimates, the claim that distillation has 'only a minimal impact' is not statistically grounded, and the small observed differences may not be meaningful. Reporting multiple seeds or bootstrap intervals is needed to support the central negative claim.
minor comments (4)
  1. [Abstract] The phrase 'does neither help with increasing their robustness' should be revised to 'does not help to increase their robustness'.
  2. [Figure 3(c)] The caption contains a duplicated phrase: 'through the through the insertion' should read 'through the insertion'.
  3. [Figure 4] The y-axis label 'Number of Examples' could be clarified as 'Number of adversarial examples' to avoid ambiguity.
  4. [Section 3.5] The softmax equation would be clearer with explicit exponentials, e.g., exp(l_i/T), and with the summation index shown explicitly, to avoid the current inline notation that is easy to misread.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is an empirical measurement study with no fitted prediction or derivation that reduces to its inputs.

full rationale

This paper does not contain a derivation chain that could be circular. It implements an existing text attack algorithm (Samanta and Mehta) and an existing defense (defensive distillation, following Papernot et al.), runs both on two datasets, and reports the measured success rates, perturbation counts, and transferability rates. No parameter is fitted to the evaluation data and then renamed as a prediction; no equation in the paper defines the measured outcome in terms of the defense or attack implementation; and the conclusion that defensive distillation has minimal effect is a direct empirical observation, not a consequence of an assumed premise. The skeptical concern that the distilled network was trained with a 10% hard-label loss component rather than soft labels only is a substantive threat to the generality of the conclusion, but it is not a circularity: the authors explicitly report their training setup, and the result is a contingent measurement of that setup. Likewise, the choice of a single attack algorithm and the paper's own discussion that the gradient is used only for word importance ranking are validity limitations, not instances of the conclusion being assumed in the inputs. No self-citations appear, and no uniqueness theorem or imported ansatz is relied upon. The paper is self-contained against external benchmarks and the findings stand or fall on the reported experiments, so the appropriate circularity score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper is an empirical evaluation rather than a derivation, so there are no fitted constants or invented entities. The central claim rests on representativeness assumptions about the attack algorithm, the network architecture, the distillation implementation, and the word embeddings, listed above.

assumptions (4)
  • domain assumption The Samanta-Mehta attack algorithm, as adapted here, is a valid and representative method for generating adversarial text examples.
    The paper uses only this attack to evaluate robustness; if the attack is weak or unrepresentative, the negative result may not generalize (Section 4).
  • domain assumption The CNN architecture (Kim-style with kernel sizes 3, 4, 5) is representative of text classification networks.
    Results are shown for one architecture; defensive distillation might behave differently for RNNs or transformers (Section 3.3).
  • domain assumption The implementation of defensive distillation, trained on a weighted combination of hard and soft labels with a 10%/90% loss split, faithfully represents the defense proposed by Papernot et al.
    The paper cites Papernot et al. and Hinton et al., but the specific loss weighting is a design choice that could differ from the original method; a mismatch could affect the negative result (Section 3.5).
  • domain assumption Word2vec embeddings from the Google News corpus provide a fair input representation for both datasets.
    The attack modifies words to other words in the candidate pool, relying on the embedding space and the vocabulary coverage (Section 3.2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evaluating Defensive Distillation For Defending Text Processing Neural Networks Against Adversarial Examples." pith.science (2026). https://pith.science/paper/37ZELTWW

@misc{pith2026190807899,
  author       = {Pith},
  title        = {Pith review of: Evaluating Defensive Distillation For Defending Text Processing Neural Networks Against Adversarial Examples},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/37ZELTWW}},
  note         = {Machine review of arXiv:1908.07899}
}
read the original abstract

Adversarial examples are artificially modified input samples which lead to misclassifications, while not being detectable by humans. These adversarial examples are a challenge for many tasks such as image and text classification, especially as research shows that many adversarial examples are transferable between different classifiers. In this work, we evaluate the performance of a popular defensive strategy for adversarial examples called defensive distillation, which can be successful in hardening neural networks against adversarial examples in the image domain. However, instead of applying defensive distillation to networks for image classification, we examine, for the first time, its performance on text classification tasks and also evaluate its effect on the transferability of adversarial text examples. Our results indicate that defensive distillation only has a minimal impact on text classifying neural networks and does neither help with increasing their robustness against adversarial examples nor prevent the transferability of adversarial examples between neural networks.

Figures

Figures reproduced from arXiv: 1908.07899 by the authors.

Figure 1
Figure 1. Single-layer convolutional neural network model with different kernel [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Generated adversarial examples on the AG dataset. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Generated adversarial examples on the Amazon movie dataset. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Distribution of changes for generated adversarial examples for networks [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

29 extracted references · 14 canonical work pages

  1. [18]

    In: IEEE Symposium on Security and Privacy

    Papernot, N., McDaniel, P., Wu, X., Jha, S., Swami, A.: Distillation as a defense to adversarial perturbations against deep neural networks. In: IEEE Symposium on Security and Privacy. pp. 582–597 (2016). DOI: 10.1109/SP.2016.41

    doi:10.1109/sp.2016.41 2016

  2. [1]

    AG’s corpus of news articles, http://www.di.unipi.it/~gulli/AG_corpus_of_ news_articles.html, accessed online on the 27th of October 2017

    work page 2017

  3. [2]

    IEEE Access 6, 14410–14430 (2018)

    Akhtar, N., Mian, A.: Threat of adversarial attacks on deep learning in com- puter vision: A survey. IEEE Access 6, 14410–14430 (2018). DOI: 10.1109/AC- CESS.2018.2807385

    arXiv 2018

  4. [3]

    CoRR abs/1704.01547 (2017)

    Brendel, W., Bethge, M.: Comment on ”biologically inspired protection of deep networks from adversarial attacks”. CoRR abs/1704.01547 (2017)

    arXiv 2017

  5. [4]

    CoRR abs/1607.04311 (2016)

    Carlini, N., Wagner, D.A.: Defensive distillation is not robust to adversarial exam- ples. CoRR abs/1607.04311 (2016)

    arXiv 2016

  6. [5]

    (ed.): WordNet: An Electronic Lexical Database

    Fellbaum, C. (ed.): WordNet: An Electronic Lexical Database. MIT Press, Cam- bridge, MA (1998)

    work page 1998

  7. [6]

    In: International Conference on Learning Representations (2015)

    Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples. In: International Conference on Learning Representations (2015)

    work page 2015

  8. [7]

    In: NIPS Deep Learning Workshop (2014)

    Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. In: NIPS Deep Learning Workshop (2014)

    work page 2014

Show all 29 references
  1. [8]

    In: Proceedings of the Conference on Empirical Methods in Natural Language Processing

    Jia, R., Liang, P.: Adversarial examples for evaluating reading comprehension sys- tems. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing. pp. 2021–2031 (2017). DOI: 10.18653/v1/D17-1215

    2017 doi

  2. [9]

    In: Proceedings of the Conference on Empirical Methods in Natural Language Processing

    Kim, Y.: Convolutional neural networks for sentence classification. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing. pp. 1746–1751 (2014)

    2014

  3. [10]

    http://snap.stanford.edu/data (2014)

    Leskovec, J., Krevl, A.: SNAP Datasets: Stanford large network dataset collection. http://snap.stanford.edu/data (2014)

    2014

  4. [11]

    In: Proceedings of the International Joint Conference on Artificial Intelli- gence

    Liang, B., Li, H., Su, M., Bian, P., Li, X., Shi, W.: Deep text classification can be fooled. In: Proceedings of the International Joint Conference on Artificial Intelli- gence. pp. 4208–4215 (2018). DOI: 10.24963/ijcai.2018/585

    2018 doi

  5. [12]

    In: IEEE International Conference on Computer Vision

    Lu, J., Issaranon, T., Forsyth, D.: Safetynet: Detecting and rejecting adversarial examples robustly. In: IEEE International Conference on Computer Vision. pp. 446–454 (2017). DOI: 10.1109/ICCV.2017.56 12 M. Soll, T. Hinz, S. Magg, S. Wermter

    2017 doi

  6. [13]

    In: Proceedings of the International Con- ference on World Wide Web

    McAuley, J.J., Leskovec, J.: From amateurs to connoisseurs: modeling the evolution of user expertise through online reviews. In: Proceedings of the International Con- ference on World Wide Web. pp. 897–908 (2013). DOI: 10.1145/2488388.2488466

    2013

  7. [14]

    In: International Conference on Learning Representations (2013)

    Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word represen- tations in vector space. In: International Conference on Learning Representations (2013)

    2013

  8. [15]

    In: Advances in Neural Information Processing Systems, pp

    Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed repre- sentations of words and phrases and their compositionality. In: Advances in Neural Information Processing Systems, pp. 3111–3119 (2013)

    2013

  9. [16]

    Mitton, R.: Corpora of misspellings for download, http://www.dcs.bbk.ac.uk/ ~ROGER/corpora.html, accessed online on the 10th of November 2017

    2017

  10. [17]

    CoRR abs/1703.09202 (2017)

    Nayebi, A., Ganguli, S.: Biologically inspired protection of deep networks from adversarial attacks. CoRR abs/1703.09202 (2017)

    2017 arXiv

  11. [19]

    CoRR abs/1705.05264 (2017)

    Papernot, N., McDaniel, P.D.: Extending defensive distillation. CoRR abs/1705.05264 (2017)

    2017 arXiv

  12. [20]

    CoRR abs/1611.03814 (2016)

    Papernot, N., McDaniel, P.D., Sinha, A., Wellman, M.P.: Towards the science of security and privacy in machine learning. CoRR abs/1611.03814 (2016)

    2016 arXiv

  13. [21]

    In: IEEE Winter Conference on Applications of Computer Vision

    Rozsa, A., Gunther, M., Boult, T.E.: Towards robust deep neural networks with bang. In: IEEE Winter Conference on Applications of Computer Vision. pp. 803– 811 (2018). DOI: 10.1109/WACV.2018.00093

    2018

  14. [22]

    CoRR abs/1707.02812 (2017)

    Samanta, S., Mehta, S.: Towards crafting text adversarial samples. CoRR abs/1707.02812 (2017)

    2017 arXiv

  15. [23]

    IEEE Transactions on Evolutionary Computation (2019)

    Su, J., Vargas, D.V., Sakurai, K.: One pixel attack for fooling deep neu- ral networks. IEEE Transactions on Evolutionary Computation (2019). DOI: 10.1109/TEVC.2019.2890858

    2019

  16. [24]

    In: International Conference on Learning Representations (2014)

    Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., Fergus, R.: Intriguing properties of neural networks. In: International Conference on Learning Representations (2014)

    2014

  17. [25]

    In: International Conference on Learning Representations (2018)

    Tram` er, F., Kurakin, A., Papernot, N., Boneh, D., McDaniel, P.: Ensemble ad- versarial training: Attacks and defenses. In: International Conference on Learning Representations (2018)

    2018

  18. [26]

    CoRR abs/1704.03453 (2017)

    Tram` er, F., Papernot, N., Goodfellow, I., Boneh, D., McDaniel, P.: The space of transferable adversarial examples. CoRR abs/1704.03453 (2017)

    2017 arXiv

  19. [27]

    CoRR abs/1901.06796 (2019)

    Zhang, W.E., Sheng, Q.Z., Alhazmi, A.A.F., Li, C.: Generating textual adversarial examples for deep learning models: A survey. CoRR abs/1901.06796 (2019)

    2019 arXiv

  20. [28]

    In: Advances in Neural Information Processing Systems, pp

    Zhang, X., Zhao, J., LeCun, Y.: Character-level convolutional networks for text classification. In: Advances in Neural Information Processing Systems, pp. 649–657 (2015)

    2015

  21. [29]

    In: Proceedings of the Inter- national Joint Conference on Natural Language Processing

    Zhang, Y., Wallace, B.: A sensitivity analysis of (and practitioners’ guide to) con- volutional neural networks for sentence classification. In: Proceedings of the Inter- national Joint Conference on Natural Language Processing. pp. 253–263 (2017)

    2017

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.