REVIEW 5 major objections 5 minor 6 references
Attacks on the neural network and defense methods
T0 review · 5 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read Standard adversarial attacks and data poisoning reduce an audio classifier's test accuracy from 0.99 to as low as 0.0, and ready-made defense libraries did not restore it.
desk verdict A routine attack-on-audio report whose only non-routine claim, that off-the-shelf defenses fail, is not supported by any actual defense experiment. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism that carries the experiment is a four-layer convolutional network that turns 48 kHz audio into 64-filter Mel-spectrograms, applies shifts and zero-padding as augmentation, and classifies each spectrogram into one of ten digits. The attack machinery is the standard set from image adversarial examples: FGSM takes one step along the sign of the loss gradient, PGD takes several smaller clipped steps, CW iteratively minimizes a distance norm while forcing the most-confident wrong class to dominate, and data poisoning adds random uniform noise to the signals before training and testing. All of these produce perturbations in the spectrogram input space, and the reported accuracy drops are the observable result.
What would settle it
Re-implement the described architecture and attacks, train on AudioMNIST, and compare the final test accuracies with the reported $0.17$, $0.0$, $0.36$, and $0.17$; a mismatch would show the vulnerability is implementation-specific rather than a property of the methods.
Extended reading notes
Core claim
The central claim is a set of controlled measurements on one audio classification pipeline. A convolutional network with four convolutional layers, batch normalization, ReLU activations, adaptive pooling, and a fully connected layer reaches 99 percent accuracy on Mel-spectrograms of spoken digits. Under the FGSM attack the test accuracy falls to $0.17$ at $\epsilon = 0.95$; under PGD, a repeated small-step version of FGSM, it falls to $0.0$ at the same $\epsilon$; the CW attack leaves accuracy at $0.36$; and adding random noise drawn from a uniform distribution to the training and test signals, the paper's data-poisoning recipe, leaves accuracy at $0.17$. The paper further reports that protection attempts with two ready-made libraries did not recover accuracy, attributing the failure to framework incompatibility and to the libraries being designed for image datasets rather than audio input. The conclusion the paper draws is that existing defenses are not portable to this audio setting.
Load-bearing premise
The reported numbers depend on the authors' implementations of FGSM, PGD, CW, and data poisoning matching the standard algorithms, since no code is released and only partial hyperparameters are given.
Editorial extensions
If this is right
- A high clean-data accuracy on an audio task does not imply robustness; a small perturbation in spectrogram space can drive accuracy to zero.
- Defense libraries built for one deep-learning framework or for image benchmarks cannot be assumed to transfer to a PyTorch-based audio pipeline.
- Data poisoning can be as damaging as gradient-based attacks in this setting, so the integrity of training data matters as much as input-time defenses.
- Audio classifiers should be evaluated under adversarial conditions, not only on clean test sets, if their real-world deployment is to be trusted.
Reading between the lines
- Beyond the paper, the reported zero under PGD suggests testing the full epsilon sweep for each attack; a cliff-shaped accuracy drop would mean the model fails at a much smaller perturbation than the final value suggests.
- Beyond the paper, the failed defense attempts do not rule out adversarial training or input denoising on the same architecture, so those are the natural next experiments the numbers invite.
- Beyond the paper, repeating the attacks on raw waveforms or on other audio features such as MFCCs would separate 'audio models are fragile' from 'Mel-spectrograms are a fragile input representation.'
- Beyond the paper, an audio-specific benchmark reporting accuracy-versus-perturbation curves for standard attacks would let future defense methods be compared on equal footing.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper describes a convolutional neural network trained on the AudioMNIST dataset to classify spoken digits, then applies four adversarial attacks: FGSM, PGD, CW, and data poisoning. The authors report test-set accuracies of 0.17, 0.0, 0.36, and 0.17 after these attacks, respectively. They also report attempts to defend the model using the ART-IBM and advertorch libraries, concluding that ready-made defense methods failed or were incompatible with their audio-based model. The attack portion is a routine application of well-known methods; the defense portion is the paper's potential novelty.
Significance. If substantiated, the claim that existing adversarial-defense libraries fail on audio classifiers would be a useful negative result for the adversarial-robustness community. The paper's strengths are its use of a public dataset (AudioMNIST), the selection of standard attacks, and the disclosure of several attack hyperparameters. However, the defense claim—the main contribution—is not supported by any reported experiment: no defense method is named, no hyperparameters are given, no accuracy before or after defense is reported, and no baseline defense is run for comparison. The attack results, while plausible, also lack error bars, repeated runs, and code, making them difficult to verify. As it stands, the manuscript is a brief report of routine attack applications with an unsupported conclusion about defenses.
major comments (5)
- [Defense section] The central claim that ART-IBM and advertorch defenses failed is not backed by measurements. The text states that ART-IBM 'led to errors in the work of protective methods, or did not help restore the accuracy rate close to the original' and that advertorch methods are 'aimed at datasets containing images' and 'most likely led to the failure of their application in this case.' No defense method is named, no hyperparameters are given, no accuracy numbers before/after defense are reported, and no non-defended baseline is provided. Because the title, abstract, and conclusion foreground the defense-failure claim, these qualitative statements cannot support the paper's main takeaway. Please provide a table listing each defense method tried, its configuration, the observed accuracy or error condition, and a comparison with the no-defense baseline; alternatively, explicitly reframe the paper as an exploratory report rather than a demonstration of defense failure.
- [Attacks: PGD paragraph] The PGD parameter description is internally inconsistent. The text first lists 'Eps = 0.2 (maximum distortion)' as a fixed parameter, but immediately afterward says 'the initial value of Eps is 0.05, incremented by 0.05 in the loop and re-applied against the model' and attributes the lowest accuracy (0) to eps equal to 0.95. Clarify whether the 0.2 value was a starting point, whether the sweep over eps was performed, and at which eps value the reported accuracy of 0.0 occurred. Without this clarification, the PGD result is not reproducible.
- [Attacks: CW paragraph] The sentence 'When considering various values of the parameters of this method, no noticeable changes in the drop in accuracy were found' is unsupported because no range of parameter values or resulting accuracies is given. The only reported number is a single accuracy of 0.36. Please report the configurations actually tried (e.g., different learning rates, iteration counts, or norms) and the accuracy obtained for each, so the reader can judge both the attack result and the claimed insensitivity.
- [Attacks: data poisoning] The data-poisoning description is not reproducible: the noise tensor is said to be 'generated randomly from a uniform distribution,' but the bounds of that distribution are not specified. Without this information, the poisoning experiment cannot be repeated or compared with other poisoning studies. Additionally, the separate 'SPT' recognition experiment is unrelated to the reported accuracy metric and should either be removed or integrated with the main evaluation, and the acronym 'SPT' is never defined.
- [General evaluation methodology] Accuracy metrics are reported as single numbers without error bars, repeated runs, or standard deviations, and no code or model weights are provided. Since the reported attack accuracies depend on the exact implementations of FGSM, PGD, CW, and the poisoning procedure, and since only partial hyperparameters are given (e.g., CW lacks a norm specification and the FGSM sweep is only coarsely described), the reader cannot verify that the implementations match the cited algorithms. Please include code or, failing that, a complete hyperparameter and random-seed table, and report variability over multiple runs.
minor comments (5)
- [Abstract and opening] The sentence 'The aim of this paper is to consider the application of FGSM, PGD, CW and data poisoning attacks within the framework of using a neural network trained on audio data' is grammatically awkward; consider revising for precision and clarity.
- [Table 1] The column headers 'Kernel Shift' and 'Indent' are nonstandard and ambiguous; they presumably denote stride and padding. Please use the standard terminology so the architecture is unambiguous.
- [Neural network description] The text says 'the RELU activation function' and 'BatchNorm2d' without defining them; use proper notation (ReLU, batch normalization) and briefly explain the purpose of these components.
- [References] Reference [3] is the Szegedy et al. paper on intriguing properties of neural networks, but the CW attack was introduced by Carlini and Wagner; the manuscript should cite the correct reference for the CW attack. References [4] and [5] are about weight poisoning, which is not the additive-noise poisoning described here; a citation for standard data-poisoning attacks would be more appropriate.
- [Conclusion] The conclusion says 'there is a suspicion that it will be necessary to use manually written attack methods, or change the framework'; this phrasing is purely speculative and should be removed or replaced with concrete findings from the reported experiments.
Circularity Check
No significant circularity: the paper reports empirical attack measurements; unsupported defense conclusions are an evidence problem, not a circular derivation.
full rationale
The manuscript contains no derivation chain that reduces to its own inputs. It trains a convolutional neural network on AudioMNIST and applies standard external attack methods (FGSM, PGD, CW, and data poisoning), reporting measured test accuracies (0.17, 0.0, 0.36, and 0.17, respectively). No parameter is fitted to a target accuracy and then renamed as a prediction; no claimed result is defined in terms of another claimed result; no uniqueness theorem or ansatz is imported from prior work to force a conclusion. The only self-citation is reference [5], which includes a coauthor, but it is cited for the general concept of a poisoning attack and is not load-bearing for the reported accuracy numbers. The defense section is indeed weakly supported: no Art-IBM or advertorch defense method is named, no hyperparameters are given, and no before/after accuracy is reported. However, that is a completeness and evidence problem, not circularity, because the paper does not claim to derive the defense outcome from the attack measurements by construction. The conclusion that defenses were ineffective is an unsupported empirical claim, not a circular one. The honest non-finding is therefore appropriate: no significant circularity, score 0.
Assumptions & free parameters
free parameters (4)
- FGSM epsilon sweep =
0.05 to 1.0, step 0.05; lowest accuracy at 0.95
- PGD epsilon and iteration settings =
eps=0.2, eps_iter=0.1, nb_iter=5; sweep up to 1.0
- CW learning rate and max iterations =
lr=0.01, max_iterations=200
- Poisoning noise distribution =
uniform, bounds not specified
assumptions (3)
- domain assumption AudioMNIST classification accuracy is a sufficient measure of model robustness.
- domain assumption The implementations of FGSM, PGD, CW, and data poisoning follow the cited papers.
- domain assumption Trying Art-IBM and advertorch in the authors' configurations is a representative test of defense methods.
Cite this review
Pith. "Pith review of Attacks on the neural network and defense methods." pith.science (2026). https://pith.science/paper/7NG2L5WF
@misc{pith2026241220529,
author = {Pith},
title = {Pith review of: Attacks on the neural network and defense methods},
year = {2026},
howpublished = {\url{https://pith.science/paper/7NG2L5WF}},
note = {Machine review of arXiv:2412.20529}
}
read the original abstract
This article will discuss the use of attacks on a neural network trained on audio data, as well as possible methods of protection against these attacks. FGSM, PGD and CW attacks, as well as data poisoning, will be considered. Within the framework of protection, Art-IBM and advertorch libraries will be considered. The obtained accuracy metrics within the framework of attack applications are presented
Reference graph
Works this paper leans on
-
[1]
Audio MNIST Dataset https://www.kaggle.com/datasets/alanchn31/free- spoken-digits
-
[2]
Ian J. Goodfellow. Explaining and Harnessing Adversarial Example / Ian J. Goodfellow, Jonathon Shlens, Chri stian Szegedy // 3rd International Confer- ence on Learning Repr esentations, ICLR 2015. – San Diego, CA, USA, 2015. – p. 1-10
work page 2015
-
[3]
Goodfellow, Rob Fergus // 2nd Intern ational Conference on Learning Representa- tions, ICLR 2014
Intriguing properties of neural networks / Chris- tian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, Rob Fergus // 2nd Intern ational Conference on Learning Representa- tions, ICLR 2014. – Banff, AB, Canada, 2014, - p. 1-10
work page 2014
-
[4]
– URL: https://arxiv.org/abs/2004.06660, free (last accessed : 10.09.2024)
Weight Poisoning Attacks on Pre-trained Models [Electronic resource ]. – URL: https://arxiv.org/abs/2004.06660, free (last accessed : 10.09.2024)
arXiv 2004
-
[5]
Investigation of errors arising from a kit poison- ing attack on a biometric authentication system when changing the poisoning percentage on different input data / Y. Usolcev, B. Lodonova, A. Konovalov , Y . Kostuchenko // Collection of selected articles of the scientific session of TUSUR. – 2020. – № 1-2. – P. 101-
work page 2020
-
[104]
_____________________________________________
(in Russian). _____________________________________________
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.