{"id":"cad4390d-9807-453a-9d13-3f3fbd6fe2b7","arxiv_id":"2412.20529","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A CNN trained on AudioMNIST is shown to be vulnerable to FGSM, PGD, CW, and data poisoning attacks, while two off-the-shelf defense libraries fail to restore accuracy.","lead":"This paper tests four known adversarial attacks on a neural network trained to recognize spoken digits, and reports that all four reduce accuracy. It also reports that two existing defense libraries failed, but without enough detail to show whether the attacks and defenses were correctly applied.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The load-bearing claim is the failure of defenses, but the paper reports no actual defense method, no hyperparameters, and no accuracy after defense, so the conclusion is unsupported.","rationale":"I read the paper in good faith as a short report attempting to apply known adversarial attacks to an audio classifier and test two defense libraries. The attack numbers, if reproducible, are plausible but not novel. The defense claim is the part that would make the paper useful, and it is unsupported by the manuscript's own text: the authors report only that Art-IBM produced errors or failed to restore accuracy and that advertorch was 'most likely' incompatible. There is no named defense method, no accuracy table after defense, no comparison to a simple baseline such as adversarial training, and no code or configuration that would let a reader verify the failure. The reader's weakest assumption about attack implementations is legitimate, but it is secondary: even with perfect attack implementations, the central defense conclusion would still lack evidence. I therefore partly agree with the reader, but I identify a different load-bearing gap. I also note the PGD parameter description is internally inconsistent: the text first fixes Eps=0.2 and then says Eps is looped from 0.05 to 1, so the reported PGD accuracy of 0.0 at eps=0.95 is not reproducible from the given information. The proposed test would settle the defense question directly by running one standard defense method and reporting before/after accuracies under the same attack settings. No theatrical judgment is needed; the manuscript simply does not provide the data required to support its conclusion.","tokens_in":3015,"tokens_out":2807,"duration_ms":31104,"concrete_test":"Run a controlled defense experiment on the same AudioMNIST CNN: apply PGD adversarial training (or Art-IBM's AdversarialTrainer with a PyTorch wrapper) using the paper's reported attack parameters, then evaluate the defended model against FGSM, PGD, and CW at the reported eps values. Report test accuracy before and after each attack. If accuracy returns near the clean baseline, the paper's conclusion that defenses fail is wrong. If the libraries cannot run on mel-spectrogram inputs, document the exact class/function called, the input format, and the full error traceback instead of a summary, so the failure can be attributed to library incompatibility rather than to the defense method.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's title, abstract, and conclusion present two central claims: that standard attacks reduce accuracy and that ready-made defense libraries fail on audio classifiers. The first claim, even if fully correct, is a routine application of known attacks. The second claim is the novel and load-bearing one, and the manuscript does not actually support it. In the Defense section, the authors write that Art-IBM 'led to errors in the work of protective methods, or did not help restore the accuracy rate close to the original,' and that advertorch methods are aimed at images and 'most likely led to the failure of their application in this case.' These are not experimental results: no Art-IBM or advertorch defense method is named, no hyperparameters are given, no accuracy numbers before and after a defense are reported, and no baseline defense is run for comparison. The conclusion that 'low accuracy of the model or incompatibility in its current configuration' was obtained is therefore based on an unspecified collection of errors and likelihoods, not on measurements. Even if all attack implementations are correct, a correctly configured defense could restore accuracy, which would invalidate the paper's main takeaway. This is more load-bearing than the reader's implementation-matching concern, because the attack results only replicate prior work, whereas the defense claim is the potential contribution. Additionally, the PGD paragraph is internally ambiguous: it states Eps=0.2 as a fixed parameter but then says Eps starts at 0.05 and is incremented in a loop, so the reported PGD result is not well defined. The defense conclusion should be rejected unless a controlled defense experiment is reported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes a convolutional neural network trained on the AudioMNIST dataset to classify spoken digits, then applies four adversarial attacks: FGSM, PGD, CW, and data poisoning. The authors report test-set accuracies of 0.17, 0.0, 0.36, and 0.17 after these attacks, respectively. They also report attempts to defend the model using the ART-IBM and advertorch libraries, concluding that ready-made defense methods failed or were incompatible with their audio-based model. The attack portion is a routine application of well-known methods; the defense portion is the paper's potential novelty.","tokens_in":3291,"tokens_out":5172,"duration_ms":50506,"significance":"If substantiated, the claim that existing adversarial-defense libraries fail on audio classifiers would be a useful negative result for the adversarial-robustness community. The paper's strengths are its use of a public dataset (AudioMNIST), the selection of standard attacks, and the disclosure of several attack hyperparameters. However, the defense claim—the main contribution—is not supported by any reported experiment: no defense method is named, no hyperparameters are given, no accuracy before or after defense is reported, and no baseline defense is run for comparison. The attack results, while plausible, also lack error bars, repeated runs, and code, making them difficult to verify. As it stands, the manuscript is a brief report of routine attack applications with an unsupported conclusion about defenses.","major_comments":[{"comment":"The central claim that ART-IBM and advertorch defenses failed is not backed by measurements. The text states that ART-IBM 'led to errors in the work of protective methods, or did not help restore the accuracy rate close to the original' and that advertorch methods are 'aimed at datasets containing images' and 'most likely led to the failure of their application in this case.' No defense method is named, no hyperparameters are given, no accuracy numbers before/after defense are reported, and no non-defended baseline is provided. Because the title, abstract, and conclusion foreground the defense-failure claim, these qualitative statements cannot support the paper's main takeaway. Please provide a table listing each defense method tried, its configuration, the observed accuracy or error condition, and a comparison with the no-defense baseline; alternatively, explicitly reframe the paper as an exploratory report rather than a demonstration of defense failure.","section":"Defense section"},{"comment":"The PGD parameter description is internally inconsistent. The text first lists 'Eps = 0.2 (maximum distortion)' as a fixed parameter, but immediately afterward says 'the initial value of Eps is 0.05, incremented by 0.05 in the loop and re-applied against the model' and attributes the lowest accuracy (0) to eps equal to 0.95. Clarify whether the 0.2 value was a starting point, whether the sweep over eps was performed, and at which eps value the reported accuracy of 0.0 occurred. Without this clarification, the PGD result is not reproducible.","section":"Attacks: PGD paragraph"},{"comment":"The sentence 'When considering various values of the parameters of this method, no noticeable changes in the drop in accuracy were found' is unsupported because no range of parameter values or resulting accuracies is given. The only reported number is a single accuracy of 0.36. Please report the configurations actually tried (e.g., different learning rates, iteration counts, or norms) and the accuracy obtained for each, so the reader can judge both the attack result and the claimed insensitivity.","section":"Attacks: CW paragraph"},{"comment":"The data-poisoning description is not reproducible: the noise tensor is said to be 'generated randomly from a uniform distribution,' but the bounds of that distribution are not specified. Without this information, the poisoning experiment cannot be repeated or compared with other poisoning studies. Additionally, the separate 'SPT' recognition experiment is unrelated to the reported accuracy metric and should either be removed or integrated with the main evaluation, and the acronym 'SPT' is never defined.","section":"Attacks: data poisoning"},{"comment":"Accuracy metrics are reported as single numbers without error bars, repeated runs, or standard deviations, and no code or model weights are provided. Since the reported attack accuracies depend on the exact implementations of FGSM, PGD, CW, and the poisoning procedure, and since only partial hyperparameters are given (e.g., CW lacks a norm specification and the FGSM sweep is only coarsely described), the reader cannot verify that the implementations match the cited algorithms. Please include code or, failing that, a complete hyperparameter and random-seed table, and report variability over multiple runs.","section":"General evaluation methodology"}],"minor_comments":[{"comment":"The sentence 'The aim of this paper is to consider the application of FGSM, PGD, CW and data poisoning attacks within the framework of using a neural network trained on audio data' is grammatically awkward; consider revising for precision and clarity.","section":"Abstract and opening"},{"comment":"The column headers 'Kernel Shift' and 'Indent' are nonstandard and ambiguous; they presumably denote stride and padding. Please use the standard terminology so the architecture is unambiguous.","section":"Table 1"},{"comment":"The text says 'the RELU activation function' and 'BatchNorm2d' without defining them; use proper notation (ReLU, batch normalization) and briefly explain the purpose of these components.","section":"Neural network description"},{"comment":"Reference [3] is the Szegedy et al. paper on intriguing properties of neural networks, but the CW attack was introduced by Carlini and Wagner; the manuscript should cite the correct reference for the CW attack. References [4] and [5] are about weight poisoning, which is not the additive-noise poisoning described here; a citation for standard data-poisoning attacks would be more appropriate.","section":"References"},{"comment":"The conclusion says 'there is a suspicion that it will be necessary to use manually written attack methods, or change the framework'; this phrasing is purely speculative and should be removed or replaced with concrete findings from the reported experiments.","section":"Conclusion"}],"recommendation":"reject","confidential_remarks":"The manuscript is very short and its central contribution—the defense-failure claim—is entirely unsupported by experimental data; the authors themselves qualify the defense results with 'mostly likely' and 'suspicion.' The attack results, even if valid, are routine applications of known methods. In my view, acceptance would require substantial new experimental work (proper defense evaluations, reproducible attack settings, code/weights), which goes well beyond a standard minor or major revision. If the venue publishes exploratory negative-result notes, a major revision could be considered, but for a regular journal archival paper the current content is not sufficient."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a routine application of FGSM, PGD, CW, and data poisoning to AudioMNIST. The attack numbers are plausible but underreported, and the paper's only non-routine claim — that ready-made defense libraries fail on audio classifiers — is not backed by any measured result.\n\nCredit where it is due: the paper is short and honest about what was done. The architecture and training settings are specified, the split is stated, and the note that a poisoned sample is indistinguishable by ear while the recognizer still classifies it correctly is a small, genuine observation. If the attack accuracies are reproducible, they confirm known behavior on a new dataset.\n\nThe soft spots are significant. The defense section is the load-bearing part of the paper, and it contains no experiment. No defense method is named, no hyperparameters are given, and no accuracy before and after a defense is reported. Phrases like \"led to errors in the work of protective methods\" and \"most likely led to the failure\" are not results. The stress-test note is right: the conclusion that the defenses fail is unsupported. The PGD paragraph is internally inconsistent — it lists Eps=0.2 as fixed, then describes a loop where eps starts at 0.05 and increases — so the reported PGD accuracy is not well defined. The CW attack is cited to Szegedy et al. rather than Carlini and Wagner, which is a citation error. The accuracy numbers are single values with no error bars or repeated runs, and reporting only the lowest accuracy from an epsilon sweep without the curve is cherry-picking. The data-poisoning noise distribution is not specified. No code is provided.\n\nThis reads like a student project write-up rather than a research contribution. There is no new attack, no new defense, no systematic comparison to prior audio adversarial work, and the one potentially interesting conclusion is unsupported. I would not bring it to a reading group and would not cite it. A serious referee would send it back for the missing defense experiment and code; as submitted, there is little to engage with. Recommend desk reject.","headline":"A routine attack-on-audio report whose only non-routine claim, that off-the-shelf defenses fail, is not supported by any actual defense experiment.","tokens_in":3848,"tokens_out":1861,"would_cite":false,"duration_ms":19945,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Standard adversarial attacks and data poisoning reduce an audio classifier's test accuracy from 0.99 to as low as 0.0, and ready-made defense libraries did not restore it.","keywords":["adversarial attacks","FGSM","PGD","CW attack","data poisoning","audio classification","Mel-spectrogram","neural network defense"],"falsifier":"Re-implement the described architecture and attacks, train on AudioMNIST, and compare the final test accuracies with the reported $0.17$, $0.0$, $0.36$, and $0.17$; a mismatch would show the vulnerability is implementation-specific rather than a property of the methods.","tokens_in":2837,"feed_emoji":"🎙️","tokens_out":12122,"duration_ms":107373,"temperature":0.7,"pith_summary":"This paper tries to establish that adversarial attacks developed for image classifiers also break a neural network trained on audio, and that ready-made defense libraries do not yet repair the damage. The authors train a small convolutional network on AudioMNIST, a collection of 30,000 recordings of spoken digits, and report 99 percent accuracy on training, validation, and test sets. Applying FGSM, PGD, CW, and data poisoning, they report test accuracies of $0.17$, $0.0$, $0.36$, and $0.17$, respectively. They also report that two existing defense libraries failed to restore accuracy, with one tied to a different deep-learning framework and the other oriented to image benchmarks, and conclude that audio-specific defenses are still missing. In concrete terms, the paper claims that a near-perfect audio classifier can be rendered useless by small spectrogram perturbations or poisoned training data.","feed_headline":"PGD attack drives audio-network accuracy to zero","feed_subtitle":"All four attacks break the 99%-accurate audio classifier, and ready-made defenses fail to restore it.","key_machinery":"The mechanism that carries the experiment is a four-layer convolutional network that turns 48 kHz audio into 64-filter Mel-spectrograms, applies shifts and zero-padding as augmentation, and classifies each spectrogram into one of ten digits. The attack machinery is the standard set from image adversarial examples: FGSM takes one step along the sign of the loss gradient, PGD takes several smaller clipped steps, CW iteratively minimizes a distance norm while forcing the most-confident wrong class to dominate, and data poisoning adds random uniform noise to the signals before training and testing. All of these produce perturbations in the spectrogram input space, and the reported accuracy drops are the observable result.","core_discovery":"The central claim is a set of controlled measurements on one audio classification pipeline. A convolutional network with four convolutional layers, batch normalization, ReLU activations, adaptive pooling, and a fully connected layer reaches 99 percent accuracy on Mel-spectrograms of spoken digits. Under the FGSM attack the test accuracy falls to $0.17$ at $\\epsilon = 0.95$; under PGD, a repeated small-step version of FGSM, it falls to $0.0$ at the same $\\epsilon$; the CW attack leaves accuracy at $0.36$; and adding random noise drawn from a uniform distribution to the training and test signals, the paper's data-poisoning recipe, leaves accuracy at $0.17$. The paper further reports that protection attempts with two ready-made libraries did not recover accuracy, attributing the failure to framework incompatibility and to the libraries being designed for image datasets rather than audio input. The conclusion the paper draws is that existing defenses are not portable to this audio setting.","pith_inferences":["Beyond the paper, the reported zero under PGD suggests testing the full epsilon sweep for each attack; a cliff-shaped accuracy drop would mean the model fails at a much smaller perturbation than the final value suggests.","Beyond the paper, the failed defense attempts do not rule out adversarial training or input denoising on the same architecture, so those are the natural next experiments the numbers invite.","Beyond the paper, repeating the attacks on raw waveforms or on other audio features such as MFCCs would separate 'audio models are fragile' from 'Mel-spectrograms are a fragile input representation.'","Beyond the paper, an audio-specific benchmark reporting accuracy-versus-perturbation curves for standard attacks would let future defense methods be compared on equal footing."],"forward_implications":["A high clean-data accuracy on an audio task does not imply robustness; a small perturbation in spectrogram space can drive accuracy to zero.","Defense libraries built for one deep-learning framework or for image benchmarks cannot be assumed to transfer to a PyTorch-based audio pipeline.","Data poisoning can be as damaging as gradient-based attacks in this setting, so the integrity of training data matters as much as input-time defenses.","Audio classifiers should be evaluated under adversarial conditions, not only on clean test sets, if their real-world deployment is to be trusted."],"supporting_citations":[{"why":"Provides the AudioMNIST dataset of 30,000 spoken-digit wav files that the model is trained and tested on.","marker":"[1]"},{"why":"Source of the FGSM gradient-sign attack and its epsilon-based distortion procedure.","marker":"[2]"},{"why":"Cited as the basis of the CW attack, the norm-minimizing perturbation that forces a confident wrong class.","marker":"[3]"},{"why":"Supply the data-poisoning method, including a weight-poisoning approach and a prior biometric-authentication poisoning study.","marker":"[4-5]"}],"fun_headline_variants":["PGD zeroes audio classifier accuracy","99% audio accuracy falls to 0% under PGD","Audio AI: attacks drop accuracy, defenses fail","FGSM, PGD, CW, poisoning break audio net","Ready-made defenses can't save audio neural nets"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported numbers depend on the authors' implementations of FGSM, PGD, CW, and data poisoning matching the standard algorithms, since no code is released and only partial hyperparameters are given.","fun_headline_variants_meta":{"raw":{"variants":["PGD zeroes audio classifier accuracy","99% audio accuracy falls to 0% under PGD","Audio AI: attacks drop accuracy, defenses fail","FGSM, PGD, CW, poisoning break audio net","Ready-made defenses can't save audio neural nets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000818,"raw_usage":{"total_tokens":3503,"prompt_tokens":790,"completion_tokens":2713,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":406,"completion_tokens_details":{"reasoning_tokens":2651}},"tokens_in":406,"tokens_out":2713,"duration_ms":18733,"temperature":1.0,"reasoning_tokens":2651,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:18:06.084197+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-implement the described architecture and attacks, train on AudioMNIST, and compare the final test accuracies with the reported $0.17$, $0.0$, $0.36$, and $0.17$; a mismatch would show the vulnerability is implementation-specific rather than a property of the methods.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the AudioMNIST dataset of 30,000 spoken-digit wav files that the model is trained and tested on."},{"cited_title":"Goodfellow","cited_arxiv_id":null,"evidence_quote":"Source of the FGSM gradient-sign attack and its epsilon-based distortion procedure."},{"cited_title":"Goodfellow, Rob Fergus // 2nd Intern ational Conference on Learning Representa- tions, ICLR 2014","cited_arxiv_id":null,"evidence_quote":"Cited as the basis of the CW attack, the norm-minimizing perturbation that forces a confident wrong class."}],"review_version":1}