{"id":"2c26d2bc-d3ed-4e03-940f-3a30c3b68d9b","arxiv_id":"1908.00635","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A black-box Carlini-Wagner attack on CNN and LSTM modulation classifiers causes a reported 60% performance drop, demonstrating transferability of adversarial examples in radio signal classification.","lead":"This paper tests whether deep-learning radio signal classifiers can be fooled by a strong adversarial attack called Carlini-Wagner, using a substitute model to attack without knowing the target. It reports a 60% accuracy drop on both CNN and LSTM classifiers, with few experimental details.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 3 step 6 assumes transferability from the surrogate to CNN/LSTM targets without measuring it; if transfer fails, the claimed 60% black-box drop is unsupported.","rationale":"The central claim is that a black-box C-W attack, built via a substitute model, can degrade CNN/LSTM modulation classifiers by 60%. For that claim to hold, the adversarial examples crafted against the substitute must transfer to the target architectures. Section 3 step 6 invokes transferability as an established property, but no evidence is given for this particular source/target pair. In the image domain transferability is common but not universal; for radio-signal classifiers with different architectures and small substitute training sets, transfer rates can vary widely. The manuscript also omits baseline accuracies, attack hyperparameters, and per-SNR curves, so the reported 60% drop cannot be checked. None of this proves the claim false; it means the paper is conditional on a missing measurement. The proposed test directly measures the transfer gap and would settle whether the black-box procedure is responsible for the reported drop. I therefore keep the reader's CONDITIONAL verdict.","tokens_in":3163,"tokens_out":4660,"duration_ms":48105,"concrete_test":"Reproduce the attack on RML2016.10a with the stated protocol: train a fully connected surrogate on 10% of the target model's test queries, craft C-W adversarial examples against the surrogate, then score the exact same adversarial set on the target CNN and LSTM. Report four numbers for each target: clean accuracy, surrogate clean accuracy, surrogate accuracy under C-W, and target accuracy on the transferred adversarial set. The claim requires the last number to show approximately a 60% relative drop from the clean target accuracy. If the transfer gap (surrogate attack success minus target attack success) is large, or if the drop is concentrated only at low SNR where clean accuracy is already near chance, the black-box conclusion fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The decisive step in the proposed black-box procedure is Section 3, step 6: adversarial examples that fool the fully connected surrogate DNN 'are then transferred to black-box DL-based modulation classifier which according to the transferability property of adversarial examples will compromise the performance.' The paper provides no measurement of the transfer rate between the surrogate and the two target architectures. Transferability is an empirical property, not a theorem; it depends on architecture distance, training data, and input representation, and it is known to be imperfect even in vision. For modulation classifiers on I/Q samples, where the surrogate is trained on only 10% of the target's test queries, the transfer gap could be substantial. If most surrogate-fooling examples fail on the CNN or LSTM targets, the reported 60% drop would not be produced by the described black-box mechanism. The missing quantity is the per-target transfer gap: the accuracy of each target on the exact adversarial set that fools the surrogate, alongside the target's clean accuracy.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a black-box adversarial attack against deep learning based modulation classifiers. The authors use the RML2016.10a dataset, train a fully-connected surrogate DNN on 10% of the target model's test queries, craft adversarial examples against the surrogate using the Carlini-Wagner (C-W) attack, and then transfer those examples to CNN and LSTM based target classifiers, invoking the transferability property of adversarial examples. The paper claims a 60% performance drop on both target architectures and concludes that state-of-the-art deep learning modulation classifiers are not robust to adversarial attacks. The manuscript is presented as a short ACM-format paper with three figures and a brief experimental description.","tokens_in":3299,"tokens_out":2855,"duration_ms":29867,"significance":"If the central claim is substantiated, the paper would extend adversarial machine learning research to modulation classification in a black-box transfer setting, going beyond prior FGSM-based attacks with a stronger C-W attack. The step-by-step attack procedure is a reasonable template for future work, and the use of a standard public dataset (RML2016.10a) is a strength. However, the current evidence is far too thin: there are no reported baseline accuracies, no SNR-resolved accuracy curves, no error bars, no attack hyperparameters, and no measurement of the transfer rate between the surrogate and the target models. The significance of the paper is therefore conditional on substantially improved experimental reporting and validation.","major_comments":[{"comment":"The black-box attack relies entirely on the transferability of adversarial examples from the surrogate DNN to the CNN and LSTM target classifiers, yet the paper reports no measurement of the transfer rate. The manuscript states that adversarial examples compromising the surrogate 'are then transferred to black-box DL-based modulation classifier which according to the transferability property... will compromise the performance,' but no experiment verifies that a sufficient fraction of surrogate-fooling examples actually fool the targets. Without reporting the target clean accuracy and the target accuracy on the exact adversarial set that fools the surrogate, the claimed 60% drop cannot be attributed to the described black-box transfer mechanism.","section":"Section 3, step 6"},{"comment":"The results are presented through Figures 1 and 2 (and Figure 3 for LSTM), but the manuscript contains only figure captions and no actual plots. The reader cannot verify the claimed 60% performance drop, check SNR-dependent behavior, or see the pre-attack baseline accuracy. Additionally, Figure 1 is referenced both as a diagram of the attack procedure and as a performance plot, which is confusing. The authors must include the actual figures with clear axes, SNR ranges, and baseline curves.","section":"Section 3, Figures 1 and 2"},{"comment":"The C-W attack formulation in Eq. (2) depends on the cost constant c, the chosen norm p, and an iterative optimization procedure, but none of these are specified in the paper. Without reporting the value of c, the norm type (e.g., L2 or L∞), the number of optimization iterations, the learning rate, and whether the attack is targeted or untargeted, the attack is not reproducible and the reported results cannot be independently assessed.","section":"Section 3, Eq. (2)"},{"comment":"The surrogate DNN is described only as 'a fully connected DNN model,' with no information about the number of layers, neurons per layer, activation functions, training epochs, optimizer, or the exact split of the 10% query subset. The training procedure and the surrogate's own accuracy before the attack are essential for understanding the surrogate's quality and for interpreting the transferability results.","section":"Section 3, paragraph 3"},{"comment":"The conclusion states 'We have achieved a 60% performance drop in both CNN and LSTM based modulation classification,' but the body of the paper does not define the baseline against which this drop is measured. Is it the target's clean accuracy averaged over all SNRs, at a single SNR, or something else? Without this definition and without error bars or multiple runs, the headline number is not statistically meaningful.","section":"Conclusions"}],"minor_comments":[{"comment":"The abstract contains a grammatical subject-verb agreement error: 'the robustness of these modulation classifiers have not been evaluated' should be 'has not been evaluated.'","section":"Abstract"},{"comment":"The paper does not provide a quantitative comparison with the prior FGSM-based attacks of Sadeghi et al. and Kokalj-Filipovic et al., which would help the reader understand the additional value of the C-W attack in this domain.","section":"Related work"},{"comment":"Equation (1) is written as x* = x + arg min over η, but the optimization should be over the perturbation η that minimizes its norm subject to f(x + η) = t; the current notation is ambiguous about the objective and constraints.","section":"Section 1, Eq. (1)"},{"comment":"The paper uses the placeholder ACM format with 'Conference'17' and a dummy DOI, which is inappropriate for a journal submission; the authors should use the journal's formatting template.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"This manuscript reads as a very short workshop-style draft rather than a full journal paper. The central research question is relevant and the attack pipeline is sensible, but the experimental section is so underspecified that the headline result cannot be verified. I would only recommend acceptance after a substantial expansion that includes full figures, baselines, hyperparameters, transferability measurements, and statistical significance analysis. Given the journal context, the authors should also be asked to clarify the novelty claim relative to prior FGSM work."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know: this is a 3-page extended abstract that applies the Carlini-Wagner attack to modulation classification in a black-box setting and claims a 60% accuracy drop on both CNN and LSTM classifiers. The result is probably true, but the paper gives you almost nothing to check it against.\n\nWhat's genuinely new is narrow but real: prior work on adversarial attacks on modulation classification used FGSM, and this is the first in its own cited set to use C-W in a black-box transfer setting. The core procedure is standard and sound — query a surrogate, train on the queried labels, attack the surrogate, then transfer the adversarial examples. That is a legitimate extension to a new domain, and the authors are honest about the limited novelty. The citation pattern is fair; they properly cite the prior FGSM work and the C-W attack.\n\nThe soft spots are in the evidence. Section 3, step 6 invokes the transferability property without ever measuring the transfer rate from the fully connected surrogate to the CNN and LSTM targets. That is the load-bearing step: if transfer fails, the reported 60% drop is not produced by the described black-box mechanism. The text says the transfer \"according to the transferability property\" will compromise the target, but transferability is an empirical property, not a theorem, and it can be weak when the surrogate and target architectures differ substantially. The authors need to report clean accuracy, attacked accuracy, and the actual transfer rate for each target.\n\nThe experimental reporting is otherwise very thin. There are no baseline accuracies, no attack hyperparameters (C-W constant c, iterations, learning rate), no error bars, and the figure references are broken — the text points to \"Figure 1\" multiple times and then to \"Figures 1 and 2\" for results that are actually in Figures 2 and 3. The dataset (RML2016.10a) is a good choice, and the authors do say they used only 10% of test examples for the surrogate, but that is far from a complete configuration.\n\nWho is this for? People working on adversarial machine learning for radio signal classification who want a quick indicator that C-W also works in this domain. It is not a systematic study; it is a results flash. As a serious referee, I would want the authors to add a clean table of clean versus attacked accuracy per SNR, the attack configuration, and a direct measure of transferability. If they provide that, the claim becomes credible and useful. If they cannot, the empirical contribution is too thin.\n\nMy call: send it to peer review but expect major revision on the experimental reporting. The right venue is a workshop or short paper track; it does not have the depth for a top journal as is.","headline":"A plausible but underreported extension of the Carlini-Wagner attack to modulation classification; the claimed 60% drop is believable but currently unverifiable from the text.","tokens_in":3867,"tokens_out":1877,"would_cite":false,"duration_ms":19744,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that deep-learning modulation classifiers are not robust to adversarial examples, and demonstrates a black-box Carlini-Wagner attack that cuts CNN and LSTM classifier accuracy by about 60%.","keywords":["adversarial machine learning","modulation classification","black-box attack","Carlini-Wagner attack","transferability","deep learning","CNN","LSTM"],"falsifier":"Train a CNN and an LSTM modulation classifier on RML2016.10a; build a fresh surrogate, craft C-W adversarial examples against it, then replay them on the victims across the full SNR range. If the average accuracy drop is far below 60% or statistically insignificant for either architecture, the transferability assumption behind the black-box attack fails.","tokens_in":2915,"feed_emoji":"📡","tokens_out":4091,"duration_ms":38782,"temperature":0.7,"pith_summary":"The paper tries to establish that state-of-the-art deep-learning modulation classifiers, which identify radio signal types, can be broken by an adversary who has no access to the model weights, only to its input-output behavior. It does this by training a small surrogate DNN on query responses from the victim, crafting adversarial examples with the Carlini-Wagner attack, and transferring those examples to CNN and LSTM classifiers trained on the RML2016.10a radio dataset. The reported result is a 60% performance drop in both classifiers. This matters because modulation classification is a building block for cognitive and self-driving networks, so brittleness here implies a practical security risk.","feed_headline":"Black-box attack drops modulation classifier accuracy by 60%","feed_subtitle":"Surrogate-trained adversarial examples fool CNN and LSTM radio classifiers, showing deep-learning signal recognition is brittle.","key_machinery":"The mechanism is adversarial-example transferability together with the Carlini-Wagner optimization. The adversary first queries the black-box classifier with test inputs, records the labels, and trains a surrogate fully-connected DNN on about 10% of the test set. Adversarial perturbations are then computed against the surrogate by minimizing $\\|\\eta\\|_p + c \\cdot g(x^*)$ subject to the box constraint $x^* \\in [0,1]^n$, where $g(x^*)$ encodes the goal of wrong classification. The resulting examples, already misclassified by the surrogate, are replayed against the CNN and LSTM targets, relying on the property that adversarial examples often transfer across models. The dataset is the GNU Radio RML2016.10a collection of 11 analog and digital modulation schemes over SNR from -20 to 18 dB.","core_discovery":"The central claim is that adversarial examples transfer across deep-learning architectures in the radio-signal domain: perturbations crafted to fool a fully connected surrogate DNN also fool CNN and LSTM modulation classifiers treated as black boxes. The paper reports that after the transfer, both the CNN and LSTM classifiers suffer a 60% drop in classification performance, demonstrating that C-W attacks, previously studied mostly in computer vision, are effective against wireless signal classifiers. The authors present this as the first black-box Carlini-Wagner evaluation of such modulation classifiers.","pith_inferences":["The attack may extend to other radio-signal classifiers, such as emitter identification or spectrum sensing, if transferability is a general property of RF deep-learning models; this is an extrapolation, not demonstrated in the paper.","An over-the-air version would probably face additional distortions such as channel fading, noise, and receiver impairments, so the laboratory 60% drop is an upper bound unless the perturbations are physically robust.","A testable extension is to vary the surrogate architecture, dataset split, and SNR to map precisely when transferability breaks down, which would reveal whether the vulnerability is universal or depends on model similarity.","Defenders could exploit the query phase: since the adversary collects query-response pairs, monitoring for repeated query patterns may allow detection of this specific attack."],"forward_implications":["If the transferability result holds, an attacker needs only query access and a small surrogate to significantly degrade CNN- and LSTM-based modulation classifiers.","Deployed cognitive radios that rely on deep-learning modulation recognition would face a realistic black-box attack threat, not just a white-box theoretical one.","The 60% accuracy drop indicates that defenses such as adversarial training, input perturbation detection, or confidence-based rejection should be incorporated into RF deep-learning systems.","Because the surrogate used only 10% of the test examples, the attack appears inexpensive to mount, strengthening the case that robustness evaluation should become standard for wireless ML."],"supporting_citations":[{"why":"Supplies the Carlini-Wagner attack optimization used to craft the adversarial examples.","marker":"[1]"},{"why":"Introduces adversarial examples and the transferability property that the black-box attack relies on.","marker":"[3]"},{"why":"Provides the RML2016.10a radio dataset of 11 modulation schemes used to train the surrogate and target classifiers.","marker":"[6]"},{"why":"Previous FGSM-based adversarial attack on deep-learning radio signal classification that this paper extends by using C-W in a black-box setting.","marker":"[7]"}],"fun_headline_variants":["First C-W attack shows radio classifiers drop 60% under black-box assault","Surrogate-trained C-W attack slashes CNN and LSTM modulation accuracy by 60%","Black-box C-W attack transfers to radio CNNs and LSTMs, causing 60% drop","C-W attack transfers to CNN and LSTM radio classifiers, dropping accuracy 60%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole attack depends on the assumption that an input designed to fool a locally trained stand-in model will also fool a different, unseen target model; if that transfer does not happen, the reported performance drop would not occur.","fun_headline_variants_meta":{"raw":{"variants":["First C-W attack shows radio classifiers drop 60% under black-box assault","Surrogate-trained C-W attack slashes CNN and LSTM modulation accuracy by 60%","Black-box C-W attack transfers to radio CNNs and LSTMs, causing 60% drop","C-W attack transfers to CNN and LSTM radio classifiers, dropping accuracy 60%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000945,"raw_usage":{"total_tokens":3927,"prompt_tokens":731,"completion_tokens":3196,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":347,"completion_tokens_details":{"reasoning_tokens":3104}},"tokens_in":347,"tokens_out":3196,"duration_ms":23274,"temperature":1.0,"reasoning_tokens":3104,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T15:41:08.332315+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a CNN and an LSTM modulation classifier on RML2016.10a; build a fresh surrogate, craft C-W adversarial examples against it, then replay them on the victims across the full SNR range. If the average accuracy drop is far below 60% or statistically insignificant for either architecture, the transferability assumption behind the black-box attack fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Carlini-Wagner attack optimization used to craft the adversarial examples."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the RML2016.10a radio dataset of 11 modulation schemes used to train the surrogate and target classifiers."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Previous FGSM-based adversarial attack on deep-learning radio signal classification that this paper extends by using C-W in a black-box setting."}],"review_version":1}