{"id":"1c4a29c4-7358-4e18-a28d-7d647d94fd17","arxiv_id":"2507.13727","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Output-space adversarial training improved clean-data performance and adversarial robustness of two bird sound classifiers across seven soundscape test sets, and stabilized prototype-based explanations.","lead":"Bird sound classifiers trained on curated recordings often fail on noisy field recordings. This paper shows adversarial training, adding small worst-case perturbations during training, improved average test accuracy by up to 10.5 percent relative across seven soundscape datasets and hardened models against attacks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"AT-O recipe bundles FGSM, AWP weight perturbation, and weight randomization; without ablations, the clean-performance gain cannot be attributed to adversarial input perturbations.","rationale":"The reader's weakest assumption identifies exactly the load-bearing confound. The central claim is causal: adversarial input perturbations, not just the surrounding TRADES-AWP machinery, improve generalization under distribution shifts. Since AWP and random weight perturbation are known to flatten loss landscapes and can boost clean accuracy, the Table 2 gains could arise without any worst-case input perturbation. The missing control with random perturbations is therefore the decisive experiment. The absence of error bars and code further supports a conditional rather than unconditional acceptance, but they are secondary to the attribution problem. The paper is not internally inconsistent; the reported numbers are plausible, and the robustness comparisons (Tables 3-5) are less affected by this confound because all AT variants share the same AWP machinery and AT-O still wins. This is why I keep the reader's CONDITIONAL verdict rather than moving to reject: the empirical result may survive, but the interpretation requires an additional control experiment.","tokens_in":31807,"tokens_out":7828,"duration_ms":89432,"concrete_test":"Run an AudioProtoPNet ablation on POW and held-out PER and HSN, five seeds each, reporting mean cmAP with per-seed SD: (a) OT; (b) full AT-O as in the paper (FGSM, AWP, weight randomization, epsilon=0.1); (c) the full recipe with the FGSM direction replaced by a fixed random sign vector of the same infinity-norm magnitude, redrawn once per epoch; (d) FGSM training with AWP and weight randomization disabled (TRADES-only). If (c) matches (b), the adversarial gradient direction is not the active ingredient; if (d) matches (b), the clean-data gain is attributable to the input perturbation itself. Either outcome settles whether the paper's causal attribution is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical claim is that output-space adversarial training (AT-O) improves clean-data cmAP from 0.38 to 0.42 on average (Table 2, AudioProtoPNet), and the abstract/causal framing attributes this to worst-case input perturbations forcing invariant features. However, Section 2.2 (Eq. 9) and Section 3.2 define AT-O as a single recipe bundling three ingredients: the FGSM input perturbation, AWP layer-wise weight perturbation (gamma=0.005, 8-epoch warm-up), and weight randomization (1e-4) during perturbation generation. The only experimental contrasts are OT versus AT-E versus AT-O; all three ingredients vary together between OT and either AT variant. There is no ablation with AWP and weight randomization but no input perturbation, and no control with random or fixed-direction perturbations of the same magnitude. If the clean-data gain comes from AWP or weight randomization—both established loss-landscape-flattening regularizers—the headline attribution 'adversarial training ... using output-space attacks improves clean test data performance' is unsupported, even though the measured numbers would stand. The AT-E versus AT-O gap shows the attack objective affects the ranking of the two AT variants, but it does not identify which ingredient is responsible for the OT-to-AT gain. The Discussion's open question of why AT-O confers broader gains than AT-E reinforces that the mechanism is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper adapts the TRADES-AWP adversarial training framework to multi-label bird sound classification and compares ordinary training (OT), embedding-space adversarial training (AT-E), and output-space adversarial training (AT-O) on two architectures, ConvNeXt and AudioProtoPNet. On seven BirdSet soundscape test sets, the authors report that AT-O improves mean clean-data cmAP from 0.38 to 0.42 for AudioProtoPNet and to 0.41 for ConvNeXt, while also improving PRS against untargeted PGD attacks and TARS for targeted embedding-space attacks on prototypes. The paper interprets these results as evidence that adversarial training, particularly with output-space attacks, improves generalization under distribution shifts while preserving prototype stability.","tokens_in":32082,"tokens_out":6067,"duration_ms":68429,"significance":"If the reported effects are causal, the paper would provide practically useful evidence that adversarial training can serve as an augmentation strategy in bioacoustics, where distribution shifts are severe, and that it can stabilize prototype-based explanations. The evaluation is grounded in an external benchmark (BirdSet), spans seven geographically diverse test sets, covers two architectures, and uses five seeds per configuration. The PRS/DRS/TARS metrics are restated from prior work, and the core measurements appear internally consistent: AT-O improves most datasets and both architectures, with only UHH regressing marginally for ConvNeXt. The main gap is that the training recipe is not ablated, so the attribution of the gains to adversarial input perturbations is not established.","major_comments":[{"comment":"The central attribution of the clean-data gains to adversarial input perturbations is not supported by the experimental design. The AT-O recipe bundles the TRADES consistency term, the AWP weight perturbation (gamma=0.005, warm-up 8 epochs), and weight randomization (1e-4) together with the FGSM input perturbation, and the OT/AT-E/AT-O comparisons vary all of these ingredients at once. A control with AWP and weight randomization but no input perturbation, or with random/fixed-direction input perturbations of the same magnitude, is needed to identify which ingredient drives the improvements. If such a control reproduces the clean-data gains, the headline claim that output-space adversarial training improves generalization is misattributed, even though the measured numbers in Table 2 would stand. The paper's own Discussion leaves 'why AT-O confers broader performance enhancements than AT-E' as an open question, which further indicates that the mechanism is not established by the presented experiments.","section":"Section 2.2, Eq. (9); Section 3.2"},{"comment":"The paper states that training was repeated five times with different seeds, but all tables report single point estimates without variance, confidence intervals, or per-seed breakdowns. As a result, small observed differences such as the ConvNeXt AT-E mean cmAP improvement from 0.38 to 0.39 in Table 2 and the UHH decrease from 0.24 to 0.23 cannot be distinguished from seed-to-seed noise. Given that the average improvements of 2.6-10.5% are the quantitative basis of C2, please report mean and standard deviation (or equivalent) for at least the main performance and robustness tables, and state whether the AT-O over OT ordering is consistent across seeds.","section":"Section 3.2; Tables 2, 3, and 5"}],"minor_comments":[{"comment":"The epsilon=0.1 setting was selected on the POW validation set, yet POW is displayed in Table 2 alongside the seven test sets; since POW was used for hyperparameter selection, its column is not an independent estimate, and the table should label it as validation or omit it from the headline mean.","section":"Section 4.1; Table 2"},{"comment":"The objective in Eq. (9) shows a maximization over the weight perturbation v but does not explicitly show the maximization over the input perturbation delta; the surrounding text describes this, but the equation should be completed for clarity.","section":"Section 2.2, Eq. (9)"},{"comment":"The cosine-similarity expressions use a dot product notation without bold vector symbols for z(hz,wz), which makes the local spatial vectors hard to distinguish from tensors; please introduce explicit vector notation.","section":"Equations (4) and (7)"},{"comment":"The 'weight randomization with a strength of 1e-4' is not formalized; please specify the distribution and whether the randomization is applied only during adversarial input generation or also during the outer minimization.","section":"Section 3.2"},{"comment":"Reference [12] is cited as an arXiv preprint without a version identifier or access date; since BirdSet is the backbone of the experimental protocol, a citable version with a DOI would be preferable.","section":"References"},{"comment":"The UMAP visualizations are qualitative; please state the number of instances projected and whether all test segments are used, and consider adding a quantitative cluster-separation measure to support the visual claims.","section":"Figures 3 and 5"}],"recommendation":"major_revision","confidential_remarks":"The empirical measurements are likely to survive a re-run, but the paper's title and abstract make a causal claim about adversarial perturbations that the current experimental design cannot isolate. The requested ablations are within the scope of the manuscript; absent them, the contribution is mainly a benchmark application rather than an explanation of why adversarial training helps under distribution shifts. The paper's self-citations to [71], [33], and [12] are appropriate given that the metrics and benchmark come from those works."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid empirical paper with one real gap. It shows that a specific TRADES-AWP recipe with output-space FGSM perturbations (AT-O) improves held-out soundscape cmAP for two architectures on six of seven BirdSet test sets, and that it also improves adversarial robustness and prototype stability. That is a useful, practical result for bioacoustics and for anyone doing multi-label audio classification under distribution shift. The systematic comparison of output-space vs embedding-space adversarial training in this setting is new, and the prototype-stability evaluation under targeted embedding attacks is a nice addition.\n\nWhat the paper does well: the evaluation is thorough. Seven soundscape datasets, two architectures, multiple robustness metrics, and the epsilon ablation on POW guides the choice of perturbation strength. The numbers are internally consistent, and the gains on the hardest dataset (PER) are substantial. The authors fully restate the PRS/DRS/TARS metrics from their own prior work in Section 3.4, so the paper is self-contained. They also acknowledge limitations, including the slight UHH regression and the bird-sound-specific scope.\n\nThe soft spot: the attribution. AT-O is not just FGSM. Equation 9 bundles the TRADES consistency loss, AWP weight perturbation (gamma=0.005, 8-epoch warm-up), and weight randomization (1e-4) during perturbation generation. The only contrasts are OT vs AT-E vs AT-O; all three ingredients vary together. There is no ablation with AWP but no input perturbation, or with random or fixed-direction perturbations of the same magnitude. So the claim that adversarial input perturbations are what improves clean generalization is not supported. The measured numbers stand—AT-O as a recipe works—but the mechanism is unidentified. That matters for the abstract's causal framing. A second, minor issue: no error bars or variance across the five seeds, and the best-validation checkpoint per config is reported. That makes it hard to know if the 0.38-to-0.42 difference is significant. No code is released yet, which also limits verification.\n\nNone of this is fatal. The result is useful even without a clean mechanism, and the missing ablations are straightforward to run. This paper deserves a serious referee, and the review should request code and ablations before acceptance.","headline":"Solid empirical result on adversarial training for bioacoustics, but the recipe's active ingredient is unidentified—worth refereeing with requests for ablations and code.","tokens_in":32616,"tokens_out":1782,"would_cite":true,"duration_ms":20727,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adversarial training, aimed at attacks, also raises clean accuracy under real-world audio distribution shifts.","keywords":["adversarial training","bird sound classification","bioacoustics","distribution shift","multi-label audio classification","prototype-based models","adversarial robustness","generalization"],"falsifier":"Train the same two architectures with random input noise of the same magnitude in place of FGSM perturbations, keeping the TRADES-AWP weight perturbation and weight randomization identical; if the cmAP gains over ordinary training persist or shrink only slightly, then the paper's attribution of the gains to adversarial input perturbation is not supported.","tokens_in":31564,"feed_emoji":"🐦","tokens_out":6277,"duration_ms":65742,"temperature":0.7,"pith_summary":"This paper tries to establish that adversarial training, a technique usually aimed at defending against malicious input perturbations, also improves ordinary generalization when a model faces the large distribution shifts typical of real-world audio. Using bird sound classification as a test bed, the authors compare ordinary training with two adversarial variants on a convolutional network (ConvNeXt) and a prototype-based model (AudioProtoPNet). They report that training with single-step output-space FGSM perturbations under the TRADES-AWP recipe raises the average class-mean average precision on seven held-out soundscape datasets from 0.38 to 0.42 for AudioProtoPNet and to 0.41 for ConvNeXt. The same training also strengthens resistance to untargeted and targeted attacks and stabilizes the learned prototypes. If correct, this means adversarial training can serve as a data-augmentation strategy for bioacoustic classification without architectural changes.","feed_headline":"Adversarial training lifts bird-song accuracy on shifted data by 10.5%","feed_subtitle":"Output-space attacks raise average cmAP from 0.38 to 0.42 and harden models against perturbations.","key_machinery":"The load-bearing mechanism is the TRADES-AWP training objective, a minimax loss in which the model is trained against worst-case input perturbations and adversarial weight perturbations. The objective combines a classification loss on clean inputs, a TRADES consistency term penalizing divergence between outputs on clean and perturbed inputs, and Adversarial Weight Perturbation that flattens the loss landscape; the authors adapt it to multi-label bird sound classification with an asymmetric loss. During training the input perturbation is generated with single-step FGSM attacks, either output-space attacks that maximize the classification loss or embedding-space attacks that maximize average spatial cosine distance between embeddings, with weight randomization applied during perturbation generation. The central comparison is between these two variants (AT-O and AT-E) and ordinary training, evaluated by class-mean average precision and by robustness scores that separate performance degradation from embedding deformation.","core_discovery":"On the paper's own terms, the central discovery is that output-space adversarial training (AT-O) improves clean-data performance under distribution shifts while simultaneously improving adversarial robustness. In the ablation on the POW validation set, AT-O raises ConvNeXt cmAP from 0.47 to 0.56 and AudioProtoPNet from 0.49 to 0.59 at perturbation strength $\\epsilon=0.1$. Across seven soundscape test datasets the mean cmAP rises from 0.38 to 0.41 for ConvNeXt and from 0.38 to 0.42 for AudioProtoPNet, a relative gain of up to 10.5 percent. For robustness, AT-O lifts AudioProtoPNet's Performance Robustness Score from 0.74 to 0.89 against untargeted embedding-space attacks at $\\epsilon=0.01$ and from 0.02 to 0.56 at $\\epsilon=0.05$, and raises the Total Adversarial Robustness Score against targeted prototype attacks from 0.27 to 0.65 at $\\epsilon=0.1$. The authors conclude that the accuracy-robustness trade-off is not necessarily induced by adversarial training in this setting.","pith_inferences":["The paper attributes the clean-data gain to the worst-case input perturbation, but its AT recipe also contains layer-wise weight perturbation, TRADES consistency, and weight randomization; a control experiment with random or fixed perturbations of equal magnitude would show whether the input perturbation is the active ingredient.","If the active mechanism is a flatter loss landscape rather than the specific adversarial objective, similar clean-data gains should appear on other audio tasks with strong domain shift, such as acoustic scene classification or industrial anomaly detection, whenever baselines are low.","Embedding-space adversarial training, despite smaller clean-data gains here, may be most useful for retrieval-oriented models and self-supervised pretraining, where embedding integrity rather than classification accuracy is the primary goal.","A natural extension is to test AT-O against natural corruptions such as reverberation and device mismatch, since the robustness it confers appears to be against worst-case bounded perturbations rather than those specific distortions."],"forward_implications":["Output-space adversarial training raises average clean-data cmAP from 0.38 to 0.42 for AudioProtoPNet and to 0.41 for ConvNeXt across seven soundscape test sets.","The largest clean-data gains appear where the distribution shift is strongest: on the PER dataset, AT-O improves cmAP by about 28 percent for AudioProtoPNet and 24 percent for ConvNeXt.","AT-O improves robustness to untargeted embedding-space attacks, with AudioProtoPNet's PRS rising from 0.74 to 0.89 at $\\epsilon=0.01$ and from 0.02 to 0.56 at $\\epsilon=0.05$.","AT-O stabilizes AudioProtoPNet prototypes against targeted embedding attacks, raising TARS from 0.27 with ordinary training to 0.65 at $\\epsilon=0.1$.","These benefits come without architectural modifications or extra inference cost, so adversarial training can be added to existing audio classifiers."],"supporting_citations":[{"why":"Supplies the TRADES-AWP minimax objective that the paper adapts for multi-label adversarial training.","marker":"[37]"},{"why":"Provides the TRADES consistency term that penalizes divergence between clean and perturbed outputs.","marker":"[36]"},{"why":"Introduces FGSM, the single-step attack used to generate training perturbations.","marker":"[35]"},{"why":"Supplies the weight randomization technique and the perturbation magnitude used during adversarial input generation.","marker":"[38]"},{"why":"Defines the AudioProtoPNet architecture and the spectrogram preprocessing used in all experiments.","marker":"[33]"},{"why":"Provides the BirdSet benchmark, training protocol, and the seven soundscape test datasets with their distribution shifts.","marker":"[12]"},{"why":"Supplies the asymmetric loss that adapts the TRADES-AWP objective to multi-label classification.","marker":"[82]"},{"why":"Defines the PRS, DRS, and TARS metrics used to measure adversarial robustness and prototype stability.","marker":"[71]"},{"why":"Establishes class-mean average precision as the primary performance metric for multi-label audio classification.","marker":"[34]"}],"fun_headline_variants":["Adversarial training hardens bird audio models against shifts and attacks","10.5% accuracy gain on shifted bird sounds via adversarial training","Adversarial training beats distribution shifts and attacks in bird audio","Output-space attacks improve bird audio accuracy under shifts and harden models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the clean-data improvements are caused by the adversarial input perturbations themselves, but the training recipe bundles those perturbations with weight perturbation, a TRADES consistency term, and weight randomization, and the paper does not run a control that isolates the input perturbation.","fun_headline_variants_meta":{"raw":{"variants":["Adversarial training hardens bird audio models against shifts and attacks","10.5% accuracy gain on shifted bird sounds via adversarial training","Adversarial training beats distribution shifts and attacks in bird audio","Output-space attacks improve bird audio accuracy under shifts and harden models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000841,"raw_usage":{"total_tokens":3714,"prompt_tokens":1043,"completion_tokens":2671,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":659,"completion_tokens_details":{"reasoning_tokens":2597}},"tokens_in":659,"tokens_out":2671,"duration_ms":23750,"temperature":1.0,"reasoning_tokens":2597,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T16:18:21.427089+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same two architectures with random input noise of the same magnitude in place of FGSM perturbations, keeping the TRADES-AWP weight perturbation and weight randomization identical; if the cmAP gains over ordinary training persist or shrink only slightly, then the paper's attribution of the gains to adversarial input perturbation is not supported.","supporting_citations":[{"cited_title":"Asymmetric loss for multi-label classification","cited_arxiv_id":null,"evidence_quote":"Supplies the asymmetric loss that adapts the TRADES-AWP objective to multi-label classification."},{"cited_title":"Targeted adversarial attacks on wind power forecasts","cited_arxiv_id":null,"evidence_quote":"Defines the PRS, DRS, and TARS metrics used to measure adversarial robustness and prototype stability."}],"review_version":1}