{"id":"29cbc8ea-4b56-415f-aee1-7634ef4b1b30","arxiv_id":"2411.09220","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Applying momentum and variance-tuned gradient attacks in the time domain with a VAD mask improves transferable attack success against Whisper and Speech2Text ASR models.","lead":"This paper tests black-box transferable adversarial attacks against five speech recognition models and proposes a speech-aware gradient optimization method. The proposed attacks raise word error rates beyond a standard baseline, suggesting that commercial ASR systems may be vulnerable to imperceptible audio perturbations.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The transferability gain over PGD is confounded because PGD is not specified to use the same differentiable feature extractor; the proposed methods' advantage could stem from the surrogate front-end rather than the VAD mask or momentum.","rationale":"The reader's weakest_assumption correctly flags the underspecified feature extractor and VAD as reproducibility risks. My stress-test sharpens this into a confounding concern: the PGD baseline may not use the same feature extractor, which would make the comparison unfair and would mean the claimed improvements might stem from the surrogate front-end rather than the proposed SAGO, MI-FGSM, or VMI-FGSM. This is load-bearing because the paper's central claim is specifically that these optimizers yield higher WER than PGD. However, the paper does report white-box results where PGD reaches 100% WER, indicating PGD is not trivially weak, and the transfer improvements are plausible given known momentum/variance techniques from computer vision. Therefore the concern does not justify rejection; it demands a controlled experiment and additional implementation details. The reader's CONDITIONAL verdict remains appropriate, with the condition being that the feature extractor and baseline must be clearly specified and an ablation must be provided. My agreement is partial because the reader focused on replicability rather than the confound between feature extractor and optimizer, which I see as the more direct threat to the central claim.","tokens_in":10516,"tokens_out":7624,"duration_ms":80158,"concrete_test":"Re-run all four attacks (PGD, SAGO, MI-FGSM, VMI-FGSM) using the identical differentiable feature extractor pipeline and the same loss Ladv on the first 500 LJSpeech utterances, with the same 50 iterations and perturbation bounds (ξ=0.002, 0.0035). If PGD with the surrogate feature extractor achieves transfer WER comparable to the proposed methods, the proposed methods add nothing beyond the feature extractor; if PGD remains lower, the specific optimizers or VAD mask are responsible. Report WER means and standard deviations over at least three random seeds to verify that the observed differences exceed run-to-run noise.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that SAGO, MI-FGSM, and VMI-FGSM outperform PGD for black-box transfer. However, the PGD baseline is described only as 're-implement [8]'; the paper never states whether PGD runs in the time domain with the same differentiable 80-dim log-filterbank feature extractor used for the proposed attacks (Section II-C). Since this feature extractor is a core component of the proposed methods, a fair comparison requires PGD to use the identical feature extractor and loss Ladv. If PGD instead uses each source model's native front-end (e.g., Whisper's internal log-mel with normalization), then the proposed attacks' advantage could be due to the surrogate front-end providing better gradient alignment across models, not from SAGO's VAD mask or momentum/variance tuning. The VAD mask algorithm is also only described as a 'cepstral power measurement' with no parameters, making its contribution impossible to isolate or replicate. Without ablations that hold the feature extractor fixed across all methods, the empirical claim collapses into a confounded comparison, and the reported WER improvements cannot be attributed to the claimed innovations.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies untargeted transferable adversarial attacks against automatic speech recognition (ASR) systems. It proposes three time-domain attack methods: SAGO, which masks the gradient with a voice-activity-detection (VAD) mask; MI-FGSM, a momentum-based iterative attack; and VMI-FGSM, a variance-tuned momentum attack. All three are applied through a differentiable 80-dimensional log-filterbank feature extractor so that gradients can propagate to the raw audio. The authors evaluate the methods on five ASR models (Whisper-tiny, Whisper-base, Speech2Text small/medium/large) and two datasets (LJ-Speech and LibriSpeech), reporting WER under clean, noise, and attack conditions, and they claim higher transferability than a PGD baseline plus human imperceptibility of the adversarial samples.","tokens_in":10728,"tokens_out":4773,"duration_ms":50058,"significance":"If the claims are correct, the work offers a useful tool for robustness evaluation of black-box ASR models and an interesting new application of VAD for adversarial transferability. The idea of focusing adversarial gradients on speech regions via a VAD mask is original and the evaluation across five models and two datasets is broader than many prior audio-attack papers. However, the paper is purely empirical and the current evidence has several gaps: the PGD baseline is not specified to use the same feature extractor, the VAD algorithm is not described reproducibly, no statistical uncertainty is reported, and the human-imperceptibility claim is unverified. A strength is that the authors provide a link to adversarial samples and code, which facilitates independent verification even though the code is not included in the manuscript.","major_comments":[{"comment":"The PGD baseline is described only as \"re-implement [8]\" and the paper never states whether PGD operates in the time domain using the same 80-dimensional log-filterbank differentiable feature extractor and the same adversarial loss as the proposed methods. Because the differentiable front-end is a central component of the proposed time-domain attacks, the reported advantage of SAGO/MI-FGSM/VMI-FGSM over PGD could be due to the surrogate front-end rather than to the VAD mask or momentum/variance tuning. Please specify the PGD implementation completely and, ideally, add an ablation where PGD uses the identical feature extractor and loss as the proposed attacks.","section":"§II-C, §III-B"},{"comment":"The VAD mask is described only as a \"cepstral power measurement\" that trims non-speech parts from the front and end of the audio; no algorithm, threshold parameters, or mask shape (binary or soft) are given. This makes SAGO unreproducible and prevents isolating the contribution of the mask to transferability. Please provide the exact VAD procedure and parameters, and include an ablation of SAGO with and without the mask under otherwise identical conditions.","section":"§II-B.1, §III-B"},{"comment":"The central claim that the proposed attacks \"generally induce significant performance degradation, yielding higher WER compared to baselines\" is made without error bars, confidence intervals, or statistical tests, and several cells in Tables I–II contradict a consistent superiority (e.g., Table I, S2T-small source to S2T-medium target at SNR 30: PGD 30.15 versus VMI-FGSM 28.94). Please report per-condition win rates, standard deviations across utterances or random subsets, and a paired significance test over the 500 test utterances.","section":"§IV-C, Tables I–II"},{"comment":"The human-imperceptibility claim is supported only by the assertion that the adversarial samples \"sound similar\" to the original audio. No listening test, no perceptual quality metrics (e.g., PESQ or STOI), and no measurement of the actual perturbation level relative to the signal are reported. Since the abstract and introduction explicitly claim minimal impact on human perceptibility, please provide either a formal listening study or objective perceptual metrics, or explicitly soften the claim to a statement about the bounded ℓ∞ perturbation.","section":"§IV-D"},{"comment":"The paper mentions prior transferable ASR attacks [27], [28] but does not compare against them; the only attack baseline is a self-implemented PGD. Given the paper's stated goal of a \"comprehensive exploration\" of black-box transferability, please compare with at least one existing transfer-based ASR attack or justify their exclusion with concrete reasons.","section":"§III-B, §IV"}],"minor_comments":[{"comment":"\"log filterback features\" should be \"log filterbank features.\"","section":"§II-C"},{"comment":"\"our VIM-FGSM\" should be \"our VMI-FGSM.\"","section":"§II-B.2"},{"comment":"\"differnt SNR values\" should be \"different SNR values.\"","section":"§IV"},{"comment":"\"V AD\" should be written as \"VAD\" for consistency.","section":"Throughout"},{"comment":"The table captions contain formatting errors such as \"LJSPEECH\" and \"S OURCE\"; also, the text inconsistently uses \"Speech2text\" and \"S2T.\"","section":"Tables I–II"},{"comment":"The exact Hugging Face model identifiers or versions (e.g., openai/whisper-tiny, facebook/s2t-small-librispeech) are not given; please list them for reproducibility.","section":"§III-B"}],"recommendation":"major_revision","confidential_remarks":"I recommend major revision. The core idea is interesting and potentially useful, but the missing specification of the PGD baseline and the VAD mask are load-bearing confounds that must be resolved. The paper could become acceptable if the authors rerun or clarify the baselines with a fixed feature extractor, add VAD ablations, and report uncertainty and perceptual evidence. The manuscript does not currently justify the strong transferability and imperceptibility claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing worth knowing: this is the first systematic look I know of at transfer-based adversarial attacks against modern ASR models like Whisper and Speech2Text, and the WER tables across five models and two datasets give a real empirical map of cross-family transfer. That part is genuinely useful. The second thing: the central comparison to PGD is under-specified enough that the claimed advantage may not be real. The paper never states whether PGD runs through the same 80-dimensional differentiable log-filterbank feature extractor as the proposed methods. If PGD uses each source model's native front-end, then the improvement from SAGO/MI-FGSM/VMI-FGSM could come from the surrogate feature extractor's gradient alignment, not from the VAD mask or momentum. That is a load-bearing hole, not a nitpick.\n\nWhat is new and good: SAGO, a VAD-masked PGD variant, is not in the cited prior literature, and applying MI-FGSM and VMI-FGSM to raw audio through a differentiable feature extractor is a legitimate, if modest, extension of known vision attacks. The experiments cover a broad set of source-target model pairs, include meaningful noise baselines (white, babble, music, natural), and report WER on held-out audio. No circular fitting is going on; the results are measured, not derived from assumptions.\n\nThe soft spots are real but fixable. There are no error bars or statistical tests, so many of the differences could be noise, especially in the 1-3 WER range. No comparison with prior ASR transfer attacks means the novelty claim is weaker than the authors imply. The VAD is described only as a ‘cepstral power measurement’ with no parameters, so it is not reproducible as written. There is no ablation that holds the feature extractor fixed and isolates the VAD mask, which is exactly the experiment needed to support the SAGO contribution. The human imperceptibility claim is asserted with no listening test, and the 'sounds similar' statement in Section IV-D is not evidence. The stress-test note about the PGD baseline is correct and should be the first thing the authors are asked to fix.\n\nWho is this for? Adversarial audio researchers and anyone thinking about robustness of deployed Whisper/S2T systems. It deserves a serious referee, not a desk reject: the empirical phenomenon is real, the scope is valuable, and the paper can be repaired with clarifications, ablations, and error bars. My own verdict leans skeptical on the attribution of the gains, but the underlying attack results are worth having in the literature. I would send it to review and request major revision.","headline":"A useful first empirical sweep of transfer attacks against Whisper and Speech2Text, but the headline gains over PGD are confounded because the paper never shows PGD uses the same differentiable feature extractor as the proposed attacks.","tokens_in":11251,"tokens_out":1660,"would_cite":true,"duration_ms":20112,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adversarial audio crafted on a surrogate model degrades black-box ASR systems more than PGD across five models.","keywords":["adversarial attacks","automatic speech recognition","transferable attacks","voice activity detection","momentum iterative fast gradient sign method","variance tuning","differentiable feature extraction","word error rate"],"falsifier":"A direct falsifier would be to re-run the three attack recipes with the authors' exact feature extractor and VAD threshold on the same five models and check whether the transfer WER still exceeds PGD; if a correct reimplementation reproduces numbers at or below the PGD baseline under the same $\\ell_\\infty$ budgets of 0.002 and 0.0035, the transferability claim is refuted. A narrower check: ablate the VAD mask by replacing it with a random mask of the same sparsity; if SAGO's transfer gain disappears, the paper's explanation of speech-aware targeting is wrong even if the attacks still work.","tokens_in":10317,"feed_emoji":"🎙️","tokens_out":4993,"duration_ms":48812,"temperature":0.7,"pith_summary":"The paper argues that black-box automatic speech recognition (ASR) models can be broken by adversarial audio crafted against a surrogate model, without seeing the target's weights. It proposes three transferable attacks applied directly in the time domain through a differentiable feature extractor: SAGO, which masks gradients to the speech regions identified by a voice activity detection rule, and MI-FGSM and VMI-FGSM, which stabilize gradient updates with momentum and variance tuning. Across five Whisper and Speech2Text models and two datasets, the attacks raise word error rate on black-box targets beyond the strong PGD baseline while keeping perturbations small enough to be inaudible. The paper's central message is that ASR robustness should be evaluated under transfer attacks, not only white-box attacks, because a surrogate model is enough to degrade real systems.","feed_headline":"Audio attacks transfer across black-box ASR models","feed_subtitle":"Momentum, variance tuning, and a voice-activity mask beat the PGD baseline on five ASR models.","key_machinery":"The central machinery is a differentiable feature extractor that turns raw audio into 80-dimensional log-filterbank features so gradients flow to the waveform itself, enabling time-domain attacks. On top of that, SAGO multiplies the adversarial loss gradient by a voice activity detection mask, built from a cepstral power threshold, to restrict updates to detected speech; MI-FGSM adds a momentum term to the sign gradient; and VMI-FGSM augments that with neighborhood gradient variance tuning. All three iterate fifty steps under an $\\ell_\\infty$ constraint with perturbation bounds 0.002 and 0.0035, and the adversarial loss is the negative cross-entropy between the ground-truth text and the model's predicted tokens.","core_discovery":"On the paper's own terms, the discovery is that adversarial perturbations optimized on a white-box source model transfer to black-box ASR systems when the gradient is computed all the way back through a differentiable 80-dimensional log-filterbank front end. The paper reports that the resulting time-domain attacks, especially MI-FGSM and VMI-FGSM for Whisper models and SAGO for Speech2Text models, produce higher WER than the PGD baseline on held-out target models. The mechanism attributed for the gain is that momentum and variance tuning avoid poor local optima in the loss landscape, while the VAD mask focuses perturbation energy on speech segments that ASR models rely on and that human listeners attend to.","pith_inferences":["The paper leaves open whether the VAD mask specifically causes the transferability gain; an ablation that applies the same mask to PGD or that reverses the mask to non-speech regions would isolate its contribution, and the current tables only report the combined recipes.","Because the feature extractor and VAD algorithm are not fully specified, the results are not yet independently reproducible; spelling out the exact front end would turn the headline comparison into a checkable recipe.","The same time-domain attack recipe could be tested against newer encoder-decoder and CTC ASR families, such as Wav2Vec2, Conformer, or multilingual Whisper variants; if it transfers there, the claim generalizes beyond the five tested models.","For defenders, the natural counterpart experiment is to measure whether the transfer gains persist under acoustic preprocessing such as resampling, codec compression, or background-noise suppression, since physical-world attacks rarely reach the decoder as clean 16 kHz waveforms."],"forward_implications":["If true, black-box ASR deployment should assume attackers can degrade WER substantially using only a surrogate model, so defenses need to target transfer rather than just white-box accuracy.","WER increases of 65% or more on a target from a surrogate S2T-medium attack imply concrete failures in captioning and car navigation systems fed by such models.","The success of momentum and variance-tuned gradient methods on Whisper suggests that large transformer ASR models inherit the transferability vulnerabilities already known in image classifiers.","VAD masking can act as a general plug-in for other gradient-based ASR attacks, since SAGO's gains come from concentrating perturbation on speech regions.","Human imperceptibility and transferability can hold simultaneously, so low-perturbation attacks are not automatically stopped by a small $\\ell_\\infty$ budget."],"supporting_citations":[{"why":"Supplies the PGD baseline and the earlier Whisper white-box attack that the proposed methods are compared against.","marker":"[8]"},{"why":"Introduces MI-FGSM, the momentum-based gradient recipe the paper adapts to time-domain audio.","marker":"[31]"},{"why":"Introduces variance tuning, the mechanism behind VMI-FGSM.","marker":"[32]"},{"why":"Defines the Whisper ASR architecture used as source and target model.","marker":"[29]"},{"why":"Defines the Speech2Text (S2T) transformer ASR architecture used as source and target model.","marker":"[30]"},{"why":"Supplies the LibriSpeech evaluation data.","marker":"[44]"},{"why":"Supplies the LJ-Speech evaluation data.","marker":"[45]"},{"why":"Provides the babble, music, and natural noise baselines used to show adversarial perturbations outperform additive noise.","marker":"[39]"}],"fun_headline_variants":["Transferable audio attacks fool black-box ASR","Black-box speech models vulnerable to transfer attacks","Momentum and VAD mask boost ASR attack transfer","SAGO and momentum beat PGD on black-box ASR"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that gradients computed through the paper's 80-dimensional log-filterbank feature extractor on a surrogate model keep pointing in directions that hurt arbitrary black-box ASR models, and that the voice-activity mask identifies exactly the speech regions that decide recognition.","fun_headline_variants_meta":{"raw":{"variants":["Transferable audio attacks fool black-box ASR","Black-box speech models vulnerable to transfer attacks","Momentum and VAD mask boost ASR attack transfer","SAGO and momentum beat PGD on black-box ASR"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000381,"raw_usage":{"total_tokens":1976,"prompt_tokens":856,"completion_tokens":1120,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":472,"completion_tokens_details":{"reasoning_tokens":1055}},"tokens_in":472,"tokens_out":1120,"duration_ms":9141,"temperature":1.0,"reasoning_tokens":1055,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T20:53:07.988544+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct falsifier would be to re-run the three attack recipes with the authors' exact feature extractor and VAD threshold on the same five models and check whether the transfer WER still exceeds PGD; if a correct reimplementation reproduces numbers at or below the PGD baseline under the same $\\ell_\\infty$ budgets of 0.002 and 0.0035, the transferability claim is refuted. A narrower check: ablate the VAD mask by replacing it with a random mask of the same sparsity; if SAGO's transfer gain disappears, the paper's explanation of speech-aware targeting is wrong even if the attacks still work.","supporting_citations":[{"cited_title":"There is more than one kind of robustness: Fooling Whisper with adversarial examples,","cited_arxiv_id":null,"evidence_quote":"Supplies the PGD baseline and the earlier Whisper white-box attack that the proposed methods are compared against."},{"cited_title":"Boosting adversarial attacks with momentum,","cited_arxiv_id":null,"evidence_quote":"Introduces MI-FGSM, the momentum-based gradient recipe the paper adapts to time-domain audio."},{"cited_title":"Enhancing the transferability of adversarial attacks through variance tuning,","cited_arxiv_id":null,"evidence_quote":"Introduces variance tuning, the mechanism behind VMI-FGSM."},{"cited_title":"Fairseq s2t: Fast speech-to-text modeling with fairseq,","cited_arxiv_id":null,"evidence_quote":"Defines the Speech2Text (S2T) transformer ASR architecture used as source and target model."},{"cited_title":"Librispeech: an asr corpus based on public domain audio books,","cited_arxiv_id":null,"evidence_quote":"Supplies the LibriSpeech evaluation data."}],"review_version":1}