{"id":"11196b2c-5005-4a99-98c3-720d905c9577","arxiv_id":"2501.11902","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A transferable GAN-based attack that preserves transcription and perceptual quality can substantially degrade current audio deepfake detection systems.","lead":"This paper proposes a GAN-based adversarial attack that combines an ensemble of audio deepfake detectors with a transcription-preserving loss to generate transferable attack audio. It reports that state-of-the-art detectors drop sharply in accuracy under the attack, especially in white-box and gray-box settings across three datasets.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Transcription preservation is measured with the same embedding cosine used as the training loss (Eq. 8); Fig. 5 shows transcript edits, so the central 'bypass while preserving transcription' claim is not yet supported.","rationale":"I read the paper as claiming an attack that is both transferable and high-quality (transcription- and perception-preserving). The vulnerability numbers may be right, but the paper's own evaluation of the quality component is circular: the metric in Table 5 is the same embedding cosine used as the training loss, and Fig. 5 shows highlighted transcript changes. The reader's weakest assumption captures exactly this, and the concern is load-bearing because the 'preserves transcription' clause is part of the central claim and the paper's stated novelty. I considered the inverted discriminator loss (Eq. 11) as an alternative primary concern: it is a clear internal inconsistency and should be fixed, but the attack's success is driven primarily by the surrogate forensics loss, so the transcription circularity is more directly tied to the central claim. If the WER test shows no meaningful transcript changes, the concern would be resolved and the conditional verdict could be upgraded. As it stands, CONDITIONAL remains the appropriate verdict.","tokens_in":12864,"tokens_out":7422,"duration_ms":71203,"concrete_test":"Take a random subset (e.g., 50 pairs) of original and attacked audio from ASVspoof2019 and In-the-Wild; transcribe both with an independent ASR system (e.g., Whisper large-v3) and also compare against ground-truth transcripts when available. Compute word error rate (WER) and character error rate (CER) between original and attacked transcripts. Additionally, inspect the highlighted transcript diffs in Fig. 5: if any highlighted change is a content-word substitution or deletion, then transcription is not preserved. The concern is settled if median WER is zero (or below a pre-specified tolerance such as 5%) and the Fig. 5 diffs are only formatting-level; it lands if WER is non-trivial or if the diffs include word substitutions.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's distinctive claim is that the attack preserves transcription and perceptual integrity (Sec. 1, Fig. 2). The only quantitative support is Table 5's 'Text Similarity', computed as cosine similarity between BERT embeddings of wav2vec transcripts (Eq. 8). This is the same objective optimized by LTranscription, making the evaluation circular: if the ASR/BERT pipeline maps altered phonemes to the same or similar tokens, cosine similarity can remain high while the actual speech content changes. Fig. 5 is direct evidence that this concern is not hypothetical: it states 'The attack is evident in the transcript texts' and highlights 'changes in the transcript before and after the attack'; on In-the-Wild the reported similarity is only 0.87, implying non-negligible transcript edits. Since the strongest claim explicitly includes 'while the attacked audio preserves transcription', an independent transcription metric is load-bearing. The inverted discriminator loss (Eq. 11) and missing baselines are additional reproducibility issues, but the transcription circularity is the one that, if unresolved, removes the paper's core novelty over earlier attacks.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a transferable GAN-based adversarial attack framework for audio deepfake detection (ADD). The framework uses an ensemble of surrogate ADD models, a discriminator, and a transcription model (wav2vec + BERT) with losses for perceptual quality, forensics, transcription, and adversarial objectives. The authors evaluate the attack on ASVspoof2019, In-the-Wild, and WaveFake datasets under white-box, gray-box, and black-box settings, reporting large accuracy drops in state-of-the-art ADD systems and claiming that the adversarial audio preserves transcription and perceptual integrity.","tokens_in":13063,"tokens_out":4524,"duration_ms":46132,"significance":"If validated, the attack would be a useful stress test for ADD robustness and would extend GAN-based adversarial attacks to the audio deepfake domain, particularly by targeting transcription-based detectors. The inclusion of three datasets and five target models is a positive breadth. The paper also makes an effort to analyze attack quality via PSNR, SSIM, and text similarity. However, the central claim of transcription preservation is currently supported only by a circular evaluation, and the training objective as written contains a likely error in the discriminator loss. The black-box cross-dataset results are suggestive but lack baselines, error bars, and implementation details. The contribution is therefore not yet established at the level required for publication.","major_comments":[{"comment":"The discriminator loss is written as LD = log(1 − D(A)) + log(1 − D(G(A'))). Under this objective, the discriminator is trained to assign the 'fake' label to both real audio A and generated audio G(A'). This is not the standard GAN minimax objective (which would be log D(A) + log(1 − D(G(A')))), and it would prevent the discriminator from learning a meaningful real-vs-fake decision. If this is a typographical error, the correct form must be given; if it is intentional, the rationale and training behavior need a detailed explanation. As written, the training procedure is not a valid adversarial game.","section":"§3.4, Eq. (11)"},{"comment":"The 'Text Similarity' metric in Table 5 is exactly 1 − LTranscription, i.e., the same cosine similarity between wav2vec+BERT embeddings that is optimized as the training loss in Eq. (8). This makes the transcription-preservation claim circular: the evaluation metric is the training objective, so high values only indicate that the optimization succeeded, not that spoken content is preserved in an independent sense. Furthermore, Fig. 5 explicitly states that 'The attack is evident in the transcript texts' and highlights changes in transcripts before and after the attack. On In-the-Wild, the reported similarity is only 0.87, implying non-negligible transcript edits. An independent measure such as word error rate against ground-truth transcripts, or a listening study, is required to support the claim that the attack preserves semantic transcription content.","section":"§4.3.2, Table 5 and §3.4, Eq. (8)"},{"comment":"The numbers reported in the text and the conclusion are inconsistent with Table 4 and with the abstract. The text in §4.2.1 says the average white-box performance drop is about 57%, but Table 4's 'Average' row shows post-attack accuracies of 70.01, 46.48, and 67.80 for the three datasets, which are not drops; the conclusion then describes 'accuracy drops of 57%, 30.5%, and 6%' and 'up to 70%, 47%, and 68% on ASVspoof2019, In-the-Wild, and WaveFake datasets' as if the post-attack accuracies were drops. Additionally, the white-box results are obtained using the same surrogate models on which the generator was trained, so the accuracy collapse is expected and does not demonstrate transferability. The black-box results are the meaningful evidence, and they should be reported with confidence intervals, a clear definition of the reported percentage (absolute vs. relative drop), and a comparison to the baseline performance of the same models on clean data.","section":"§4.2.1–4.2.3, Table 4, and §5"},{"comment":"The experimental protocol omits essential details for reproducibility and fair comparison. There is no perturbation budget (e.g., maximum l_p norm or PESQ threshold), no number of independent runs or error bars, no description of how the test set is split for attack generation, and no inference-time query budget for the black-box scenario. No baseline attacks are evaluated: previous GAN-based attacks (e.g., Rabhi et al. [20]) and traditional perturbation methods are cited but not compared against. The reader therefore cannot judge whether the proposed attack is stronger than existing methods or whether the reported differences are statistically meaningful.","section":"§4.2–§4.3, Tables 4 and 6"},{"comment":"The forensics loss is written as LForensics = − Σ_{i∈F} Σ_{j∈N} log F_i(G(A')). The notation is ambiguous: it is not clear what the inner sum over j ∈ N represents, and no target labels are specified. If this is intended to be a cross-entropy loss encouraging the generated audio to be classified as real by the surrogate models, the formula should be written explicitly with the true label convention (e.g., one-hot real labels for the fake class) or with a softmax/log-softmax formulation. The current expression is not precise enough to be implemented from the paper.","section":"§3.4, Eq. (5)"}],"minor_comments":[{"comment":"There are many typographical and grammatical issues, including 'data sets' in the abstract, missing articles, and inconsistent capitalization. The paper would benefit from careful proofreading.","section":"Throughout"},{"comment":"The discriminator architecture table contains unclear entries such as '(64, prev)' and fractions like '4' in the kernel shape column; these should be replaced with explicit dimensions and stride/padding details.","section":"Table 2"},{"comment":"The generator's 'feature map reduction module' is described only as a 3×3 convolutional layer with stride 1 and Tanh activation; details on padding, stride, and how the output length L is preserved should be provided.","section":"§3.3.1"},{"comment":"The weighting coefficients λ1, λ2, λ3, and λ4 are introduced but their final values are not fully specified. Table 6 only varies λ2 in a limited range; the values of the other weights used in the main experiments should be stated.","section":"§3.4"},{"comment":"The paper uses 'Wave2Vec', 'Wave2Vec', and 'wav2vec' inconsistently, and the reference [1] is to wav2vec 2.0. The exact model version and configuration should be specified, as well as the BERT model used for text embedding.","section":"§3.3.4"},{"comment":"The figure's solid and dotted lines for forward pass and back-propagation are not clearly explained, especially for the transcription model loop. The reader cannot tell which gradients flow to the generator from the transcription loss.","section":"Fig. 4"},{"comment":"For each dataset, the number of samples used for training the generator and for evaluation is not reported. The paper should state the test split sizes and any filtering used.","section":"§4.1"},{"comment":"Reference [21] is attributed only to 'N Reimers'; the correct citation for Sentence-BERT is Reimers and Gurevych (2019). Also, several URLs in footnotes are incomplete or may not resolve; please verify them.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses an important topic and the cross-dataset black-box results, if reproducible and correctly compared, could be a useful contribution. However, the circular transcription-preservation evaluation and the apparent error in the discriminator loss are load-bearing issues that must be resolved before the claims can be trusted. I would also urge the authors to add baseline comparisons and statistical significance tests; without those, the current evidence is anecdotal. The paper may be suitable for a forensic security venue after major revision, but not in its present form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful core of this paper is a straightforward, believable result: a GAN-based attack trained against an ensemble of surrogate audio deepfake detectors transfers to other detectors, and the transfer shows up even across datasets. That part deserves credit. The authors test white-box, gray-box, and black-box settings across ASVspoof2019, In-the-Wild, and WaveFake, and the black-box drops on ResNet and MS-ResNet are nontrivial.\n\nThe new piece relative to Rabhi et al. and Kawa et al. is a transcription-preserving loss meant to keep the attacked audio semantically unchanged. That is the paper's distinctive claim, and the paper's own evidence undercuts it. The only quantitative support is Table 5's 'Text Similarity,' which is exactly 1 minus the training loss in Eq. (8). The same wav2vec-BERT cosine used to optimize the generator is used to evaluate it. That is circular. Figure 5 shows actual transcript changes, and In-the-Wild similarity is only 0.87, implying real edits. The authors need an independent metric—word error rate against ground-truth transcripts or human listening—before they can claim the attack preserves transcription.\n\nOther soft spots, in decreasing order of severity. The discriminator loss in Eq. (11) looks inverted: real samples should maximize log D(A), not log(1−D(A)). Without code or data I can't tell if it's a typo or the implementation. There's no comparison to existing attacks, so it's hard to tell whether the transfer numbers beat a simple PGD baseline or earlier GAN attacks. The headline numbers are inconsistent across the abstract, Table 4, and the conclusion. And there are no error bars or variance estimates, which matters for black-box transfer claims.\n\nThe central vulnerability finding is plausible and probably true; the white-box collapses are expected, but the cross-dataset black-box effects are genuine evidence. If the authors fix the metric, add baselines, and release code, this becomes a solid contribution. I'd send it to peer review rather than desk-reject, but I'd expect major revision. For a reading group, it's a maybe—the method is understandable and the flaws are instructive.","headline":"Plausible transferable attack on audio deepfake detectors, but the transcription-preservation claim is circular and the paper needs major revision.","tokens_in":13628,"tokens_out":3620,"would_cite":false,"duration_ms":34184,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a transferable GAN-based attack can bypass state-of-the-art audio deepfake detectors while preserving the spoken content and the perceived quality of the audio.","keywords":["audio deepfake detection","transferable adversarial attack","GAN-based attack","transcription integrity","perceptual integrity","surrogate models","ASVspoof2019","black-box transferability"],"falsifier":"Run a word error rate comparison against ground-truth transcripts on the paper's released attacked samples: if words change while the reported text embedding similarity stays near 0.95–1.00, the transcription-preservation claim fails. A simpler version is to have listeners transcribe original and attacked clips; any systematic word substitutions would falsify the claim that the attack preserves semantic content.","tokens_in":12618,"feed_emoji":"🎧","tokens_out":9311,"duration_ms":79970,"temperature":0.7,"pith_summary":"This paper sets out to show that state-of-the-art audio deepfake detectors can be defeated by adversarial examples that transfer across models. The authors build a generative adversarial network (GAN) whose generator turns a deepfake clip into an attacked clip that a frozen ensemble of surrogate detectors classifies as real, while a self-supervised speech model and a text embedder keep the words and the sound quality close to the original. On the ASVspoof2019 benchmark, they report detector accuracy falling from around 98% to 26% when the attacker knows the model, and smaller but still measurable drops when the attacker has partial or no knowledge. If the results hold, existing ADD systems are not yet robust to transferable attacks, and claimed detection accuracy overstates their protection in the wild.","feed_headline":"GAN attack drops deepfake detector accuracy from 98% to 26%","feed_subtitle":"The attack transfers to unseen detectors while the audio sounds unchanged, so current defenses may overstate safety.","key_machinery":"The load-bearing object is the generator network, trained against three frozen critics: a discriminator, an ensemble of surrogate ADD models, and a wav2vec-to-BERT transcription model. The generator loss is a weighted sum of four terms, $L_G = \\lambda_1 L_{\\text{Perceptual}} + \\lambda_2 L_{\\text{Forensics}} + \\lambda_3 L_{\\text{Transcription}} + \\lambda_4 L_{\\text{Adversarial}}$, where the perceptual term is L1 distance to the input, the forensics term is cross-entropy against the surrogate ensemble, the transcription term is $1 - \\cos(f, f')$ between BERT embeddings of the original and attacked transcripts, and the adversarial term is the discriminator's fake loss. A residual connection scaled by a learned $\\alpha$ plus high-pass filtering keeps the output close to the input while removing artifacts. The ensemble's diversity is what converts a single-model attack into a transferable one.","core_discovery":"The paper's central claim is that a transferable GAN-based adversarial attack can bypass current state-of-the-art audio deepfake detectors while preserving the transcribed content and the perceived quality of the audio. The generator is trained with an ensemble of surrogate ADD models plus a wav2vec-based transcription module, so the resulting perturbations are not tuned to one detector's weights. In white-box settings, Res-TSSDNet accuracy on ASVspoof2019 falls from 98.46% to 26.33%, and the average white-box accuracy across the two surrogate models drops to 70.01%, 46.48%, and 67.80% on ASVspoof2019, In-the-Wild, and WaveFake, respectively; gray-box and black-box detectors also lose accuracy but less dramatically. The authors interpret these results as showing that current ADD systems are vulnerable to transferable, perceptually clean adversarial audio.","pith_inferences":["If the embedding-similarity proxy is looser than true transcription, the attack may be changing words without the reported text-similarity metric catching it; a word-error-rate audit would settle this.","The same generator-plus-transcription recipe could be pointed at automatic speech recognizers or voice assistants, since any differentiable audio classifier can serve as the surrogate ensemble.","A practical defense could run a second, independent ASR model on the audio and reject clips whose transcript disagrees with the claimed one; this directly targets the transcription-preservation assumption.","The lack of improvement when moving from two to three surrogates hints that architectural diversity, not the raw number of surrogates, drives transferability."],"forward_implications":["White-box ADD systems are highly vulnerable: on ASVspoof2019, Res-TSSDNet drops from 98.46% to 26.33% accuracy after the attack.","Attack transfer is real across architectures and datasets: models never used in training, such as RawNet2, ResNet, and MS-ResNet, lose accuracy on In-the-Wild and WaveFake.","Transcription-based ADD systems that verify text are also bypassed because the attack keeps its transcript aligned with the original audio while flipping the detector's decision.","The reported PSNR (39–44 dB), SSIM (0.96–0.99), and text similarity (0.87–1.00) imply that the attacked clips are hard to distinguish from the originals by ear or by waveform inspection.","Gray-box robustness improves with larger model variants, so model capacity appears to matter for resisting transferable attacks."],"supporting_citations":[{"why":"Supplies the Res-TSSDNet and Inc-TSSDNet surrogate models used to train the generator and as white-box targets.","marker":"[8]"},{"why":"Supplies the ResNet1D model tested as a black-box victim.","marker":"[7]"},{"why":"Supplies the RawNet2 model tested as a black-box victim.","marker":"[23]"},{"why":"Supplies the MS-ResNet model tested as a black-box victim.","marker":"[33]"},{"why":"Supplies the wav2vec model that transcribes original and attacked audio for the transcription loss.","marker":"[1]"},{"why":"Supplies the BERT text embedder whose cosine similarity defines the transcription loss and the reported text-similarity metric.","marker":"[21]"},{"why":"Provides the ASVspoof2019 LA dataset used to train surrogates and report main attack results.","marker":"[34]"},{"why":"Provides the In-the-Wild dataset used to test generalization.","marker":"[18]"},{"why":"Provides the WaveFake dataset used to test generalization.","marker":"[5]"},{"why":"Prior GAN-based adversarial attack and transcription-matching countermeasure that the proposed framework extends and contrasts against.","marker":"[20]"}],"fun_headline_variants":["Transferable GAN attack drops deepfake detector accuracy to 26%","Audio deepfake detectors fall to transferable GAN attacks","GAN attack exposes audio deepfake detector vulnerabilities","Transferable audio attack slashes deepfake detection accuracy","Deepfake audio detectors fooled by transferable GAN attack"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The core claim rests on the assumption that cosine similarity between wav2vec+BERT embeddings of original and attacked transcripts faithfully measures whether the words stayed the same, an assumption the paper never checks against an independent word-error-rate score.","fun_headline_variants_meta":{"raw":{"variants":["Transferable GAN attack drops deepfake detector accuracy to 26%","Audio deepfake detectors fall to transferable GAN attacks","GAN attack exposes audio deepfake detector vulnerabilities","Transferable audio attack slashes deepfake detection accuracy","Deepfake audio detectors fooled by transferable GAN attack"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000481,"raw_usage":{"total_tokens":2392,"prompt_tokens":972,"completion_tokens":1420,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":588,"completion_tokens_details":{"reasoning_tokens":1339}},"tokens_in":588,"tokens_out":1420,"duration_ms":10700,"temperature":1.0,"reasoning_tokens":1339,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T17:45:13.631324+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a word error rate comparison against ground-truth transcripts on the paper's released attacked samples: if words change while the reported text embedding similarity stays near 0.95–1.00, the transcription-preservation claim fails. A simpler version is to have listeners transcribe original and attacked clips; any systematic word substitutions would falsify the claim that the attack preserves semantic content.","supporting_citations":[{"cited_title":"To- wards end -to-end synthetic speech detection","cited_arxiv_id":null,"evidence_quote":"Supplies the Res-TSSDNet and Inc-TSSDNet surrogate models used to train the generator and as white-box targets."},{"cited_title":"Holmes: health online model ensemble serving for deep learning models in intensive care units","cited_arxiv_id":null,"evidence_quote":"Supplies the ResNet1D model tested as a black-box victim."},{"cited_title":"End-to-end anti-spoofing with rawnet2","cited_arxiv_id":null,"evidence_quote":"Supplies the RawNet2 model tested as a black-box victim."},{"cited_title":"wav2vec 2.0: A framework for self-supervised learning of speech representations","cited_arxiv_id":null,"evidence_quote":"Supplies the wav2vec model that transcribes original and attacked audio for the transcription loss."},{"cited_title":"Asvspoof 2019: A large-scale public database of syn- thesized, converted and replayed speech","cited_arxiv_id":null,"evidence_quote":"Provides the ASVspoof2019 LA dataset used to train surrogates and report main attack results."}],"review_version":1}