{"id":"d0b82142-32f8-47c4-a5b4-2f9d867e95cc","arxiv_id":"2505.17513","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Small semantic-preserving changes to transcripts, passed through text-to-speech, significantly reduce the accuracy of both open-source and commercial audio anti-spoofing detectors.","lead":"This paper shows that small wording changes in the text used to generate a voice can fool audio deepfake detectors into thinking the fake audio is real. The authors attacked several open-source and commercial systems, and one commercial detector's accuracy fell from 100% to 32% after such changes.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Open-source attack rates may be inflated by per-voice batch-norm calibration; no uncalibrated baseline is reported (App. B.1), so the main quantitative claims are not yet grounded.","rationale":"The reader's weakest assumption identifies exactly the concern I consider most load-bearing: the open-source detectors are batch-norm calibrated to each TTS voice until accuracy exceeds 90%, and no uncalibrated baseline is provided. This matters because the quantitative open-source claims, including the headline 'attack success rates surpass 60%' and the contribution that accuracy drops from over 90% to below 20%, are all produced on these adapted models. If calibration creates a sharp, voice-specific decision boundary, then the vulnerability being measured may be a property of the adaptation procedure rather than of the released detectors. The commercial API-A result is not affected by this confound, so the paper's qualitative finding that at least one commercial detector is sensitive to transcript changes remains plausible; that is why I would not move the verdict to REJECT. The reader's CONDITIONAL verdict already reflects this uncertainty, and my read does not change that verdict. I agree with the reader's framing, and the proposed test is the minimal experiment that would settle whether the calibration step is the cause of the reported open-source attack success. I did not find a more fundamental internal inconsistency: the attack pipeline is well specified, the equations are coherent, and the feature analysis is appropriately described as correlational. The remaining gaps, such as unreported semantic threshold delta and lack of human evaluation, are secondary to the calibration confound for the central quantitative claim.","tokens_in":16860,"tokens_out":8568,"duration_ms":76842,"concrete_test":"Rerun Algorithm 1 with the same TTS voices, transcripts, and attack methods (at least TextFooler and BERTAttack) on AASIST-2, CLAD, and RawNet-2 without the batch-norm calibration described in Appendix B.1, using the same 1,439 VoiceWukong transcripts as in Table 3. Report OC, AUA, and ASR, computing ASR only on the subset of samples the uncalibrated detector originally labels as spoof. Also report the calibration-set size and the semantic threshold delta used in Eq. (2). If uncalibrated OC is below 90% for most TTS-voice pairs, or if uncalibrated ASR is more than 20 points lower than the calibrated ASR in Table 3, the reported open-source vulnerability is substantially attributable to the per-voice calibration step; if ASR remains similar, the confound is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The open-source results in Table 3 and the contribution claim that accuracy drops 'from over 90% to just below 20%' are measured on detectors adapted by batch-norm calibration on a small, non-overlapping set of the same TTS voice until detection accuracy exceeds 90% (Appendix B.1). The paper reports no comparison with uncalibrated detectors and no calibration-set size. If this adaptation step, rather than an intrinsic property of the detectors, is what positions the decision boundary near the synthetic-voice manifold, then the transcript substitutions could be exploiting a boundary created by the calibration procedure. The commercial API-A result (100% to 32%, Table 4) is independent of this step and would keep the qualitative finding alive, but the open-source ASR numbers and the feature analysis built on 108 open-source experiments would not support the broad conclusion that current SOTA anti-spoofing systems are vulnerable to linguistic perturbations. The concern is concrete: because calibration is applied only to open-source detectors and only until a high accuracy threshold is hit, the reported attack success rates conflate detector sensitivity with the effect of the adaptation protocol.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces transcript-level adversarial attacks against audio anti-spoofing detectors (AASs). The authors formulate a black-box optimization problem in which a transcript is minimally perturbed with synonym or MLM-based substitutions, synthesized through a TTS model, and scored by the target detector. They evaluate three open-source detectors (AASIST-2, CLAD, RawNet-2) and two commercial APIs across multiple TTS voices, reporting attack success rates above 60% for several open-source pairs and a drop for one commercial API from 100% to 32% accuracy. They also present a logistic-regression feature analysis over linguistic, acoustic, and model-level features, and a case study simulating the Brad Pitt voice-cloning scam. The central claim is that AASs are vulnerable to minor transcript-level linguistic perturbations, not just acoustic manipulations.","tokens_in":17074,"tokens_out":5569,"duration_ms":45277,"significance":"If substantiated, the paper identifies a genuinely under-explored attack surface: transcript variation that propagates through TTS synthesis and flips the decisions of state-of-the-art anti-spoofing systems. The work is broad in scope: multiple detectors, TTS models, voice profiles, and perturbation strategies, plus a black-box attack formulation and a concrete commercial-API case study. The commercial API-A result (100% to 32%) and the case study provide vivid, practically relevant evidence. The feature analysis is a useful first step toward explainable vulnerability, and the authors are explicit about limitations. However, the quantitative open-source claims currently rest on a per-voice batch-norm calibration protocol whose effect is not isolated, so the strength of the evidence is not yet commensurate with the broad conclusions.","major_comments":[{"comment":"The open-source results are measured on detectors that are batch-norm calibrated to each TTS voice until detection accuracy exceeds 90% on a small non-overlapping set. The paper reports no comparison with the uncalibrated detectors, no calibration-set size or selection procedure, and no sensitivity analysis around the 90% threshold. This matters because calibration moves the decision boundary toward the synthetic-voice manifold, and the transcript substitutions may be exploiting a boundary created by the adaptation protocol rather than an intrinsic property of the detectors. Since all 108 open-source experiments and the feature regression in Table 5 use these adapted models, the reported ASR values are not yet grounded as evidence of inherent linguistic sensitivity. Please report (a) attack success rates on uncalibrated detectors, (b) the calibration-set size and how it is sampled, and (c) how ASR varies with the calibration accuracy target. The commercial API-A result in Table 4 is independent of this issue and would keep the qualitative finding alive, but the open-source numbers and the 'from over 90% to just below 20%' contribution claim require this comparison.","section":"Appendix B.1; Tables 3 and 5"},{"comment":"The CLAD American Male PWWS row reports OC = 34.0%, while the other three attack methods for the same detector-voice pair report OC = 99.8%. Original accuracy before attack should be identical across attack methods because it is measured on the same unperturbed transcripts. This entry appears erroneous, and it propagates to the averaged row in Table 3 (OC = 83.3%, ASR = 56.6%) and to aggregate statements about detector vulnerability. Please correct the entry and recompute all affected averages and any claims derived from them. The presence of this inconsistency also raises a data-integrity concern for the other appendix tables.","section":"Table A5 (CLAD, American Male, PWWS row)"}],"minor_comments":[{"comment":"The semantic preservation constraint depends on a threshold δ, but the paper never reports the value of δ or the sensitivity of ASR to this threshold. Please state δ and, ideally, show how attack success varies with δ.","section":"Eq. (2), Section 4"},{"comment":"The commercial-API results are based on 100 randomly sampled items per pair, but no confidence intervals are reported. Given the headline claim of a drop from 100% to 32%, exact binomial confidence intervals would help quantify the uncertainty, especially for the 100% entries.","section":"Table 4, Section 5.2.2"},{"comment":"In the definition of the audio input, the text says 'L is the wavelength of an audio'; this should be 'length of an audio.'","section":"Section 3, Problem Formulation"},{"comment":"The appendix headers and text use both 'RawNet-2' and 'Rawnet-2'; please standardize the spelling.","section":"Appendix B.4"},{"comment":"The Spoof F1 and Bona-fide F1 features are computed on the same VoiceWukong evaluation set and are therefore strongly tied to overall detector quality. Their high-magnitude coefficients are not surprising and should be interpreted as descriptive of detector quality rather than as evidence of a distinct linguistic mechanism.","section":"Section 6, Table 5"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the journal's scope and addresses an interesting and timely problem. The main risk is the calibration protocol: if the open-source results are heavily influenced by per-voice batch-norm calibration, the paper's central quantitative claims would need substantial reframing. The Table A5 anomaly should be checked independently; if it is a typo, it is fixable, but it currently undermines confidence in the appendix tables. The commercial API-A result and case study are the strongest independent evidence, and they justify further work rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nYou should read this paper because it opens a new attack surface: transcript-level perturbations on TTS input can flip audio anti-spoofing detectors. The commercial API-A result, 100% to 32%, is the cleanest evidence and is independent of the open-source calibration protocol. The Brad Pitt scam case study is a concrete demonstration of real-world risk.\n\nWhat is actually new: they adapt established text adversarial attacks (TextFooler, BAE, BERTAttack) to the TTS channel and systematically probe linguistic sensitivity across multiple detectors and voices. The feature analysis, though correlational, is a reasonable first map of why some detector-voice pairs are vulnerable and others not. The AES finding, that detectors with tighter audio embedding clusters are more robust, is useful even if descriptive.\n\nSoft spots, in proportion:\n\n1. The batch-norm calibration (App. B.1) is a real confound for the open-source attack rates. They calibrate each detector to each TTS voice until accuracy exceeds 90%, then measure attack success on the calibrated model. No uncalibrated baseline is reported. The stress-test worry is legitimate: calibration might position the decision boundary near the synthetic-voice manifold, making transcript substitutions more effective than on a stock detector. However, this does not kill the claim. The commercial API-A and API-B results involve no such adaptation, and API-A's drop from 100% to 32% shows the phenomenon is not an artifact of calibration. Also, some reported OC values are below 90% even after calibration, which is odd but not fatal.\n\n2. The semantic threshold delta in Eq. 2 is never reported. That is a reproducibility gap. They report COS stays around 90%, but they do not say what delta they used. Minor but annoying.\n\n3. No code at submission, and commercial API details are pseudonymous. The 100-sample commercial runs are small, but effect sizes are large enough that noise is unlikely.\n\nThe feature analysis is the weakest part. Logistic regression on engineered features is inherently correlational, and 108 experiments is modest relative to the feature count. But they present it as hypothesis-generating, not causal.\n\nBottom line: the paper is a genuine first step on a relevant problem. The calibration issue means I would want an uncalibrated baseline before trusting the open-source ASR numbers, but the core phenomenon, linguistic variation flips anti-spoofing predictions, is supported by the commercial results and the case study. This deserves a serious referee. I would send it to review, with a request for the delta values, an uncalibrated comparison, and ideally code. I would cite it for the attack surface and bring it to reading group.","headline":"Transcript-level perturbations flip anti-spoofing detectors; open-source numbers are clouded by batch-norm calibration, but the commercial drop and case study make the phenomenon real.","tokens_in":17592,"tokens_out":2863,"would_cite":true,"duration_ms":30965,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Rewording the transcript before text-to-speech synthesis can flip audio deepfake detectors from “spoof” to “real,” with one commercial detector dropping from 100% to 32% accuracy.","keywords":["deepfake audio detection","audio anti-spoofing","transcript-level adversarial attacks","linguistic sensitivity","text-to-speech synthesis","adversarial robustness","voice cloning fraud"],"falsifier":"Run the same transcript-level attack against the three open-source detectors without any per-voice batch-norm calibration, or with a single fixed calibration learned on one voice, and check whether attack success rates still exceed 60%; if they collapse, the linguistic sensitivity is a calibration artifact rather than a property of the detectors.","tokens_in":16639,"feed_emoji":"🔊","tokens_out":7648,"duration_ms":54868,"temperature":0.7,"pith_summary":"The paper tries to establish that audio deepfake detectors are sensitive to the wording of the transcript that is read aloud by text-to-speech, not only to the acoustics of the audio. It shows that small, meaning-preserving word substitutions can flip a detector's verdict from “spoofed” to “real” for many detector–voice pairs, with attack success rates above 60% on several open-source systems and one commercial API falling from 100% to 32% accuracy. If true, this means current anti-spoofing defenses, which are evaluated almost entirely on acoustic perturbations, can be bypassed by simply editing the scam script. The paper also argues the vulnerability is systematic: a regression analysis of linguistic, acoustic, and model-level features links susceptibility to linguistic complexity and to how clustered the detector's audio embeddings are for a given synthetic voice.","feed_headline":"Rewording the script drops a deepfake detector from 100% to 32%","feed_subtitle":"Synonym-level edits to the text fed into text-to-speech flip spoof audio to “real” in open-source and commercial systems.","key_machinery":"The load-bearing machinery is a black-box, model-agnostic transcript perturbation algorithm: it scores each word by how much its removal changes the detector's bona-fide probability, then greedily substitutes the highest-impact words with synonym or masked-language-model candidates that maximize the bona-fide score while passing a semantic-similarity threshold and part-of-speech check. The analysis side introduces Audio Encoder Similarity, the mean cosine similarity between a voice's synthesized audio embeddings and their centroid in the detector's representation space, and uses it to show that detectors whose embeddings tightly cluster a given text-to-speech voice are far less vulnerable. This metric, together with the detector's own spoof and bona-fide F1 scores, gives the strongest predictive signal for whether a transcript attack will succeed.","core_discovery":"On the paper's own terms, the central claim is that linguistic variation in a transcript propagates through a text-to-speech pipeline and changes the acoustic realization enough that state-of-the-art anti-spoofing models change their decision, even though the audio's semantic content and the synthesis system are untouched. The authors formalize this as an optimization problem: find a minimally edited transcript, constrained to preserve syntax and meaning, whose synthesized audio is labeled bona-fide. Their empirical evaluation across 1,439 transcripts, three open-source detectors, several text-to-speech voices, and two commercial APIs reports attack success rates up to 82% in open-source settings and a drop from 100% to 32% for one commercial detector when paired with a commercial TTS service. A case study replicating a celebrity voice-cloning scam shows per-utterance bona-fide probabilities rising from below 1% to 58–90% after a few lexical substitutions, which the paper presents as evidence that the vulnerability is exploitable in real-world fraud scenarios.","pith_inferences":["A natural extension the paper does not test is whether a transcript-only proxy, using linguistic features like readability and perplexity change without querying the detector, can select effective rewording, which would let an attacker operate fully offline.","The finding implies that training corpora with systematic linguistic differences between spoofed and real speech, as the paper shows for the ASVSpoof 2019 data, may teach detectors a linguistic prior; paraphrasing toward the bona-fide register could transfer across text-to-speech voices.","If the effect is driven by acoustic embedding geometry, a defense could regularize detectors to keep embeddings of paraphrases close, or text-to-speech pipelines could normalize prosody across synonym substitutions; neither direction is explored in the paper.","The paper leaves open whether human listeners would notice the edited transcripts; a user study testing naturalness and suspicion would clarify whether the attack is realistic in live conversation rather than only in automated detection pipelines."],"forward_implications":["A fraudster can evade a deployed audio deepfake detector by paraphrasing the script rather than manipulating the audio, so anti-spoofing evaluations that only add acoustic noise miss a practical attack surface.","Detector–voice pairing matters: the same detector can be nearly immune for one synthetic voice and highly vulnerable for another, so robustness claims should be reported per text-to-speech system and voice.","Commercial services are not exempt: one commercial API in the paper's tests falls from 100% to 32% accuracy under transcript attacks, and the celebrity-scam case study flips spoof utterances from below 1% to 58–90% bona-fide probability.","Feature analysis points to a usable defense signal: detectors with high Audio Encoder Similarity for a voice and balanced spoof and bona-fide F1 scores are harder to attack, so these properties could guide model selection or adaptation.","The predictive models trained on the engineered features reach about 76% F1 in approximating detector decisions, suggesting gray-box or black-box attackers could optimize transcript edits through a proxy without querying the real detector."],"supporting_citations":[{"why":"Supplies the AASIST-2 detector, the open-source model whose classifications are shown to flip under rewording.","marker":"Tak et al. (2022)"},{"why":"Supplies the RawNet-2 detector, a second open-source target used in the attack evaluation.","marker":"Tak et al. (2021)"},{"why":"Supplies the CLAD detector, the third open-source target, which shows high vulnerability on several voices.","marker":"Wu et al. (2024)"},{"why":"Provides the VoiceWukong test set from which the 1,439 evaluation transcripts are drawn.","marker":"Yan et al. (2024)"},{"why":"Provides the TextFooler substitution strategy, which yields the highest attack success rates for many detector-voice pairs.","marker":"Jin et al. (2020)"},{"why":"Provides the PWWS WordNet synonym attack strategy used as one of the four perturbation methods.","marker":"Ren et al. (2019)"},{"why":"Provides the BAE masked-language-model strategy used to propose plausible word replacements.","marker":"Garg and Ramakrishnan (2020)"},{"why":"Provides the BERTAttack strategy, the fourth perturbation method in the attack pipeline.","marker":"Li et al. (2020)"},{"why":"Provides the Universal Sentence Encoder used to enforce the semantic-preservation constraint in the attack.","marker":"Cer et al. (2018)"},{"why":"Provides the batch-norm recalibration technique used to adapt open-source detectors to each text-to-speech voice, the load-bearing assumption behind the quantitative claims.","marker":"Shomron and Weiser (2020)"}],"fun_headline_variants":["Word swaps flip a deepfake detector from 100% to 32%","Linguistic tweaks bypass commercial deepfake audio defenses","Text edits fool anti-spoofing systems 82% of the time","Synonym-level changes crash a commercial deepfake detector","Rewording TTS input evades spoof detectors"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The open-source attack numbers come from detectors that were batch-norm calibrated to each text-to-speech voice until accuracy exceeded 90%, and without a comparison to uncalibrated detectors it is possible that this calibration, not the detector itself, is what makes rewording so effective.","fun_headline_variants_meta":{"raw":{"variants":["Word swaps flip a deepfake detector from 100% to 32%","Linguistic tweaks bypass commercial deepfake audio defenses","Text edits fool anti-spoofing systems 82% of the time","Synonym-level changes crash a commercial deepfake detector","Rewording TTS input evades spoof detectors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000773,"raw_usage":{"total_tokens":3430,"prompt_tokens":964,"completion_tokens":2466,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":580,"completion_tokens_details":{"reasoning_tokens":2380}},"tokens_in":580,"tokens_out":2466,"duration_ms":19090,"temperature":1.0,"reasoning_tokens":2380,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:45:14.653207+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same transcript-level attack against the three open-source detectors without any per-voice batch-norm calibration, or with a single fixed calibration learned on one voice, and check whether attack success rates still exceed 60%; if they collapse, the linguistic sensitivity is a calibration artifact rather than a property of the detectors.","supporting_citations":[],"review_version":1}