{"id":"04240169-bfeb-4aa2-8507-762fd60abfbe","arxiv_id":"2506.14398","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A unified evaluation framework shows watermarking models perfectly separate real from fake speech in a clean lab setup, but all four tested defenses lose accuracy under channel noise, codecs, and pitch changes.","lead":"This paper compares two ways to defend against fake audio: passive detectors that judge whether audio is AI-generated, and proactive watermarking that hides a marker in audio. It proposes one scoring system so both can be tested on the same data and distortions, and finds every method degrades under real-world noise and codecs.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The proactive models' reported 0% EER is obtained by watermarking fake test utterances with a defender-chosen M_fake; this is not a realistic proactive-defense threat model and may not hold on unwatermarked fakes.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the proactive evaluation protocol gives the defender the ability to watermark fake test utterances, which is not possible in a real attack. This is not a minor modeling choice; it directly supports the paper's strongest claim that proactive models can perfectly distinguish real from spoofed speech. If the proposed concrete test shows performance degradation on unwatermarked fakes, the central comparison and the conclusion that Timbre is the most robust defense would need substantial revision. The verdict should remain CONDITIONAL: the paper presents a useful framework and the passive-detector results are still informative, but the proactive results must be re-run under a realistic protocol before the comparison can be accepted as a guide for deployment. No additional concern rises to the same level; the lack of error bars and the loose 'adversarial attack' terminology are secondary and already noted by the reader.","tokens_in":9595,"tokens_out":4739,"duration_ms":55949,"concrete_test":"Using the released code, reproduce Table 1 with one change: embed M_real into the real utterances of the ASVspoof 2019 and 2021 LA evaluation sets, but leave the fake utterances unwatermarked; compute EER with the same Eq. (1) scoring pipeline. If the EER moves from the reported 0.00% to a materially nonzero value, the reported perfect detection is an artifact of watermarking fake data, and the paper must present the realistic-protocol results separately and soften the central comparison.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim (Section 4.2: proactive models achieve 0% EER on ASVspoof 2019/2021 LA) rests on Section 3.2's protocol, where real utterances are watermarked with M_real and fake utterances with M_fake, and classification uses Eq. (1) to decide which message is recovered. This measures the watermark detector's ability to recover an embedded message, not its ability to detect deepfake speech. In any realistic proactive deployment, only genuine speech is watermarked by the defender; attacker-produced fake speech arrives unwatermarked. Embedding M_fake into test fakes assumes an oracle that can mark the attack before it is distributed. Therefore, the reported perfect separation does not establish that Timbre or AudioSeal can distinguish watermarked real speech from unwatermarked fake speech, and the headline comparison between proactive and passive defenses is not yet valid for the realistic threat model. The paper's use of 'adversarial attacks' for random distortions is secondary; the unrealistic watermarking protocol is the load-bearing issue.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a unified evaluation protocol for comparing passive deepfake detectors (AASIST, SSL-AASIST) with proactive watermarking models (Timbre, AudioSeal) on the ASVspoof 2019 and 2021 LA datasets. In the protocol, real utterances are watermarked with message M_real and fake utterances with M_fake, and the bit-level watermark detector scores are merged via Eq. (1) into a continuous real-vs-fake score. The study reports clean-condition EERs and EERs under 17 transmission/manipulation conditions, concluding that watermarking models can achieve 0% EER in clean conditions and that Timbre is the most robust model on average under distortions.","tokens_in":9786,"tokens_out":8743,"duration_ms":92512,"significance":"If the proposed protocol corresponded to a realistic deployment scenario, the paper would provide a valuable common testbed for two research communities, and the open-source code and use of standard ASVspoof protocols are strengths. However, the central protocol is not a realistic proactive-defense threat model: fake utterances are watermarked by the defender, which an attacker would not do. The headline 0% EER and the robustness ranking are therefore not established for the claimed task, although the experimental infrastructure is potentially reusable after a threat-model correction.","major_comments":[{"comment":"The protocol watermarks both real and fake test utterances with defender-chosen messages and then classifies by which message is recovered. In any realistic proactive deployment, only genuine content is watermarked by the content owner; fake speech is generated by an attacker and arrives unwatermarked. The reported 0% EER in Section 4.2 and Table 1 therefore demonstrates watermark-message recovery on defender-watermarked fakes, not the ability to distinguish watermarked real speech from unwatermarked deepfake speech. Please add an evaluation condition in which only real utterances are watermarked and fake utterances are left unwatermarked, or explicitly scope all claims to the oracle-assisted protocol used here.","section":"Section 3.2, Eq. (1)"},{"comment":"The conclusion that Timbre appears to be the most robust is based on the average EER over conditions that mix fully unseen and partially seen distortions. Because Timbre has no partially-seen asterisks in Table 1 while SSL-AASIST and AudioSeal do, the pooled average is not a like-for-like robustness measure. Please report averages separately for fully unseen conditions or provide a statistical comparison (for example, bootstrap confidence intervals over utterances) before ranking the models.","section":"Section 4.2, Table 1"}],"minor_comments":[{"comment":"The abstract says the paper analyzes robustness against 'adversarial attacks', but all conditions in Section 3.3 are non-adaptive random distortions and transmissions rather than adversarially optimized perturbations. Please use 'distortions and transmissions' or another accurate term.","section":"Abstract and Section 3.3"},{"comment":"The two messages are described as 'bit-wise disjoint', but the score formula in Eq. (1) gives zero contribution from bit positions where both M_real and M_fake contain 0. Using complementary messages (M_fake = bitwise NOT M_real) would avoid wasting watermark capacity; please clarify the message design or explain why this does not affect the results.","section":"Section 3.2, Eq. (1)"},{"comment":"EER values above 50%, such as AudioSeal's 97.40% under DAC, mean that the score convention is effectively inverted for that condition; the same discrimination would give an EER below 50% if the decision rule were flipped. This does not change the qualitative fragility finding, but the interpretation should be stated more carefully.","section":"Section 4.2, Table 1"},{"comment":"The passive detectors use released checkpoints while the proactive models are trained by the authors, but no training details such as number of epochs, learning rate, or hyperparameters are given in the paper. Since the code is available, please add a brief training setup description or point to a specific configuration file.","section":"Section 4.1"},{"comment":"For the ASVspoof 2021 LA 'clean' row in Table 1, the original corpus already contains lossy transmission channels in about 85% of the data. Please specify whether watermarking is applied before or after the channel degradation already present in the corpus, so the reader can interpret the 'None from Section 3.3' condition correctly.","section":"Section 4.1 and 3.4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is from a reputable group and the experimental infrastructure is solid in terms of datasets and code release. The main problem is that the proactive evaluation protocol watermarks the fake test utterances, which removes the core difficulty of real proactive defense. I believe an additional experiment with unwatermarked fakes and a revised framing could fix this within the manuscript's scope, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is worth your attention for two things: Eq. (1), a simple and elegant way to convert multi-bit watermark detector outputs into a real/fake score, and the first head-to-head robustness comparison of AASIST, SSL-AASIST, AudioSeal, and Timbre across 16 common distortions on ASVspoof 2019/2021 LA. The experimental design is generally careful: they train all models on the same data, use shared metrics, exclude augmentations seen in training, and include both partially-seen and unseen conditions. The code is public. That's real value for the subfield.\n\nThe soft spot, as the stress-test note says, is that the proactive evaluation protocol watermarks both real and fake test utterances with defender-chosen messages. In a realistic proactive defense, only genuine speech carries the defender's watermark; fakes arrive unwatermarked. By embedding M_fake into fakes, the task becomes message recovery, not deepfake detection. That explains the perfect 0% clean EER and almost certainly inflates the robustness of Timbre and AudioSeal relative to a deployment scenario. The authors should report a second condition where fakes are unwatermarked and classification uses watermark presence or confidence, not message identity. Until then, the absolute numbers and the \"proactive vs passive\" ranking are not yet reliable.\n\nOther, smaller issues: results are single-run without error bars, which matters for models with stochastic training; and calling these random distortions 'adversarial attacks' is loose, though the paper mostly uses 'transmission and manipulation.' The paper itself is honest about the limitations, which counts in its favor.\n\nBottom line: the framework and the Eq. (1) trick are worth publishing and building on; the empirical conclusions need an additional experiment to be fair. I'd send this to a venue with a revision-friendly review process, not desk reject it. Worth a reading group discussion on methodology.","headline":"A genuinely useful benchmark for comparing watermarking and deepfake detectors, with a clean score-mapping idea—but the proactive protocol embeds watermarks into fake speech, which flatters the 0% EER and muddies the robustness comparison.","tokens_in":10318,"tokens_out":2500,"would_cite":true,"duration_ms":29326,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proposes a unified evaluation framework that lets proactive watermarking models and passive deepfake detectors be compared head-to-head, and reports that watermarking models can reach 0% EER on clean ASVspoof 2019 and 2021 LA…","keywords":["deepfake speech detection","audio watermarking","proactive defense","passive detection","ASVspoof","equal error rate","robustness evaluation","speech anti-spoofing"],"falsifier":"Run the same clean-set evaluation but watermark only the real utterances with $M_{real}$ and leave all fake utterances unwatermarked, classifying an utterance as real only if the recovered message matches $M_{real}$; if the EER rises far above 0%, the paper's claim of perfect proactive detection under realistic conditions would be undermined.","tokens_in":9418,"feed_emoji":"🔊","tokens_out":5373,"duration_ms":51629,"temperature":0.7,"pith_summary":"The paper's goal is to make proactive watermarking models and passive deepfake detectors directly comparable for binary real-vs-fake speech classification. It sets up a common protocol: all models train and test on the same ASVspoof LA datasets, all outputs get converted to a single realness score, and all are scored with equal error rate (EER). Under this protocol, the watermarking models Timbre and AudioSeal classify real and spoofed utterances perfectly on clean audio (0% EER), matching or beating the passive detectors AASIST and SSL-AASIST. Once transmissions and manipulations are applied, every model degrades; Timbre is on average the most robust, but codecs and pitch shift still break it. A careful reader would care because choosing a defense currently lacks a fair basis, and this framework is a first attempt at providing one.","feed_headline":"Watermarking detects deepfake speech with 0% error in clean tests","feed_subtitle":"A shared evaluation protocol shows proactive watermarks match passive detectors, but all models fail under codecs and pitch shifts.","key_machinery":"The load-bearing mechanism is the scoring formula in Eq. (1), which converts a watermarking model's per-bit logits into a single realness score without retraining: two bit-wise disjoint random messages $M_{real}$ and $M_{fake}$ are assigned to real and fake utterances, and the score is $s = \\frac{1}{L} \\sum_{l=1}^{L} (s_{w,l} q(m_{real,l}) - s_{w,l} q(m_{fake,l}))$ with $q(1)=1$, $q(0)=-1$, so that agreement with $M_{real}$ pushes the score up and agreement with $M_{fake}$ pushes it down. This, together with the shared evaluation protocol (same datasets, same EER metric, and a common set of transmissions and manipulations applied after watermarking), is what makes the head-to-head comparison possible.","core_discovery":"The central discovery, stated on the paper's own terms, is that under a unified evaluation protocol, proactive watermarking models can perfectly separate real from spoofed speech on the ASVspoof 2019 LA and ASVspoof 2021 LA evaluation sets, reaching 0% EER even for unseen speakers and unseen generators, and that no model, proactive or passive, remains reliable under all distorted channel and manipulation conditions tested. In particular, the paper finds Timbre the most robust on average (8.87% and 9.02% EER across distorted conditions), while AudioSeal collapses on similar codecs despite its training augmentation, and AASIST degrades sharply on the 2021 LA transmission-heavy set.","pith_inferences":["An implication the authors leave implicit is that the protocol's 0% EER depends on watermarking the fake utterances too; in a real proactive defense only genuine content carries a watermark, so an attacker who simply emits unwatermarked deepfakes would bypass the classifier, and the reported perfect separation may not transfer.","The scoring formula treats each bit's logit as evidence for one of two messages; a natural testable extension is to evaluate whether the choice of message pair (e.g., Hamming distance or bit length) changes the EER, since Eq. (1) should be invariant to message complement in theory but bit-level calibration may not be.","Because proactive and passive models fail on different distortions (e.g., Timbre on pitch shift, AASIST on time stretch), an implicit avenue is fusing both score types; the paper does not attempt this fusion.","The framework could be extended to a true scenario where fakes are not watermarked, likely changing the ranking of proactive versus passive methods and providing a more direct measure of real-world usability."],"forward_implications":["If the framework is correct, defense selection can be based on measured EER across a shared protocol rather than on community-specific benchmarks.","The perfect clean-set performance of watermarking models implies that when the defender controls both embedding and extraction, binary deepfake detection is solved in the clean case; the hard remaining problem is robustness.","Timbre's robustness to time-stretch but vulnerability to pitch shift and codecs indicates that frequency-domain watermarking has a specific weakness profile that should be addressed directly.","AudioSeal's failure on similar codecs despite augmentation suggests that model-specific augmentation does not generalize across codec families, so robustness must be evaluated rather than assumed.","All models degrade on the 2021 LA set, indicating that real-world transmission is the dominant challenge for deployment.","The protocol can be reused to benchmark future watermarking models and detectors on equal footing.","The reported EERs provide concrete baselines: Timbre at 8.87% and SSL-AASIST at 12.41% average on distorted 2019 LA data."],"supporting_citations":[{"why":"Provides AASIST, one of the two passive deepfake detectors, trained on ASVspoof 2019 LA with official checkpoints.","marker":"[1]"},{"why":"Supplies the ASVspoof 2019 LA training, development, and test sets and protocol used for both model training and evaluation.","marker":"[3]"},{"why":"Supplies Timbre, one of the two proactive watermarking models, embedding watermarks in the spectral domain.","marker":"[5]"},{"why":"Supplies AudioSeal, the other proactive watermarking model, an encoder-decoder trained with data augmentation including codecs.","marker":"[6]"},{"why":"Supplies the ASVspoof 2021 LA evaluation set, which contains transmission-heavy conditions used to test generalization.","marker":"[19]"},{"why":"Supplies SSL-AASIST, a passive detector with a self-supervised front-end, which proves robust to additive noise.","marker":"[20]"}],"fun_headline_variants":["Proactive watermarks perfect on clean deepfake speech sets","0% EER for watermarked deepfake speech on clean tests","Watermarks match passive detectors on clean deepfake speech","Perfect clean-set detection for proactive watermarks","Shared protocol shows watermarks perfect on clean deepfake speech"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole comparison rests on the assumption that a watermarking model can be evaluated as a deepfake detector by watermarking both real and fake utterances with different messages and classifying by which message is recovered; in actual proactive defense, only genuine content carries the defender's watermark, so this protocol may not reflect the real attack scenario.","fun_headline_variants_meta":{"raw":{"variants":["Proactive watermarks perfect on clean deepfake speech sets","0% EER for watermarked deepfake speech on clean tests","Watermarks match passive detectors on clean deepfake speech","Perfect clean-set detection for proactive watermarks","Shared protocol shows watermarks perfect on clean deepfake speech"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001455,"raw_usage":{"total_tokens":5780,"prompt_tokens":794,"completion_tokens":4986,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":410,"completion_tokens_details":{"reasoning_tokens":4907}},"tokens_in":410,"tokens_out":4986,"duration_ms":32241,"temperature":1.0,"reasoning_tokens":4907,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:17:34.318809+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same clean-set evaluation but watermark only the real utterances with $M_{real}$ and leave all fake utterances unwatermarked, classifying an utterance as real only if the recovered message matches $M_{real}$; if the EER rises far above 0%, the paper's claim of perfect proactive detection under realistic conditions would be undermined.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides AASIST, one of the two passive deepfake detectors, trained on ASVspoof 2019 LA with official checkpoints."},{"cited_title":"1, upon which we compare deepfake detectors and watermarking mod- els for binary deepfake detection","cited_arxiv_id":null,"evidence_quote":"Supplies the ASVspoof 2019 LA training, development, and test sets and protocol used for both model training and evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies Timbre, one of the two proactive watermarking models, embedding watermarks in the spectral domain."},{"cited_title":"This study was partially supported by JST AIP Acceleration Research (JP- MJCR24U3), MEXT KAKENHI Grant (24H00732), and JST PRESTO (JPMJPR23P9)","cited_arxiv_id":null,"evidence_quote":"Supplies AudioSeal, the other proactive watermarking model, an encoder-decoder trained with data augmentation including codecs."},{"cited_title":"RawBoost: A raw data boosting and augmentation method applied to automatic speaker verification anti-spoofing,","cited_arxiv_id":null,"evidence_quote":"Supplies the ASVspoof 2021 LA evaluation set, which contains transmission-heavy conditions used to test generalization."},{"cited_title":"ISO/IEC 30 107-1:2023, 2023","cited_arxiv_id":null,"evidence_quote":"Supplies SSL-AASIST, a passive detector with a self-supervised front-end, which proves robust to additive noise."}],"review_version":1}