{"id":"a52fa6ea-8a6c-4ce1-9803-c9b0b6af8ed0","arxiv_id":"2411.17349","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Across Whisper and Wav2Vec 2.0, larger ASR models improve speech deepfake detection up to a medium size, after which performance plateaus or declines.","lead":"This paper tests whether bigger, better speech-recognition models also make better speech deepfake detectors. It finds that detection improves only up to a medium model size, then plateaus, so scaling ASR models alone will not keep improving fake-audio detection.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The plateau conclusion rests on one fixed training pipeline; if larger ASR models simply need different hyperparameters, the headline result is an artifact.","rationale":"The reader's weakest-assumption analysis identifies the fixed training pipeline as the main threat to the central claim, and I agree. The paper's headline conclusion is that ASR performance and deepfake detection are only partially aligned, with a plateau beyond medium-sized models. All of the quantitative evidence for this plateau comes from comparing models trained with identical hyperparameters. Since the models differ not only in size but also in embedding dimensionality, number of layers, and pretraining data, the fixed pipeline does not by itself separate 'size' from 'optimization adequacy.' The proposed test directly addresses this by allowing each model to be tuned to its own reasonable operating point; if the plateau disappears, the conclusions in Section V would need to be substantially revised. This is a correctness risk, not a stylistic or novelty objection. The reader's verdict of CONDITIONAL is appropriate, and the specific test below would determine whether the condition should be lifted or made stricter. Secondary issues such as lack of repeated runs and confidence intervals would also be resolved by the same experiment. The overlap matrix is interesting but not decisive for the plateau question, because conditional miss rates are compatible with either a true non-hierarchy or simply different error patterns under suboptimal training.","tokens_in":20631,"tokens_out":5017,"duration_ms":53768,"concrete_test":"For each Whisper and Wav2Vec 2.0 size, run a small per-model hyperparameter search on the ASVspoof2019 dev partition: learning rates {3e-5, 1e-4, 3e-4}, classifier hidden widths {64, 128, 256}, and early-stopping patience {10, 20}, with 3 seeds per configuration, selecting by dev EER. Then evaluate the selected models on all five test corpora and report mean and standard deviation of EER/AUC. If the large model's average EER after tuning is not statistically better than the medium or small model, the plateau survives; if it is, the fixed-pipeline assumption in Section III-B is the cause and the headline conclusion should be weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim, stated in the Conclusions, is that speech-deepfake detection performance improves with ASR model size only up to a medium scale and then plateaus. Section III-B says all versions of Whisper and Wav2Vec 2.0 use the same training pipeline (lr=1e-4, batch size 64, 100 epochs, early stopping 10, one classifier architecture per family) \"to ensure that the differences in performance are due solely to the initial pre-trained weights.\" This isolation is valid only if the same hyperparameters are near-optimal for every model size. They are not obviously so: Whisper embeddings have dimension X in {384, 512, 768, 1024, 1280} and sequence length 1500, while Wav2Vec 2.0 models have 12 versus 24 transformer layers and different hidden-state counts, so the trainable heads see very different input geometries and the weighted-sum pooling has a different number of terms. A single learning rate and epoch budget can over- or under-fit different sizes, making the plateau visible in Table I and Figure 4 an artifact of optimization rather than a genuine property of ASR representations. The overlap analysis in Figure 5 does not settle this: even an undertrained large model can miss different tracks than a small model. Because the paper's practical guidance is to expect saturation and not rely on the largest ASR model, the fixed-pipeline assumption is the load-bearing point.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether larger ASR-pretrained models (Whisper and Wav2Vec 2.0) yield better speech deepfake detectors. The authors freeze each pretrained backbone, train small task-specific classifier heads on the ASVspoof 2019 LA training partition, and evaluate on ASVspoof 2021 DF, In-the-Wild, TIMIT-TTS/LJspeech, and FakeOrReal. They report EER/AUC values, show an overlap matrix of detection errors across Whisper sizes, and conclude that detection performance improves with model size up to a medium scale and then plateaus or degrades, so ASR performance is only partially aligned with deepfake detection.","tokens_in":20879,"tokens_out":6474,"duration_ms":64507,"significance":"This is a timely and practically relevant empirical question. The study covers two major self-supervised ASR families and multiple out-of-distribution datasets, and the overlap matrix in Figure 5 is a useful diagnostic for showing that detection failures are not nested by model size. If the conclusions survive a more rigorous comparison, they would caution against selecting ASR backbones solely by parameter count or WER. The main weaknesses are the absence of uncertainty quantification and the lack of evidence that a single training pipeline is fair for all model sizes; both directly affect the central claim.","major_comments":[{"comment":"The claim that the same training pipeline 'ensures that the differences in performance are due solely to the initial pre-trained weights' is not justified. With Whisper parameter counts ranging from 39M to 1550M and embedding widths from 384 to 1280, and with Wav2Vec 2.0 models differing in depth (12 vs 24 layers), a single learning rate (1e-4), batch size (64), epoch budget (100), early-stopping patience (10), and one classifier architecture per family are unlikely to be near-optimal for all sizes. The plateau in Table I and Figure 4 could therefore reflect over- or under-trained large models rather than a genuine saturation of ASR knowledge for deepfake detection. Please provide per-size hyperparameter sweeps or convergence evidence, or at least discuss this confound explicitly.","section":"Section III-B, Table I, Figure 4"},{"comment":"The central 'plateau' claim rests on small differences that are reported without any uncertainty estimate. For instance, on ASVspoof 2019 the Whisper medium EER is 1.58% and large is 2.00%; on the average row, medium is 16.08% and large is 14.46%. These differences are plausibly within run-to-run variability of training a small classifier head. Without multiple seeds, confidence intervals, or a paired significance test, the conclusion that larger models do not consistently outperform medium models is not statistically supported.","section":"Table I"},{"comment":"The overlap matrix is a valuable diagnostic, but the text does not define the normalization (row-wise vs column-wise) or the population (e.g., tracks missed by the row model, or by the column model). More importantly, the observed non-hierarchy cannot distinguish a genuine lack of nesting from an undertrained large model; this connects to the concern in Section III-B. If the training-pipeline confound is resolved, the overlap result would strengthen the paper, but as it stands it does not settle the central claim.","section":"Figure 5 and Section IV"}],"minor_comments":[{"comment":"The statement 'WER VALUES ARE TAKEN FROM [17]' is inaccurate for Wav2Vec 2.0; reference [17] is the Whisper paper. Please cite the original WER source for each model family and specify the test set (presumably LibriSpeech) and decoding details.","section":"Table I caption"},{"comment":"For Wav2Vec 2.0, the weighted-sum pooling over hidden layers introduces a different number of trainable weights per model (12 vs 24 layers), so classifier capacity is not matched across model sizes; please state how this is handled in the comparison.","section":"Section III-B"},{"comment":"The phrase 'Future words could focus' should read 'Future work could focus'.","section":"Section V"},{"comment":"Please add a colorbar and a caption that explains whether the entries are percentages of the row model's missed tracks or of the column model's missed tracks, and clarify the denominator used for normalization.","section":"Figure 5"},{"comment":"The text refers to 'ASR methods,' but Whisper is an encoder-decoder trained with multiple speech objectives; using 'ASR-pretrained models' would be more precise.","section":"Abstract and Introduction"},{"comment":"The paper does not mention whether code or trained detectors will be released; adding a reproducibility statement or link would be helpful.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is a useful empirical contribution, but the headline conclusion requires stronger support. The fixed-pipeline confound in Section III-B is the main technical gap; adding a hyperparameter sensitivity analysis for the largest models and uncertainty quantification would address it. The WER citation error in Table I should also be fixed. I see no circularity or fabrication issues."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, read this if you work on speech deepfake detection or on transfer from self-supervised ASR features. The paper asks whether better ASR models (lower WER) give better deepfake detectors, using frozen Whisper and Wav2Vec 2.0 embeddings plus a trained head. On six datasets, they see that detection improves from tiny to small/medium, then plateaus. The overlap matrix is a nice touch: larger models miss tracks that smaller ones catch, so the capabilities aren't simply nested. That part is genuinely new, as far as I know; prior work just applied one Whisper or Wav2Vec model and didn't compare sizes systematically.\n\nWhat's good: the eval covers ASVspoof 2019/2021, In-the-Wild, TIMIT-TTS, FakeOrReal, which is more than most papers in this space. They keep the feature extractor frozen, which is a clean setup for isolating the representation's contribution. They also report the WER numbers from the original model cards, so the comparison is grounded.\n\nThe soft spot is the one the stress-test flags. All models share the same training pipeline (lr 1e-4, batch 64, 100 epochs, early stopping 10). For Whisper, embedding dimensions range 384 to 1280; for Wav2Vec, 768 to 1024 and 12 vs 24 layers. It's not obvious that one LR and one epoch budget are equally good for all. The plateau could be an undertrained large model artifact. The authors claim the pipeline isolates the pre-trained weights, but that only holds if the hyperparameters are near-optimal for every size. They didn't check. No confidence intervals or repeated runs either, so the differences between small and medium/large (e.g., 13.23 vs 14.46 average EER) could be noise. The overlap matrix is interesting but doesn't rescue the claim, because an undertrained model can miss different tracks for the wrong reason. Also, the xls-r multilingual confound is acknowledged by the authors, but it muddies the Wav2Vec comparison.\n\nThat said, I don't think the concern is fatal. The paper is an empirical study, and the central observation is visible across multiple datasets. The fix is straightforward: vary the learning rate or do a small sweep per model size, report standard deviations over a few seeds, and show the plateau persists. If it does, the conclusion is solid. If it doesn't, the conclusion is an artifact. Either way, the paper deserves a serious referee. I'd send it to review, with a request for the robustness check.\n\nYou should also page the authors for code; the paper doesn't ship any, so reproducing the results would require reimplementation.\n\nRecommendation: engage with it, but push for the hyperparameter sensitivity analysis and variance estimates.","headline":"Honest, useful empirical study on ASR model size vs. deepfake detection, but the plateau conclusion needs a robustness check before it becomes guidance.","tokens_in":21455,"tokens_out":3281,"would_cite":true,"duration_ms":28523,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Bigger ASR models do not reliably make better speech deepfake detectors.","keywords":["speech deepfake detection","automatic speech recognition","self-supervised models","Whisper","Wav2Vec 2.0","model scaling","anti-spoofing","audio forensics"],"falsifier":"A model-size sweep with per-size hyperparameter tuning that shows monotonic improvement through the largest model, or an overlap matrix in which every smaller model's correct detections are a subset of the larger model's, would contradict the saturation and partial-alignment conclusion.","tokens_in":20413,"feed_emoji":"🎙️","tokens_out":9240,"duration_ms":95697,"temperature":0.7,"pith_summary":"This paper asks whether a speech deepfake detector improves when its backbone improves at automatic speech recognition. It answers by building detectors from every released size of Whisper and Wav2Vec 2.0, keeping the backbones frozen and training only a small classifier, and measuring detection across ASVspoof 2019 and four datasets the detectors have not seen. Detection does improve as models grow from tiny to small or medium, but then it plateaus: the largest Whisper and Wav2Vec 2.0 variants do not consistently beat their medium counterparts, and many tracks the largest Whisper model misses are caught by the tiny one. The paper concludes that ASR skill and deepfake detection skill are only partially aligned, so choosing a feature extractor by ASR quality or parameter count alone is not a reliable strategy.","feed_headline":"Bigger speech-recognition models stop helping deepfake detection","feed_subtitle":"Speech deepfake detectors plateau past medium-sized Whisper and Wav2Vec 2.0 backbones, and tiny models catch tracks the largest one misses.","key_machinery":"The mechanism is a controlled transfer experiment built on model-size families of frozen ASR feature extractors. Whisper provides five sizes, from tiny at 39M parameters to large at 1550M, and Wav2Vec 2.0 provides three, base at 95M, large at 317M, and xls-r at 317M; each variant feeds embeddings into its own small trainable classifier head while the ASR weights stay frozen, and the training pipeline is held fixed so that differences in detection accuracy are attributed to the pre-trained representation. Two instruments make the argument visible: a size-versus-EER sweep across five datasets, and a detection-overlap matrix that records, for each pair of Whisper sizes, what fraction of tracks missed by one model are caught by the other. That overlap matrix is what shows the relationship is partial rather than hierarchical.","core_discovery":"On the paper's own terms, the discovery is a saturation of transfer: the relationship between ASR performance and speech deepfake detection is real but bounded. In Table I, Whisper's average EER across the five test setups falls from 20.60% for tiny to 13.23% for small and then rises to 16.08% for medium and 14.46% for large, even though the large model has about six times the parameters of the small model and a better WER. Wav2Vec 2.0 large beats base on every dataset, but xls-r, which shares the large model's size and was trained multilingually, often performs worse. The overlap analysis reinforces the point: detection across sizes is not hierarchical, with 75.6% of tracks missed by Whisper large correctly identified by Whisper tiny. The authors describe this as a bottleneck effect: beyond a certain model size, more parameters and lower WER do not translate into more accurate deepfake detection.","pith_inferences":["A natural extension the paper does not run is an ensemble across model sizes: since the overlap matrix shows error sets are complementary, combining a tiny model with a large one could raise recall beyond any single backbone.","The plateau might partly be an artifact of the fixed training pipeline; per-size learning rates and regularization could push the largest models higher, which would reframe the result as a training-equivalence finding rather than a ceiling on ASR knowledge.","The xls-r result suggests that matching the pretraining language distribution to the target speech may matter more than model scale, a hypothesis that could be tested directly by evaluating the same model sizes on non-English deepfake data.","For practical deployment, the paper implies that model selection should be driven by detection benchmarks on the target domain, not by recognition leaderboards."],"forward_implications":["Choosing a speech deepfake detector's ASR backbone by WER or parameter count alone is not reliable; the authors' sweeps show detection performance stabilizes after the small-to-medium sizes.","Larger ASR backbones leave recoverable errors: on average, 75.6% of the tracks Whisper large misses are caught by Whisper tiny, so a single large model is not a superset of smaller models' detections.","A model trained on multilingual data, Wav2Vec 2.0 xls-r, can underperform an English-trained model of equal size on English deepfake benchmarks, so pretraining distribution matters at least as much as model size.","Because the plateau appears across two different ASR model families and across five evaluation datasets, it behaves like a general design constraint for this detector family rather than a quirk of one model.","The strongest practical reading is that the sweet spot for cost and recall sits around the small and medium backbones, with the largest variants adding compute without a proportional detection gain."],"supporting_citations":[{"why":"Supplies the Whisper model family in five sizes and the WER values the paper compares against detection performance.","marker":"[17]"},{"why":"Supplies the Wav2Vec 2.0 base, large, and xls-r representations used as frozen embedding extractors.","marker":"[18]"},{"why":"Shows Wav2Vec 2.0 embeddings used for spoofing and deepfake detection, the approach this paper scales across model sizes.","marker":"[14]"},{"why":"Shows Whisper features used for deepfake detection, the other approach this paper scales across model sizes.","marker":"[15]"},{"why":"ASVspoof 2019 LA is the training and primary evaluation corpus with an open-set eval partition.","marker":"[20]"},{"why":"ASVspoof 2021 DF provides the codec-distorted out-of-distribution test set.","marker":"[21]"},{"why":"AISEC In-the-Wild provides real-world synthetic speech for generalization testing.","marker":"[22]"},{"why":"TIMIT-TTS provides fake speech from ten TTS generators for the single-speaker generalization test.","marker":"[23]"},{"why":"LJspeech provides the authentic speaker-matched counterpart to TIMIT-TTS.","marker":"[24]"},{"why":"FakeOrReal provides an additional real-versus-TTS benchmark for generalization.","marker":"[25]"}],"fun_headline_variants":["ASR model size doesn't guarantee better deepfake detection","Bigger speech models plateau in deepfake detection","Tiny speech models catch fakes big ones miss","Deepfake detection saturates with ASR model growth","Whisper and Wav2Vec: bigger not better for deepfake detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The plateau conclusion rests on the assumption that one fixed training pipeline is fair for every model size; if larger ASR models simply need different learning rates or regularization, the observed saturation would be an artifact of undertraining rather than a limit of ASR knowledge.","fun_headline_variants_meta":{"raw":{"variants":["ASR model size doesn't guarantee better deepfake detection","Bigger speech models plateau in deepfake detection","Tiny speech models catch fakes big ones miss","Deepfake detection saturates with ASR model growth","Whisper and Wav2Vec: bigger not better for deepfake detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000278,"raw_usage":{"total_tokens":1669,"prompt_tokens":975,"completion_tokens":694,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":591,"completion_tokens_details":{"reasoning_tokens":621}},"tokens_in":591,"tokens_out":694,"duration_ms":5726,"temperature":1.0,"reasoning_tokens":621,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:12:42.448946+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A model-size sweep with per-size hyperparameter tuning that shows monotonic improvement through the largest model, or an overlap matrix in which every smaller model's correct detections are a subset of the larger model's, would contradict the saturation and partial-alignment conclusion.","supporting_citations":[{"cited_title":"Robust speech recognition via large-scale weak super- vision,","cited_arxiv_id":null,"evidence_quote":"Supplies the Whisper model family in five sizes and the WER values the paper compares against detection performance."},{"cited_title":"Automatic Speaker Verification Spoofing and Deepfake Detection Us- ing Wav2vec 2.0 and Data Augmentation,","cited_arxiv_id":null,"evidence_quote":"Shows Wav2Vec 2.0 embeddings used for spoofing and deepfake detection, the approach this paper scales across model sizes."},{"cited_title":"Improved DeepFake Detection Using Whisper Features,","cited_arxiv_id":null,"evidence_quote":"Shows Whisper features used for deepfake detection, the other approach this paper scales across model sizes."},{"cited_title":"ASVspoof 2019: Future horizons in spoofed and fake audio detection,","cited_arxiv_id":null,"evidence_quote":"ASVspoof 2019 LA is the training and primary evaluation corpus with an open-set eval partition."},{"cited_title":"ASVspoof 2021: accelerating progress in spoofed and deepfake speech detection,","cited_arxiv_id":null,"evidence_quote":"ASVspoof 2021 DF provides the codec-distorted out-of-distribution test set."},{"cited_title":"Does audio deepfake detection generalize?","cited_arxiv_id":null,"evidence_quote":"AISEC In-the-Wild provides real-world synthetic speech for generalization testing."},{"cited_title":"The LJSpeech dataset,","cited_arxiv_id":null,"evidence_quote":"LJspeech provides the authentic speaker-matched counterpart to TIMIT-TTS."},{"cited_title":"FoR: A dataset for synthetic speech detec- tion,","cited_arxiv_id":null,"evidence_quote":"FakeOrReal provides an additional real-versus-TTS benchmark for generalization."}],"review_version":1}