Pith. sign in

REVIEW 3 major objections 6 minor 25 references

Comparative Analysis of ASR Methods for Speech Deepfake Detection

T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Pith's one-line read Bigger ASR models do not reliably make better speech deepfake detectors.

desk verdict Honest, useful empirical study on ASR model size vs. deepfake detection, but the plateau conclusion needs a robustness check before it becomes guidance. read the letter →

arxiv 2411.17349 v1 pith:7PR37BDJ submitted 2024-11-26 cs.SD eess.AS

classification cs.SDeess.AS
keywords speechdeepfakedetectionautomaticrecognitionself-supervisedmodelsWhisperWav2Vec2.0modelscalinganti-spoofingaudioforensics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether a speech deepfake detector improves when its backbone improves at automatic speech recognition. It answers by building detectors from every released size of Whisper and Wav2Vec 2.0, keeping the backbones frozen and training only a small classifier, and measuring detection across ASVspoof 2019 and four datasets the detectors have not seen. Detection does improve as models grow from tiny to small or medium, but then it plateaus: the largest Whisper and Wav2Vec 2.0 variants do not consistently beat their medium counterparts, and many tracks the largest Whisper model misses are caught by the tiny one. The paper concludes that ASR skill and deepfake detection skill are only partially aligned, so choosing a feature extractor by ASR quality or parameter count alone is not a reliable strategy.

What carries the argument

The mechanism is a controlled transfer experiment built on model-size families of frozen ASR feature extractors. Whisper provides five sizes, from tiny at 39M parameters to large at 1550M, and Wav2Vec 2.0 provides three, base at 95M, large at 317M, and xls-r at 317M; each variant feeds embeddings into its own small trainable classifier head while the ASR weights stay frozen, and the training pipeline is held fixed so that differences in detection accuracy are attributed to the pre-trained representation. Two instruments make the argument visible: a size-versus-EER sweep across five datasets, and a detection-overlap matrix that records, for each pair of Whisper sizes, what fraction of tracks missed by one model are caught by the other. That overlap matrix is what shows the relationship is partial rather than hierarchical.

What would settle it

A model-size sweep with per-size hyperparameter tuning that shows monotonic improvement through the largest model, or an overlap matrix in which every smaller model's correct detections are a subset of the larger model's, would contradict the saturation and partial-alignment conclusion.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is a saturation of transfer: the relationship between ASR performance and speech deepfake detection is real but bounded. In Table I, Whisper's average EER across the five test setups falls from 20.60% for tiny to 13.23% for small and then rises to 16.08% for medium and 14.46% for large, even though the large model has about six times the parameters of the small model and a better WER. Wav2Vec 2.0 large beats base on every dataset, but xls-r, which shares the large model's size and was trained multilingually, often performs worse. The overlap analysis reinforces the point: detection across sizes is not hierarchical, with 75.6% of tracks missed by Whisper large correctly identified by Whisper tiny. The authors describe this as a bottleneck effect: beyond a certain model size, more parameters and lower WER do not translate into more accurate deepfake detection.

Load-bearing premise

The plateau conclusion rests on the assumption that one fixed training pipeline is fair for every model size; if larger ASR models simply need different learning rates or regularization, the observed saturation would be an artifact of undertraining rather than a limit of ASR knowledge.

Editorial extensions

If this is right

  • Choosing a speech deepfake detector's ASR backbone by WER or parameter count alone is not reliable; the authors' sweeps show detection performance stabilizes after the small-to-medium sizes.
  • Larger ASR backbones leave recoverable errors: on average, 75.6% of the tracks Whisper large misses are caught by Whisper tiny, so a single large model is not a superset of smaller models' detections.
  • A model trained on multilingual data, Wav2Vec 2.0 xls-r, can underperform an English-trained model of equal size on English deepfake benchmarks, so pretraining distribution matters at least as much as model size.
  • Because the plateau appears across two different ASR model families and across five evaluation datasets, it behaves like a general design constraint for this detector family rather than a quirk of one model.
  • The strongest practical reading is that the sweet spot for cost and recall sits around the small and medium backbones, with the largest variants adding compute without a proportional detection gain.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper does not run is an ensemble across model sizes: since the overlap matrix shows error sets are complementary, combining a tiny model with a large one could raise recall beyond any single backbone.
  • The plateau might partly be an artifact of the fixed training pipeline; per-size learning rates and regularization could push the largest models higher, which would reframe the result as a training-equivalence finding rather than a ceiling on ASR knowledge.
  • The xls-r result suggests that matching the pretraining language distribution to the target speech may matter more than model scale, a hypothesis that could be tested directly by evaluating the same model sizes on non-English deepfake data.
  • For practical deployment, the paper implies that model selection should be driven by detection benchmarks on the target domain, not by recognition leaderboards.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper investigates whether larger ASR-pretrained models (Whisper and Wav2Vec 2.0) yield better speech deepfake detectors. The authors freeze each pretrained backbone, train small task-specific classifier heads on the ASVspoof 2019 LA training partition, and evaluate on ASVspoof 2021 DF, In-the-Wild, TIMIT-TTS/LJspeech, and FakeOrReal. They report EER/AUC values, show an overlap matrix of detection errors across Whisper sizes, and conclude that detection performance improves with model size up to a medium scale and then plateaus or degrades, so ASR performance is only partially aligned with deepfake detection.

Significance. This is a timely and practically relevant empirical question. The study covers two major self-supervised ASR families and multiple out-of-distribution datasets, and the overlap matrix in Figure 5 is a useful diagnostic for showing that detection failures are not nested by model size. If the conclusions survive a more rigorous comparison, they would caution against selecting ASR backbones solely by parameter count or WER. The main weaknesses are the absence of uncertainty quantification and the lack of evidence that a single training pipeline is fair for all model sizes; both directly affect the central claim.

major comments (3)
  1. [Section III-B, Table I, Figure 4] The claim that the same training pipeline 'ensures that the differences in performance are due solely to the initial pre-trained weights' is not justified. With Whisper parameter counts ranging from 39M to 1550M and embedding widths from 384 to 1280, and with Wav2Vec 2.0 models differing in depth (12 vs 24 layers), a single learning rate (1e-4), batch size (64), epoch budget (100), early-stopping patience (10), and one classifier architecture per family are unlikely to be near-optimal for all sizes. The plateau in Table I and Figure 4 could therefore reflect over- or under-trained large models rather than a genuine saturation of ASR knowledge for deepfake detection. Please provide per-size hyperparameter sweeps or convergence evidence, or at least discuss this confound explicitly.
  2. [Table I] The central 'plateau' claim rests on small differences that are reported without any uncertainty estimate. For instance, on ASVspoof 2019 the Whisper medium EER is 1.58% and large is 2.00%; on the average row, medium is 16.08% and large is 14.46%. These differences are plausibly within run-to-run variability of training a small classifier head. Without multiple seeds, confidence intervals, or a paired significance test, the conclusion that larger models do not consistently outperform medium models is not statistically supported.
  3. [Figure 5 and Section IV] The overlap matrix is a valuable diagnostic, but the text does not define the normalization (row-wise vs column-wise) or the population (e.g., tracks missed by the row model, or by the column model). More importantly, the observed non-hierarchy cannot distinguish a genuine lack of nesting from an undertrained large model; this connects to the concern in Section III-B. If the training-pipeline confound is resolved, the overlap result would strengthen the paper, but as it stands it does not settle the central claim.
minor comments (6)
  1. [Table I caption] The statement 'WER VALUES ARE TAKEN FROM [17]' is inaccurate for Wav2Vec 2.0; reference [17] is the Whisper paper. Please cite the original WER source for each model family and specify the test set (presumably LibriSpeech) and decoding details.
  2. [Section III-B] For Wav2Vec 2.0, the weighted-sum pooling over hidden layers introduces a different number of trainable weights per model (12 vs 24 layers), so classifier capacity is not matched across model sizes; please state how this is handled in the comparison.
  3. [Section V] The phrase 'Future words could focus' should read 'Future work could focus'.
  4. [Figure 5] Please add a colorbar and a caption that explains whether the entries are percentages of the row model's missed tracks or of the column model's missed tracks, and clarify the denominator used for normalization.
  5. [Abstract and Introduction] The text refers to 'ASR methods,' but Whisper is an encoder-decoder trained with multiple speech objectives; using 'ASR-pretrained models' would be more precise.
  6. [General] The paper does not mention whether code or trained detectors will be released; adding a reproducibility statement or link would be helpful.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the plateau finding is an empirical measurement, not a quantity derived from its own inputs.

full rationale

The paper's central claim is an empirical observation: larger ASR models generally improve speech deepfake detection performance up to a medium scale, after which performance plateaus. The comparison is built from two independent quantities: ASR quality, represented by word error rates quoted from the external Whisper paper [17], and detection quality, represented by EER/AUC values that the authors measure on held-out evaluation partitions and additional unseen datasets. No fitted parameter is renamed as a prediction, and no equation defines the detection outcome in terms of the ASR input in a way that would force the plateau. The fixed training pipeline described in Section III-B is an experimental control intended to attribute performance differences to the initial pre-trained weights; whether a single learning rate and epoch budget are equally fair to all model sizes is a validity or confound concern, not a circularity concern. Self-citations appear for datasets and prior detectors, but they are not load-bearing for the plateau conclusion: TIMIT-TTS is cited as a dataset, and the bias/robustness papers are related work. The overlap analysis in Figure 5 is a direct computation over which tracks each model misses, and its conclusion that detection capabilities are not strictly hierarchical is an observed empirical pattern rather than an artifact of how the conclusion was defined. There is no self-citation chain invoked to forbid alternatives, no uniqueness theorem imported from the authors, and no ansatz smuggled in via citation. The derivation chain is therefore self-contained with respect to the paper's central empirical claim, so the circularity burden is minimal.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

No new scientific entities are introduced. The free parameters are the trained classifier weights and manually fixed training hyperparameters; the main assumptions concern the comparability of training across model sizes and the use of external WER as the ASR axis.

free parameters (3)
  • Whisper detector classifier weights = not released
    Two fully connected layers trained on ASVspoof 2019 LA for each Whisper size; the reported EER and AUC results depend on these fitted weights.
  • Wav2Vec 2.0 detector weights (classifier and hidden-layer weighted sum) = not released
    Three FC layers plus learned weights of the weighted average over transformer layers; the reported detection metrics are produced by this trained module.
  • Training hyperparameters (learning rate, batch size, epochs, early stopping, audio duration) = lr=1e-4, batch=64, epochs=100, early stop=10, fs=16 kHz, duration=5 s
    Chosen by hand and fixed across model sizes; they directly affect the plateau observation and are central to the paper's comparative claim.
assumptions (3)
  • domain assumption WER on LibriSpeech reported in the cited model papers is an adequate measure of ASR performance for the comparison.
    Table I uses WER values with no clear source for Wav2Vec 2.0; if LibriSpeech WER does not represent ASR quality on the deepfake audio, the correlation analysis is weakened.
  • domain assumption The same training pipeline is equally appropriate for all model sizes, so performance differences are attributable to the pretrained weights.
    Section III-B; larger models often require different optimization schedules, so this assumption is the main confound for the plateau conclusion.
  • domain assumption ASVspoof 2019 LA training labels and eval partition are a valid basis for training and open-set evaluation of deepfake detectors.
    Section III-A; all additional datasets are used only for testing, but the detector performance starts from this benchmark.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Comparative Analysis of ASR Methods for Speech Deepfake Detection." pith.science (2026). https://pith.science/paper/7PR37BDJ

@misc{pith2026241117349,
  author       = {Pith},
  title        = {Pith review of: Comparative Analysis of ASR Methods for Speech Deepfake Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7PR37BDJ}},
  note         = {Machine review of arXiv:2411.17349}
}
read the original abstract

Recent techniques for speech deepfake detection often rely on pre-trained self-supervised models. These systems, initially developed for Automatic Speech Recognition (ASR), have proved their ability to offer a meaningful representation of speech signals, which can benefit various tasks, including deepfake detection. In this context, pre-trained models serve as feature extractors and are used to extract embeddings from input speech, which are then fed to a binary speech deepfake detector. The remarkable accuracy achieved through this approach underscores a potential relationship between ASR and speech deepfake detection. However, this connection is not yet entirely clear, and we do not know whether improved performance in ASR corresponds to higher speech deepfake detection capabilities. In this paper, we address this question through a systematic analysis. We consider two different pre-trained self-supervised ASR models, Whisper and Wav2Vec 2.0, and adapt them for the speech deepfake detection task. These models have been released in multiple versions, with increasing number of parameters and enhanced ASR performance. We investigate whether performance improvements in ASR correlate with improvements in speech deepfake detection. Our results provide insights into the relationship between these two tasks and offer valuable guidance for the development of more effective speech deepfake detectors.

Figures

Figures reproduced from arXiv: 2411.17349 by the authors.

Figure 1
Figure 1. Whisper-based model for Speech Deepfake Detection. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Wav2Vec 2.0-based model for Speech Deepfake Detection. [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. ROC curves of the Whisper-based models for different model sizes [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: EER values across the considered datasets for Whisper-based models, [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Detection overlap matrix showing the percentage of tracks missed by [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

25 extracted references · 23 canonical work pages

  1. [1]

    AI-synthesized faces are indistinguish- able from real faces and more trustworthy,

    S. J. Nightingale and H. Farid, “AI-synthesized faces are indistinguish- able from real faces and more trustworthy,” Proceedings of the National Academy of Sciences , vol. 119, no. 8, p. e2120481119, 2022

  2. [2]

    Open challenges in synthetic speech detection,

    L. Cuccovillo, C. Papastergiopoulos, A. Vafeiadis, A. Yaroshchuk, P. Aichroth, K. V otis, and D. Tzovaras, “Open challenges in synthetic speech detection,” in IEEE International Workshop on Information Forensics and Security (WIFS) , 2022

  3. [3]

    An Overview of Recent Work in Mul- timedia Forensics,

    K. Bhagtani, A. K. S. Yadav, E. R. Bartusiak, Z. Xiang, R. Shao, S. Baireddy, and E. J. Delp, “An Overview of Recent Work in Mul- timedia Forensics,” in IEEE Conference on Multimedia Information Processing and Retrieval , 2022

  4. [4]

    AI-Synthesized V oice Detection Using Neural V ocoder Artifacts,

    C. Sun, S. Jia, S. Hou, and S. Lyu, “AI-Synthesized V oice Detection Using Neural V ocoder Artifacts,” inIEEE/CVF Conference on Computer Vision and Pattern Recognition Workshop (CVPRW) , 2023

  5. [5]

    AASIST: Audio Anti-Spoofing Using Integrated Spectro-Temporal Graph Attention Networks,

    J.-w. Jung, H.-S. Heo, H. Tak, H.-j. Shim, J. S. Chung, B.-J. Lee, H.-J. Yu, and N. Evans, “AASIST: Audio Anti-Spoofing Using Integrated Spectro-Temporal Graph Attention Networks,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022

  6. [6]

    Combin- ing Automatic Speaker Verification and Prosody Analysis for Synthetic Speech Detection,

    L. Attorresi, D. Salvi, C. Borrelli, P. Bestagini, and S. Tubaro, “Combin- ing Automatic Speaker Verification and Prosody Analysis for Synthetic Speech Detection,” in International Conference on Pattern Recognition (ICPR), 2022

  7. [7]

    A robust audio deepfake detection system via multi-view feature,

    Y . Yang, H. Qin, H. Zhou, C. Wang, T. Guo, K. Han, and Y . Wang, “A robust audio deepfake detection system via multi-view feature,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024

  8. [8]

    Fairssd: Understanding bias in synthetic speech detectors,

    A. K. S. Yadav, K. Bhagtani, D. Salvi, P. Bestagini, and E. J. Delp, “Fairssd: Understanding bias in synthetic speech detectors,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshop (CVPRW), 2024

Show all 25 references
  1. [9]

    Reliability estimation for syn- thetic speech detection,

    D. Salvi, P. Bestagini, and S. Tubaro, “Reliability estimation for syn- thetic speech detection,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2023

  2. [10]

    Is Synthetic V oice Detection Research Going Into the Right Direction?

    S. Borz `ı, O. Giudice, F. Stanco, and D. Allegra, “Is Synthetic V oice Detection Research Going Into the Right Direction?” in IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshop (CVPRW), 2022

  3. [11]

    Listening Between the Lines: Synthetic Speech Detection Disregarding Verbal Content,

    D. Salvi, T. S. Balcha, P. Bestagini, and S. Tubaro, “Listening Between the Lines: Synthetic Speech Detection Disregarding Verbal Content,” in IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW). IEEE, 2024

  4. [12]

    Improving Domain Gen- eralization in Speech Emotion Recognition with Whisper,

    E. Goron, L. Asai, E. Rut, and M. Dinov, “Improving Domain Gen- eralization in Speech Emotion Recognition with Whisper,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024

  5. [13]

    Addressing Data Scarcity in V oice Disorder Detection with Self- Supervised Models,

    R. Gupta, C. Madill, D. R. Gunjawate, D. D. Nguyen, and C. T. Jin, “Addressing Data Scarcity in V oice Disorder Detection with Self- Supervised Models,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2024

  6. [14]

    Automatic Speaker Verification Spoofing and Deepfake Detection Us- ing Wav2vec 2.0 and Data Augmentation,

    H. Tak, M. Todisco, X. Wang, J. weon Jung, J. Yamagishi, and N. Evans, “Automatic Speaker Verification Spoofing and Deepfake Detection Us- ing Wav2vec 2.0 and Data Augmentation,” in Speaker and Language Recognition Workshop (Odyssey), 2022

  7. [15]

    Improved DeepFake Detection Using Whisper Features,

    P. Kawa, M. Plata, M. Czuba, P. Szymanski, and P. Syga, “Improved DeepFake Detection Using Whisper Features,” in Conference of the In- ternational Speech Communication Association (INTERSPEECH), 2023

  8. [16]

    Audio Deepfake Detection With Self-Supervised Wavlm And Multi-Fusion Attentive Classifier,

    Y . Guo, H. Huang, X. Chen, H. Zhao, and Y . Wang, “Audio Deepfake Detection With Self-Supervised Wavlm And Multi-Fusion Attentive Classifier,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2024

  9. [17]

    Robust speech recognition via large-scale weak super- vision,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak super- vision,” in International Conference on Machine Learning (ICML) . PMLR, 2023

  10. [18]

    wav2vec 2.0: A framework for self-supervised learning of speech representations,

    A. Baevski, Y . Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” Advances in Neural Information Processing Systems (NeurIPS) , 2020

  11. [19]

    Improved DeepFake Detection Using Whisper Features,

    P. Kawa, M. Plata, M. Czuba, P. Szymanski, and P. Syga, “Improved DeepFake Detection Using Whisper Features,” in Conference of the In- ternational Speech Communication Association (INTERSPEECH), 2021

  12. [20]

    ASVspoof 2019: Future horizons in spoofed and fake audio detection,

    M. Todisco, X. Wang, V . Vestman, M. Sahidullah, H. Delgado, A. Nautsch, J. Yamagishi, N. Evans, T. Kinnunen, and K. A. Lee, “ASVspoof 2019: Future horizons in spoofed and fake audio detection,” in Conference of the International Speech Communication Association (INTERSPEECH), 2019

  13. [21]

    ASVspoof 2021: accelerating progress in spoofed and deepfake speech detection,

    J. Yamagishi, X. Wang, M. Todisco, M. Sahidullah, J. Patino, A. Nautsch, X. Liu, K. A. Lee, T. Kinnunen, N. Evans et al., “ASVspoof 2021: accelerating progress in spoofed and deepfake speech detection,” in Automatic Speaker Verification and Spoofing Countermeasures Chal- lenge, 2021

  14. [22]

    Does audio deepfake detection generalize?

    N. M. M ¨uller, P. Czempin, F. Dieckmann, A. Froghyar, and K. B¨ottinger, “Does audio deepfake detection generalize?” in Conference of the Inter- national Speech Communication Association (INTERSPEECH) , 2022

  15. [23]

    TIMIT-TTS: a Text-to-Speech Dataset for Multimodal Synthetic Media Detection,

    D. Salvi, B. Hosler, P. Bestagini, M. C. Stamm, and S. Tubaro, “TIMIT-TTS: a Text-to-Speech Dataset for Multimodal Synthetic Media Detection,” IEEE Access, 2023

  16. [24]

    The LJSpeech dataset,

    Keith Ito and Linda Johnson, “The LJSpeech dataset,” 2017. [Online]. Available: https://keithito.com/LJ-Speech-Dataset/

  17. [25]

    FoR: A dataset for synthetic speech detec- tion,

    R. Reimao and V . Tzerpos, “FoR: A dataset for synthetic speech detec- tion,” in International Conference on Speech Technology and Human- Computer Dialogue (SpeD) . IEEE, 2019

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.