Pith. sign in

REVIEW 4 major objections 5 minor 17 references

Deepfake Detection of Singing Voices With Whisper Encodings

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Whisper encoder representations, especially the medium variant with a ResNet34 classifier, detect deepfake singing voices in both isolated vocals and full mixtures, outperforming standard audio features and the Wav2vec2+AASIST baseline on…

desk verdict Useful first systematic look at Whisper encodings for singing-voice deepfake detection, but the SOTA-over-baseline claim is not yet auditable because the baseline was modified off-book and the best model is picked on test partitions. read the letter →

arxiv 2501.18919 v1 pith:GDLNCEHS submitted 2025-01-31 cs.SD cs.AIeess.AS

Falguni Sharma , Priyanka Gupta This is my paper · ORCID
classification cs.SDcs.AIeess.AS
keywords singingvoicedeepfakedetectionWhisperencodingsanti-spoofingtransferlearningSingfakedatasetsynthesisEqualErrorRateaudioforensics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the internal audio encodings of the Whisper speech-recognition model, despite coming from a system trained to be robust to noise, are in fact sensitive to background and non-speech content, and that this sensitivity can be turned into a detector for deepfake singing voices. Using the Singfake dataset, the authors feed Whisper encoder outputs into CNN and ResNet34 classifiers and report that the medium Whisper variant with ResNet34 achieves a 4.86% average equal error rate on vocals and 9.45% on mixtures, outperforming MFCC, CQCC, LFCC, and the Wav2vec2+AASIST baseline. The results matter because unauthorized synthetic vocals threaten artists' likeness and royalties, and singing voices behave differently from speech, so speech anti-spoofing systems may not carry over. If the claim is right, a readily available ASR encoder plus a small classifier is a practical starting point for singing-voice deepfake detection.

What carries the argument

The central object is the last hidden state of the Whisper encoder, a sequence of fixed-dimension vectors produced from input mel-spectrograms by a Transformer encoder with a number of blocks that varies by model size. Whisper is trained on hundreds of thousands of hours of diverse web audio, and prior work has shown that, unlike typical ASR front-ends, it encodes background sound rather than discarding it. That noise-variant representation is the mechanism: it retains acoustic context such as the abrupt transition from silence into a synthesized vocal and the separation of formants from instrumentals, giving a CNN or ResNet34 classifier enough discriminative signal to separate bonafide from deepfake singing voices without data augmentation. In the paper's experiments the medium Whisper variant performs best with both classifiers.

What would settle it

Run the Wav2vec2+AASIST baseline with RawBoost removed on the same train/validation splits and test sets used for the proposed system; if the resulting T04 equal error rates are substantially lower than the numbers the paper compares against, the reported 28.94- and 24.52-percentage-point improvements would not hold.

Watch

Extended reading notes

Core claim

The paper's central claim is that Whisper's audio representation is not a noise-invariant abstraction but a noise-conditioned one, and that very property is useful for anti-spoofing. Concretely, the authors show that feeding the encoder output of the medium Whisper variant through a ResNet34 classifier yields a 4.86% average EER on the Singfake vocals and 9.45% on mixtures, outperforming standard feature sets and the Wav2vec2+AASIST baseline. On the hardest test condition, T04 with unseen languages and musical contexts, the reported absolute EER improvements over the baseline are 28.94 percentage points for vocals and 24.52 percentage points for mixtures. Spectrographic differences illustrated in the paper indicate that deepfake vocals show sharp onsets after silence and formants that remain distinct from background instruments, while bonafide recordings fade gradually and blend with the accompaniment; the Whisper encodings are said to preserve these synthesis artifacts.

Load-bearing premise

The load-bearing premise is that the previous best system's scores after removing its data-augmentation module were computed on exactly the same train/test splits and settings as the proposed system; the paper does not document this.

Editorial extensions

If this is right

  • A single Whisper encoder variant with a ResNet34 head is enough to reach 4.86% average EER on Singfake vocals and 9.45% on mixtures, so a singing-voice deepfake detector need not start from speech-specific countermeasures.
  • The detector keeps working when the singing voice is mixed with instrumental accompaniment, though mixtures are consistently harder to detect than isolated vocals.
  • The method transfers across the Singfake testing conditions with a clear difficulty gradient: seen-singer unseen-song clips are easiest at 1.09% EER, while unseen languages and musical contexts are hardest, localizing the remaining robustness gap.
  • Whisper encodings outperform MFCC, CQCC, and LFCC for both vocals and mixtures with both classifiers, indicating the gain comes from the pre-trained representation rather than from the classifier choice.
  • Because the system uses no data augmentation, the discriminative power is carried by the encodings themselves rather than by input perturbation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same noise-variant encoding principle could be tested on speech deepfakes and non-vocal audio deepfakes: if Whisper's representations preserve the acoustic context that synthesis models reconstruct imperfectly, a similar encoder-plus-classifier pipeline may transfer without retraining the encoder.
  • A practical deployment would freeze the Whisper encoder and train only the small classification head, making the detector cheap to update as new synthesis methods appear; this is not tested in the paper but follows from its architecture.
  • The sharp T04 drop suggests the model's cues are partly tied to language and musical context seen in training; covering more languages in the training partition or adding augmentation that simulates unseen contexts would test whether the artifacts the encodings capture are truly universal.
  • Combining Whisper encodings with the best standard feature set, such as LFCC, or with augmentation might close the remaining T04 gap, an avenue the paper itself names as future work.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a singing voice deepfake detection (SVDD) system that extracts encoder representations from OpenAI's Whisper model at four sizes (tiny, base, small, medium) and feeds them to a CNN or ResNet34 classifier. Using the Singfake dataset, it evaluates the system on vocals and mixtures across test conditions T01-T04 and reports average EERs of 4.86% and 9.45% for Whisper-medium with ResNet34 on vocals and mixtures, respectively. It also reports large EER improvements over a Wav2vec2+AASIST baseline, especially on T04, and concludes that Whisper encodings outperform standard MFCC/CQCC/LFCC features.

Significance. If the reported results are reproducible under a common protocol, the paper would provide a useful empirical data point: an off-the-shelf frozen ASR encoder combined with a small classifier can be a competitive detector for singing voice deepfakes. The conceptual hook that Whisper's encodings are not noise-invariant and may carry discriminative cues is interesting, and the within-paper ordering across Whisper sizes and standard features is consistent with the main claim. However, the comparison to prior state of the art is not currently auditable because the baseline modification is undocumented, and the best configuration is selected using test-partition results. These issues must be addressed before the headline claim can be accepted.

major comments (4)
  1. [III-C and IV-C] The comparison against the Wav2vec2+AASIST baseline is load-bearing and not substantiated. Section III-C says only that in the baseline work [7] 'the RawBoost data augmentation module has been removed for fair comparisons between methods,' but it does not state whether the authors reran Wav2vec2+AASIST in their own codebase, with the same train/validation/test partitions and the same hyperparameters, or whether published numbers from [7] were used with an adjustment. Section IV-C then reports only absolute EER differences (28.94 and 24.52 percentage points on T04) and no absolute baseline EERs, so the reader cannot audit the comparison. If the published [7] numbers include RawBoost or were obtained under a different input segmentation or classifier training protocol, the claimed superiority over prior state of the art is not established. Please rerun the baseline under the common protocol and report its absolute EERs alongside the proposed system.
  2. [IV-A, Figures 3-6] The model configuration appears to be selected on the test partitions. The paper identifies W(Med.) as the best based on average 'testing' EERs in Figures 3-6, and Section IV-B then restricts all subsequent experiments to ResNet34 after observing that it gave the best performance. The baseline from [7] is not given the same selection advantage, so the reported T04 differences mix algorithmic gain with test-set selection. Please specify a validation-based model selection rule and report test EERs for the selected configuration, or provide results for all configurations so that the selection protocol is transparent.
  3. [IV-A] All EER values appear to be from a single run with no error bars, confidence intervals, or significance tests, and the paper does not mention releasing code or experimental details sufficient for reproduction. Because some reported differences between Whisper sizes and between feature sets may be small, the reader cannot determine which pairwise gaps are statistically meaningful. Please provide multiple seeds or bootstrap confidence intervals, and release code or a detailed reproducibility description.
  4. [III-C and IV-A] The comparison with standard features mixes protocols. LFCC results are taken directly from [7], while MFCC and CQCC experiments are performed by the authors, but the text does not specify whether the same CNN/ResNet34 classifiers, input pipelines, and train/validation splits were used for these standard features as for the Whisper encodings. If the LFCC numbers come from a different classifier or protocol, the claim that Whisper encodings outperform standard representations is not an apples-to-apples comparison. Please report the standard-feature EERs under the same protocol or clearly state the differences.
minor comments (5)
  1. [IV-C] The value '24.52' should read '24.52%' for consistency with the adjacent '28.94%'.
  2. [IV-B.1] There is a typo: 'T o4' should be 'T04'.
  3. [Title] The title contains an erroneous space in 'V oices'; this is likely a LaTeX formatting artifact that should be corrected in the final version.
  4. [II-A] The claim that Whisper's last hidden state has 'fixed dimensions' is imprecise, because the encoder output length depends on the input mel-spectrogram duration. The paper should state how variable-length clips are converted to the fixed-size input expected by ResNet34/CNN (for example, padding, truncation, or pooling).
  5. [II-B] The noise-variance property is attributed to [9] for general audio, but the paper does not directly measure whether the encodings for singing vocals are more noise-sensitive than standard features. This is a framing issue rather than a flaw in the performance comparison, but it should be phrased as a hypothesis supported by the prior work.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the central result is an empirical benchmark on the external Singfake dataset, and the minor self-citation is not load-bearing.

full rationale

The paper is an empirical evaluation rather than a derivation, and no load-bearing step equates a fitted parameter with a predicted quantity or imports a uniqueness result from the authors' own prior work. Whisper encodings come from a frozen, externally trained model, the classifiers are trained on the Singfake training partition, and the headline EERs are computed on held-out test partitions (T01-T04). The noise-variance premise is attributed to the external Whisper-AT study [9], not to a self-citation. The only self-citation is [2], used in the introduction to motivate concern about deepfakes; it is context, not a load-bearing argument. The baseline Wav2vec2+AASIST and LFCC results are taken from the external Singfake paper [7]. The statement "in this baseline work [7], the RawBoost data augmentation module has been removed for fair comparisons between methods" raises a legitimate comparability concern, and the selection of W(Med.) based on test-partition results is an evaluation-protocol issue, but neither makes the result circular: the proposed system's outputs do not reduce by construction to its inputs. Therefore no specific circular step can be exhibited under the hard-evidence rule, and the appropriate finding is no significant circularity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new entities or forces. The main extra assumptions are dataset reliability, transferability of Whisper's noise-variant encoding property to singing voices, and the fairness of the modified baseline. Hyperparameters such as Whisper size and classifier are selected from observed EERs, which is the main free parameter.

free parameters (2)
  • Whisper encoder size = medium
    W(Med.) is declared the best system after comparing EER across sizes in Section IV-A; this choice is made on the evaluation data rather than a separate validation split.
  • Classifier architecture = ResNet34
    ResNet34 is chosen as the classifier for all subsequent experiments because it showed the best performance on both vocals and mixtures (Section IV-B); this is a data-driven architectural selection without nested validation.
assumptions (4)
  • domain assumption Singfake dataset labels and partitions (bonafide vs deepfake, vocals vs mixtures, T01-T04) are accurate and representative.
    All conclusions are measured against this dataset; Section III-A relies on its correctness.
  • domain assumption Whisper encoder outputs retain non-speech/noise information that is discriminative for deepfake artifacts.
    Adopted from Whisper-AT [9] and applied to singing voices without a dedicated verification; Section II-B.
  • ad hoc to paper RawBoost removal from the baseline in [7] yields a fair, directly comparable reference.
    The paper modifies the published baseline but does not report rerun details; Section III-C.
  • domain assumption EER averaged over four test splits is a stable estimate of system performance.
    No confidence intervals or repeated runs are provided; Section III-B.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Deepfake Detection of Singing Voices With Whisper Encodings." pith.science (2026). https://pith.science/paper/GDLNCEHS

@misc{pith2026250118919,
  author       = {Pith},
  title        = {Pith review of: Deepfake Detection of Singing Voices With Whisper Encodings},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GDLNCEHS}},
  note         = {Machine review of arXiv:2501.18919}
}
read the original abstract

The deepfake generation of singing vocals is a concerning issue for artists in the music industry. In this work, we propose a singing voice deepfake detection (SVDD) system, which uses noise-variant encodings of open-AI's Whisper model. As counter-intuitive as it may sound, even though the Whisper model is known to be noise-robust, the encodings are rich in non-speech information, and are noise-variant. This leads us to evaluate Whisper encodings as feature representations for the SVDD task. Therefore, in this work, the SVDD task is performed on vocals and mixtures, and the performance is evaluated in \%EER over varying Whisper model sizes and two classifiers- CNN and ResNet34, under different testing conditions.

Figures

Figures reproduced from arXiv: 2501.18919 by the authors.

Figure 1
Figure 1. The proposed end-to-end SVDD system using Whisper Encoder with ResNet34 [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Bonafide vs. Deepfake Singing Voices particular, as indicated by the regions marked in green circles in [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. %EER for various splits on Singfake vocals across different repre [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: %EER for various splits on Singfake vocals across different repre [PITH_FULL_IMAGE:figures/full_fig_p003_4.png]
Figure 5
Figure 5. Figure 5: %EER for various splits on Singfake mixtures across different [PITH_FULL_IMAGE:figures/full_fig_p003_5.png]
Figure 6
Figure 6. Figure 6: %EER for various splits on Singfake mixtures across different [PITH_FULL_IMAGE:figures/full_fig_p004_6.png]
Figure 9
Figure 9. Figure 9: Comparison of the proposed system with the baseline [PITH_FULL_IMAGE:figures/full_fig_p004_9.png]
Figure 7
Figure 7. Figure 7: Testing Conditions Comparison on Vocals 2) Mixtures: It can be observed from [PITH_FULL_IMAGE:figures/full_fig_p004_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 13 canonical work pages

  1. [7]

    Singfake: Singing voice deepfake detection,

    Y . Zang, Y . Zhang, M. Heydari, and Z. Duan, “Singfake: Singing voice deepfake detection,” 2024. [Online]. Available: https://arxiv.org/ abs/2309.07525

    work page arXiv 2024

  2. [1]

    Asvspoof 5: Crowdsourced speech data, deepfakes, and adversarial attacks at scale,

    X. Wang, H. Delgado, H. Tak, J.-w. Jung, H.-j. Shim, M. Todisco, I. Kukanov, X. Liu, M. Sahidullah, T. Kinnunen et al. , “Asvspoof 5: Crowdsourced speech data, deepfakes, and adversarial attacks at scale,” arXiv preprint arXiv:2408.08739 , 2024

    arXiv 2024

  3. [2]

    Vulnerability issues in automatic speaker verification (ASV) systems,

    P. Gupta, H. A. Patil, and R. C. Guido, “Vulnerability issues in automatic speaker verification (ASV) systems,” EURASIP Journal on Audio, Speech, and Music Processing , vol. 2024, no. 1, p. 10, 2024

    work page 2024

  4. [3]

    Visinger: Variational inference with adversarial learning for end-to-end singing voice synthesis,

    Y . Zhang, J. Cong, H. Xue, L. Xie, P. Zhu, and M. Bi, “Visinger: Variational inference with adversarial learning for end-to-end singing voice synthesis,” in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2022, pp. 7237–7241

    work page 2022

  5. [4]

    Diffsinger: Singing voice synthesis via shallow diffusion mechanism,

    J. Liu, C. Li, Y . Ren, F. Chen, and Z. Zhao, “Diffsinger: Singing voice synthesis via shallow diffusion mechanism,” in Proceedings of the AAAI conference on artificial intelligence , vol. 36, no. 10, 2022, pp. 11 020– 11 028

    work page 2022

  6. [5]

    Midi-voice: Expressive zero-shot singing voice synthesis via midi-driven priors,

    D.-M. Byun, S.-H. Lee, J.-S. Hwang, and S.-W. Lee, “Midi-voice: Expressive zero-shot singing voice synthesis via midi-driven priors,” in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2024, pp. 12 622–12 626

    work page 2024

  7. [6]

    Sintechsvs: A singing technique controllable singing voice synthesis system,

    J. Zhao, L. Q. H. Chetwin, and Y . Wang, “Sintechsvs: A singing technique controllable singing voice synthesis system,” IEEE/ACM Transactions on Audio, Speech, and Language Processing , 2024

    work page 2024

  8. [8]

    Robust speech recognition via large-scale weak supervi- sion,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervi- sion,” in International conference on machine learning . PMLR, 2023, pp. 28 492–28 518

    2023

Show all 17 references
  1. [9]

    Whisper-AT: Noise- robust automatic speech recognizers are also strong general audio event taggers,

    Y . Gong, S. Khurana, L. Karlinsky, and J. Glass, “Whisper-AT: Noise- robust automatic speech recognizers are also strong general audio event taggers,” in INTERSPEECH, 2023, pp. 2798–2802

    2023

  2. [10]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” 2023. [Online]. Available: https://arxiv.org/abs/1706.03762

    2023 arXiv

  3. [11]

    Invariant representations for noisy speech recognition,

    D. Serdyuk, K. Audhkhasi, P. Brakel, B. Ramabhadran, S. Thomas, and Y . Bengio, “Invariant representations for noisy speech recognition,” NIPS End-to-end Learning for Speech and Audio Processing Workshop , 2016

    2016

  4. [12]

    Learning noise-invariant representations for robust speech recognition,

    D. Liang, Z. Huang, and Z. C. Lipton, “Learning noise-invariant representations for robust speech recognition,” in 2018 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2018, pp. 56–63

    2018

  5. [13]

    A noise-robust self-supervised pre-training model based speech representation learning for automatic speech recognition,

    Q.-S. Zhu, J. Zhang, Z.-Q. Zhang, M.-H. Wu, X. Fang, and L.-R. Dai, “A noise-robust self-supervised pre-training model based speech representation learning for automatic speech recognition,” in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Proc...

    2022

  6. [14]

    Unsupervised learning of time–frequency patches as a noise-robust representation of speech,

    M. Van Segbroeck et al. , “Unsupervised learning of time–frequency patches as a noise-robust representation of speech,” Speech Communi- cation, vol. 51, no. 11, pp. 1124–1138, 2009

    2009

  7. [15]

    Demucs: Deep extractor for music sources with extra unlabeled data remixed,

    A. D ´efossez, N. Usunier, L. Bottou, and F. Bach, “Demucs: Deep extractor for music sources with extra unlabeled data remixed,” 2019. [Online]. Available: https://arxiv.org/abs/1909.01174

    2019 arXiv

  8. [16]

    Computationally-efficient voice activity detection based on deep neural networks,

    Y . Xiong, V . Berisha, and C. Chakrabarti, “Computationally-efficient voice activity detection based on deep neural networks,” in 2021 IEEE Workshop on Signal Processing Systems (SiPS) , 2021, pp. 64–69

    2021

  9. [17]

    Pyannote. audio: neu- ral building blocks for speaker diarization,

    H. Bredin, R. Yin, J. M. Coria, G. Gelly, P. Korshunov, M. Lavechin, D. Fustes, H. Titeux, W. Bouaziz, and M.-P. Gill, “Pyannote. audio: neu- ral building blocks for speaker diarization,” in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Process...

    2020

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.