Pith. sign in

REVIEW 3 major objections 5 minor 34 references

Current speech deepfake detectors collapse against modern LLM-era voice synthesizers and everyday post-processing.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

T0 review · grok-4.5

2026-07-14 03:43 UTC pith:6A236VRB

load-bearing objection Clean public bilingual benchmark of 2025 TTS/VC that shows most off-the-shelf spoof detectors collapse; the gap claim holds. the 3 major comments →

arxiv 2607.11706 v1 pith:6A236VRB submitted 2026-07-13 cs.SD cs.AI

VoxENES 2026: Benchmarking Generalization of Speech Spoofing Detectors Against LLM-Era TTS and Voice Conversion

classification cs.SD cs.AI
keywords audio deepfake detectionspeech spoofing detectionbenchmark datasetLLM-era TTSvoice conversionequal error ratepost-processing robustnesstemporal generalization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Legacy audio-spoofing benchmarks no longer represent the speech that modern language-model text-to-speech and voice-conversion systems produce. The authors therefore release VoxENES 2026, a bilingual English-Spanish collection of more than fifty thousand clips generated by ten contemporary synthesis pipelines and then subjected to ten common transmission and editing operations. When eight widely used pretrained detectors are evaluated on this material without any retraining, the strongest model still reaches only 28.98 percent equal-error rate overall, and most of the others hover near or worse than chance. The result shows that many detectors have been learning fragile, generator-specific cues rather than durable signs of synthetic speech, and it supplies a public testbed that future countermeasures must pass if they are to remain useful in the wild.

Core claim

On a new bilingual benchmark built from ten recent LLM-era TTS and VC systems plus ten standardized post-processing conditions, eight pretrained spoofing detectors evaluated without fine-tuning suffer severe degradation: the best reaches only 28.98 percent equal-error rate overall, while most perform near or below random chance. Current detectors therefore rely on brittle artifacts that do not survive modern generators or everyday audio transformations.

What carries the argument

VoxENES 2026 itself: a controlled collection of 53,628 English and Spanish clips that pairs ten contemporary synthesis methods with ten realistic post-processing operations, used strictly as an out-of-distribution test set for pretrained detectors.

Load-bearing premise

Uniform truncation and zero-padding of every clip to four seconds does not itself create cues that let a detector tell real speech from synthetic speech.

What would settle it

Re-run the same eight detectors on the identical generators and augmentations but with variable-length audio and no padding; if equal-error rates rise substantially above the reported figures, the padding step is contaminating the claim.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces VoxENES 2026, a bilingual (English/Spanish) benchmark of 53,628 samples built from 3,028 real utterances (LibriSpeech + VoxPopuli) and synthetic speech from 10 contemporary TTS/VC systems (7 TTS, 3 VC, mostly 2024–2025 LLM-era models), each expanded by 10 standardized post-processing operations (codecs, noise, resampling, speed, loudness). Eight pretrained detectors spanning graph NNs, raw-waveform CNNs, SSL front-ends, spectrogram transformers, and speaker-embedding anomaly scoring are evaluated zero-shot (no fine-tuning). The central empirical claim, supported by Tables 6–7 and Figure 3, is that performance collapses relative to legacy benchmarks: the best model (AST-ASVspoof5) reaches only 28.98% overall EER, five of eight models are near or worse than chance, Seed-VC is hardest for every detector, and post-processing effects are highly non-uniform (e.g., white noise lowers EER for some models while MP3 raises it). The authors interpret this as evidence that current detectors rely on brittle, generator-specific artifacts and position the dataset as a practical testbed for more robust countermeasures.

Significance. If the reported EERs hold under the stated protocol, the work supplies a timely, concrete demonstration of temporal generalization failure for speech spoofing detectors against LLM-era generators and realistic transmission distortions. The contribution is primarily empirical and infrastructural: a publicly released bilingual corpus with controlled post-processing, transparent zero-shot evaluation of eight published models, and method- and augmentation-resolved tables that expose detector-specific blind spots. Strengths include the use of recent open systems (VoxCPM, Qwen3-TTS, GLM-TTS, Seed-VC, etc.), the standardized augmentation suite, and the decision not to fine-tune, which cleanly isolates out-of-distribution robustness. These elements make the resource immediately usable for subsequent detector development and for tracking the cat-and-mouse dynamic between synthesis and detection.

major comments (3)
  1. Section 3.1 and Table 5: the eight detectors are trained on heterogeneous corpora (ASVspoof 2019 LA, 2021 DF, ASVspoof 5, mixed deepfake sets, VoxCeleb). While the paper correctly notes that absolute EERs are OOD indicators rather than a controlled head-to-head ranking, the headline claim that “most perform near or below random chance” and the ranking in Table 7 still invite direct comparison. A short controlled re-evaluation (or at least an explicit stratification by training source) would strengthen the generalization narrative and prevent readers from over-interpreting cross-family differences.
  2. Section 2.1 and all reported EERs: fixed-length truncation/zero-padding to 4 s is applied uniformly, and the authors leave a detailed padding-artifact analysis to future work. Although the same procedure is used for both classes (so padding is not a pure label shortcut), residual interactions with synthesis-specific spectro-temporal structure cannot be ruled out a priori. Because every number in Tables 6–7 and Figure 3 rests on this preprocessing, a minimal ablation (variable-length scoring or energy-based endpointing on a subset) is needed to confirm that the observed method-specific and augmentation-specific patterns survive.
  3. Tables 6–7 and Figure 3: no confidence intervals, bootstrap estimates, or statistical tests accompany the EER differences. With thousands of samples the point estimates are likely stable, yet claims such as “Seed-VC is the most challenging” and “noise reduces EER for AST-ASVspoof5 from 26.7 % to 17.4 %” would be more robust if uncertainty were quantified, especially given the non-uniform and occasionally counter-intuitive augmentation effects.
minor comments (5)
  1. Table 3: RVC v2 is listed as 2023 while the abstract and introduction emphasize “LLM-era” / 2025 systems; a brief clarification of why it is retained would help.
  2. Figures 1–2: spectrograms are illustrative but lack quantitative descriptors (e.g., high-frequency energy ratios) that would link the visual differences to the detector failures discussed in §4.2.
  3. Section 4.3: the inverted-prediction behavior of AASIST2 (EER 57.86 %) is noted; a short score-distribution plot or FAR/FRR curve would make the domain-mismatch diagnosis more concrete.
  4. Minor typographic inconsistencies appear (V oxENES vs. VoxENES, occasional missing spaces after periods).
  5. The Kaggle link is given; adding a short datasheet (license, speaker-disjointness verification, exact generation seeds) would improve long-term reproducibility.

Circularity Check

0 steps flagged

No significant circularity: pure empirical zero-shot benchmarking of external detectors on a newly constructed dataset.

full rationale

VoxENES 2026 is an empirical benchmark paper. Its central claim is the measured out-of-distribution EERs of eight pretrained detectors (none fine-tuned on the new data) under modern TTS/VC generators and post-processing. There are no equations that derive a quantity from a fitted parameter and then re-present that quantity as a prediction; no uniqueness theorem is imported; and the authors’ self-citations ([1–4], [28]) appear only as motivational context for the security problem, not as load-bearing premises for the reported EERs or the brittle-artifact interpretation. The evaluation protocol (fixed 4 s truncation/zero-padding applied uniformly to both classes, inference-only use of published weights, EER from ROC) is self-contained against external models and newly generated audio. Consequently the derivation chain does not reduce any claimed result to its own inputs by construction.

Axiom & Free-Parameter Ledger

2 free parameters · 3 axioms · 1 invented entities

As an empirical benchmark paper the load-bearing content is almost entirely observational. The few free choices are design parameters of the corpus (clip length, SNR values, codec bit-rates) rather than fitted constants. Domain assumptions are the usual ones of the anti-spoofing literature (EER as primary metric, pretrained models as proxies for deployed detectors). The sole invented entity is the dataset itself.

free parameters (2)
  • clip_duration_cap = 4 s
    All audio truncated or zero-padded to exactly 4 s for detector compatibility; value chosen by hand, not derived.
  • noise_SNR_levels = 10/20/15 dB
    White-noise SNRs of 10 dB / 20 dB and babble 15 dB are design choices that affect measured EERs.
axioms (3)
  • domain assumption Equal Error Rate (FAR = FRR operating point) is a sufficient scalar summary of detector quality under class imbalance.
    Used throughout Tables 6–7 and the abstract without further justification; standard in ASVspoof literature.
  • domain assumption Evaluating published pretrained weights without fine-tuning on VoxENES measures temporal/out-of-distribution generalization rather than merely domain mismatch.
    Stated in Section 3.1; underpins the claim that detectors ‘rely on brittle artifacts’.
  • ad hoc to paper Uniform zero-padding and truncation introduce no class-discriminative cues because the same procedure is applied to bonafide and synthetic audio.
    Explicitly asserted in Section 2.1; authors defer verification to future work.
invented entities (1)
  • VoxENES 2026 dataset independent evidence
    purpose: Provide a modern bilingual testbed of LLM-era TTS/VC plus post-processing so that detector generalization can be measured.
    The corpus is newly constructed by the authors; its existence is the paper’s primary contribution.

pith-pipeline@v1.1.0-grok45 · 14458 in / 2613 out tokens · 32430 ms · 2026-07-14T03:43:17.069888+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of VoxENES 2026: Benchmarking Generalization of Speech Spoofing Detectors Against LLM-Era TTS and Voice Conversion." pith.science (2026). https://pith.science/paper/6A236VRB

@misc{pith2026260711706,
  author       = {Pith},
  title        = {Pith review of: VoxENES 2026: Benchmarking Generalization of Speech Spoofing Detectors Against LLM-Era TTS and Voice Conversion},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6A236VRB}},
  note         = {Machine review of arXiv:2607.11706}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Modern LLM-driven text-to-speech (TTS) and voice conversion (VC) systems produce synthetic speech that differs from the generators represented in many legacy spoofing benchmarks. This mismatch creates a temporal generalization gap that can overestimate detector robustness under real-world post-processing conditions. We bridge this gap by introducing VoxENES 2026, a bilingual (English and Spanish) benchmark of 53,628 audio samples generated using 10 contemporary speech synthesis methods and evaluated under 10 standardized post-processing conditions. Using VoxENES 2026, we benchmark eight pretrained detectors without fine-tuning and observe substantial performance degradation: the best model achieves 28.98\% EER overall, while most perform near or below random chance across modern generators and perturbations. Our results highlight the reliance on brittle artifacts in current detectors and establish VoxENES 2026 as a practical testbed for developing robust audio spoofing countermeasures.

Figures

Figures reproduced from arXiv: 2607.11706 by Aastha Sharma, Guangjing Wang.

Figure 1
Figure 1. Figure 1: Representative spectrograms across multiple TTS and VC systems and augmentation conditions in VoxENES 2026 [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Qualitative spectrogram comparison [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Per-method EER on Original Samples 5. Conclusion We presented VoxENES 2026, a modern bilingual benchmark for speech spoofing detection that reflects LLM-era TTS and VC generation and realistic post-processing. Using VoxENES 2026, we evaluated eight pretrained detectors without fine￾tuning and observed a substantial generalization gap: the best model achieves only 28.98% EER overall, while many detectors pe… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

34 extracted references · 5 linked inside Pith

  1. [1]

    This need is growing as voice becomes a biometric and a con- trol channel for speech-driven agents and assistive technolo- gies

    Introduction Robust speech spoofing and deepfake detection are essential to preserve trust in speech-based authentication [1, 2, 3, 4]. This need is growing as voice becomes a biometric and a con- trol channel for speech-driven agents and assistive technolo- gies. If synthetic speech becomes indistinguishable from gen- uine speech, detection failures not ...

  2. [2]

    As shown in Table 1, V oxENES includes 3,028 real speech sam- ples and 50,600 synthetic samples (original synthetic 4,600 and augmented synthetic 46,000)

    VoxENES 2026 V oxENES 2026 consists of three components: (1) real speech from established corpora, (2) synthetic audio generated by mod- ern TTS and VC systems, and (3) post-processed augmented variants simulating real-world transmission conditions. As shown in Table 1, V oxENES includes 3,028 real speech sam- ples and 50,600 synthetic samples (original s...

  3. [3]

    Evaluation Setup 3.1. Detection Baselines We evaluated eight pretrained audio deepfake detection sys- tems spanning graph neural networks, raw-waveform CNNs, self-supervised learning (SSL) models, transformer-based de- tectors, and speaker-embedding anomaly detection as shown in Table 5. None were retrained or fine-tuned on V oxENES 2026, enabling an hone...

  4. [4]

    Evaluation Results 4.1. Qualitative Spectrogram Analysis Figures 1 and 2 visualize the spectral variability across bonafide (real) speech, synthetic speech, and post-processed synthetic samples in V oxENES 2026. The examples highlight that re- Table 3:TTS and VC systems used in VoxENES 2026. System Type Architecture Developer EN ES Total Year ES Mode V ox...

  5. [5]

    Conclusion We presented V oxENES 2026, a modern bilingual benchmark for speech spoofing detection that reflects LLM-era TTS and VC generation and realistic post-processing. Using V oxENES 2026, we evaluated eight pretrained detectors without fine- tuning and observed a substantial generalization gap: the best model achieves only 28.98% EER overall, while ...

  6. [6]

    All authors reviewed, verified, and take full responsibility for the content, experiments, and conclusions presented in this paper

    Use of Generative AI Disclosure Generative AI tools were used for language editing and manuscript polishing. All authors reviewed, verified, and take full responsibility for the content, experiments, and conclusions presented in this paper

  7. [7]

    Clearmask: Noise-free and naturalness-preserving protection against voice deepfake attacks,

    Y . Wang, B. Chen, H. Guo, G. Wang, W. Ding, and Q. Yan, “Clearmask: Noise-free and naturalness-preserving protection against voice deepfake attacks,” inProceedings of the 20th ACM Asia Conference on Computer and Communications Security, 2025, pp. 696–709

  8. [8]

    Wavepurifier: Purifying audio adversarial exam- ples via hierarchical diffusion models,

    H. Guo, G. Wang, B. Chen, Y . Wang, X. Zhang, X. Chen, Q. Yan, and L. Xiao, “Wavepurifier: Purifying audio adversarial exam- ples via hierarchical diffusion models,” inProceedings of the 30th Annual International Conference on Mobile Computing and Net- working, 2024, pp. 1268–1282

  9. [9]

    Phan- tomsound: Black-box, query-efficient audio adversarial attack via split-second phoneme injection,

    H. Guo, G. Wang, Y . Wang, B. Chen, Q. Yan, and L. Xiao, “Phan- tomsound: Black-box, query-efficient audio adversarial attack via split-second phoneme injection,” inProceedings of the 26th Inter- national Symposium on Research in Attacks, Intrusions and De- fenses, 2023, pp. 366–380

  10. [10]

    Vsmask: De- fending against voice synthesis attack via real-time predictive per- turbation,

    Y . Wang, H. Guo, G. Wang, B. Chen, and Q. Yan, “Vsmask: De- fending against voice synthesis attack via real-time predictive per- turbation,” inProceedings of the 16th ACM Conference on Se- curity and Privacy in Wireless and Mobile Networks, 2023, pp. 239–250

  11. [11]

    ASVspoof 2019: Future horizons in spoofed and fake audio de- tection,

    M. Todisco, X. Wang, V . Vestman, M. Sahidullah, H. Delgado, A. Nautsch, J. Yamagishi, N. Evans, T. Kinnunen, and K. A. Lee, “ASVspoof 2019: Future horizons in spoofed and fake audio de- tection,” inProc. Interspeech, 2019, pp. 1008–1012

  12. [12]

    ASVspoof 2021: accelerating progress in spoofed and deepfake speech detection,

    J. Yamagishi, X. Wang, M. Todisco, M. Sahidullah, J. Patino, A. Nautsch, X. Liu, K. A. Lee, T. Kinnunen, N. Evans, and H. Delgado, “ASVspoof 2021: accelerating progress in spoofed and deepfake speech detection,”IEEE/ACM Trans. Audio, Speech, and Language Processing, 2024

  13. [13]

    ASVspoof 5: Design, collection and val- idation of resources for spoofing, deepfake, and adversarial attack detection using crowdsourced speech,

    X. Wang, H. Delgado, H. Tak, J.-w. Jung, H.-j. Shim, M. Todisco, I. Kukanov, X. Liu, M. Sahidullah, T. Kinnunen, N. Evans, K. A. Lee, and J. Yamagishi, “ASVspoof 5: Design, collection and val- idation of resources for spoofing, deepfake, and adversarial attack detection using crowdsourced speech,”Computer Speech & Lan- guage, vol. 95, 2026

  14. [14]

    WaveFake: A data set to facilitate audio deepfake detection,

    J. Frank and L. Sch ¨onherr, “WaveFake: A data set to facilitate audio deepfake detection,” inProc. NeurIPS Datasets and Bench- marks Track, 2021

  15. [15]

    Does audio deepfake detection generalize?

    N. M. M ¨uller, P. Czempin, F. Dieckmann, A. Frober, and K. B¨ottinger, “Does audio deepfake detection generalize?”arXiv preprint arXiv:2203.16263, 2022

  16. [16]

    MLAAD: The multi-language audio anti-spoofing dataset,

    N. M. M ¨uller, P. Kawa, W. H. Choong, E. Casanova, E. G ¨olge, T. M ¨uller, P. Syga, P. Sperl, and K. B ¨ottinger, “MLAAD: The multi-language audio anti-spoofing dataset,” inProc. IJCNN, 2024

  17. [17]

    V oiceWukong: Benchmark- ing deepfake voice detection,

    Z. Yan, Y . Zhao, and H. Wang, “V oiceWukong: Benchmark- ing deepfake voice detection,”arXiv preprint arXiv:2409.06348, 2024

  18. [18]

    V oxCPM: Tokenizer-free TTS for context- aware speech generation and true-to-life voice cloning,

    Y . Zhou, G. Zeng, X. Liu, X. Li, R. Yu, Z. Wang, R. Ye, W. Sun, J. Gui, K. Liet al., “V oxCPM: Tokenizer-free TTS for context- aware speech generation and true-to-life voice cloning,”arXiv preprint arXiv:2509.24650, 2025

  19. [19]

    Qwen3-TTS technical report,

    H. Hu, X. Zhu, T. He, D. Guo, B. Zhang, X. Wang, Z. Guo, Z. Jiang, H. Hao, Z. Guoet al., “Qwen3-TTS technical report,” arXiv preprint arXiv:2601.15621, 2026

  20. [20]

    GLM-TTS technical report,

    J. Cui, Z. Yang, N. Li, J. Tian, X. Ma, Y . Zhang, G. Chen, R. Yang, Y . Cheng, Y . Zhouet al., “GLM-TTS technical report,”arXiv preprint arXiv:2512.14291, 2025

  21. [21]

    CosyV oice 3: Towards in-the- wild speech generation via scaling-up and post-training,

    Z. Du, C. Gao, Y . Wang, F. Yu, T. Zhao, H. Wang, X. Lv, H. Wang, X. Shi, K. Anet al., “CosyV oice 3: Towards in-the- wild speech generation via scaling-up and post-training,”arXiv preprint arXiv:2505.17589, 2025

  22. [22]

    Chatterbox: Open-source flow-matching TTS with emotion exaggeration control,

    Resemble AI, “Chatterbox: Open-source flow-matching TTS with emotion exaggeration control,” https://huggingface.co/ resemble-ai/chatterbox, 2025

  23. [23]

    FlashLabs Chroma 1.0: A real-time end-to-end spoken dialogue model with personalized voice cloning,

    T. Chen, T. Chen, K. Shen, Z. Bao, Z. Zhang, M. Yuan, and Y . Shi, “FlashLabs Chroma 1.0: A real-time end-to-end spoken dialogue model with personalized voice cloning,”arXiv preprint arXiv:2601.11141, 2026

  24. [24]

    VibeV oice: Expressive podcast generation with next-token diffusion,

    Z. Peng, J. Yu, W. Wang, Y . Chang, Y . Sun, L. Dong, Y . Zhu, W. Xu, H. Bao, Z. Wanget al., “VibeV oice: Expressive podcast generation with next-token diffusion,” inProc. ICLR, 2025

  25. [25]

    Zero-shot voice conversion with diffusion trans- formers,

    S. Liuet al., “Zero-shot voice conversion with diffusion trans- formers,”arXiv preprint arXiv:2411.09943, 2024

  26. [26]

    OpenV oice V2,

    MyShell AI, “OpenV oice V2,” https://huggingface.co/myshell-ai/ OpenV oiceV2, 2024

  27. [27]

    Retrieval-based-voice-conversion- webui (RVC),

    RVC-Project, “Retrieval-based-voice-conversion- webui (RVC),” https://github.com/RVC-Project/ Retrieval-based-V oice-Conversion-WebUI, 2024

  28. [28]

    Lib- riSpeech: An ASR corpus based on public domain audio books,

    V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Lib- riSpeech: An ASR corpus based on public domain audio books,” inProc. ICASSP, 2015, pp. 5206–5210

  29. [29]

    V oxPopuli: A large-scale multilingual speech corpus for repre- sentation learning, semi-supervised learning and interpretation,

    C. Wang, M. Riviere, A. Lee, A. Wu, C. Talber, J. Bhosaleet al., “V oxPopuli: A large-scale multilingual speech corpus for repre- sentation learning, semi-supervised learning and interpretation,” inProc. ACL, 2021, pp. 993–1003

  30. [30]

    AASIST: A new end-to-end anti- spoofing system using integrated spectro-temporal graph attention networks,

    J.-w. Jung, H.-S. Heo, H. Tak, H.-j. Shim, J. S. Chung, B.-J. Lee, H.-J. Yu, and N. Evans, “AASIST: A new end-to-end anti- spoofing system using integrated spectro-temporal graph attention networks,” inProc. ICASSP, 2022, pp. 6367–6371

  31. [31]

    End-to-end spectro-temporal graph attention networks for speaker verification anti-spoofing and speech deepfake detec- tion,

    H. Tak, J.-w. Jung, J. Patino, M. Kamble, M. Todisco, and N. Evans, “End-to-end spectro-temporal graph attention networks for speaker verification anti-spoofing and speech deepfake detec- tion,” inProc. 2021 Edition of the Automatic Speaker Verification and Spoofing Countermeasures Challenge, 2021, pp. 1–8

  32. [32]

    AST: Audio spectrogram transformer,

    Y . Gong, Y .-A. Chung, and J. Glass, “AST: Audio spectrogram transformer,” inProc. Interspeech, 2021, pp. 571–575

  33. [33]

    ECAPA- TDNN: Emphasized channel attention, propagation and aggrega- tion in TDNN based speaker verification,

    B. Desplanques, J. Thienpondt, and K. Demuynck, “ECAPA- TDNN: Emphasized channel attention, propagation and aggrega- tion in TDNN based speaker verification,” inProc. Interspeech, 2020, pp. 3830–3834

  34. [34]

    Facer: Contrastive attention based expression recognition via smartphone earpiece speaker,

    G. Wang, Q. Yan, S. Patrarungrong, J. Wang, and H. Zeng, “Facer: Contrastive attention based expression recognition via smartphone earpiece speaker,” inIEEE INFOCOM 2023-IEEE conference on computer communications. IEEE, 2023, pp. 1– 10