REVIEW 4 major objections 5 minor 36 references
When Fine-Tuning is Not Enough: Lessons from HSAD on Hybrid and Adversarial Audio Spoof Detection
T0 review · 4 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash
Pith's one-line read Fine-tuning transformer models on a new hybrid spoof audio dataset lifts four-way detection past 97 percent accuracy and near 99 percent F1, and exposes that pretrained models collapse on mixed human-and-synthetic speech.
desk verdict A worthwhile hybrid-spoofing benchmark idea, but the paper's own numbers contradict each other and the central fine-tuning results are not reproducible. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
HSAD itself is the central object: a four-class taxonomy that forces models to separate human, cloned, zero-shot AI-generated, and hybrid audio, with hybrid utterances built by splicing human and synthetic segments in three arrangements (H→S, S→H, and alternating) and smoothed with cross-fade. Clean and degraded versions (additive noise, 4 kHz low-pass filtering, Opus compression) plus per-sample metadata make failure modes visible. The evaluative mechanism is the reliability score, a scalar per-sample confidence extracted from the four-class output and thresholded for binary decisions; the paper organizes its evidence around reliability-score distributions and confusion matrices before and
What would settle it
Recompute the paper's headline numbers from the published confusion matrices after fixing a concrete definition of reliability_score (for example, the softmax probability assigned to the human class) and applying the Section 6.2 threshold rule; if the fine-tuned AST models no longer land at 97-98 percent accuracy and about 99 percent F1, the central fine-tuning result is an artifact of the unspecified metric. A second test: fine-tune on HSAD clean audio only and evaluate on the three degraded versions separately, and likewise hold out one hybrid configuration such as alternating H–S–H; the cla
Extended reading notes
Core claim
The paper claims that hybrid spoofed audio—utterances that splice genuine and synthetic speech—is a distinct attack class that current binary detectors are not built to handle. It constructs HSAD to test this: 1,248 clean and 1,248 degraded utterances across human, cloned, zero-shot AI-generated, and hybrid classes, annotated with spoofing method, speaker identity, and degradation metadata. Across the evaluated transformer models (AST variants, Wav2Vec2, and HuBERT), the paper reports three lessons: pretrained models often collapse (for example, Wav2Vec2-large predicted one class for 97.67 percent of samples, and MIT-AST misclassified every genuine human utterance); spoof-specific fine-tunin
Load-bearing premise
Every reported accuracy, F1, and error-rate figure depends on converting the model's four-class outputs into a single 'reliability score', but the paper never defines that conversion, so the headline numbers cannot be reproduced from the text as written.
Editorial extensions
If this is right
- Binary genuine/spoof accuracy on clean benchmarks is not a trustworthy proxy for deployed voice authentication; mixed-source utterances must be part of standard evaluation.
- Off-the-shelf transformer encoders should not be used as spoof detectors without spoof-specific adaptation, since pretrained Wav2Vec2 and HuBERT can perform near or below chance on four-way hybrid classification.
- Fine-tuning on a hybrid-aware corpus can take the same models from near-random to above 97 percent accuracy and roughly 99 percent F1, with large reductions in both false positives on human speech and false negatives on synthetic content.
- Even after fine-tuning, complex hybrids (AI-generated + human, AI-clone + AI-generated) remain the hardest cases, so purely data-driven fine-tuning needs to be supplemented by calibration, segment-level modeling, or structural cues.
- The release of HSAD with rich metadata (speaker, spoof type, hybrid configuration, degradation) supports fine-grained analysis of which attack factors drive detector failures.
Reading between the lines
- A consequence the authors leave implicit: aggregate accuracy can be high while every genuine user is rejected (as MIT-AST did on HSAD), so deployments should report per-class and per-user error rates, not only overall F1.
- Because the clean and degraded versions of HSAD are generated from the same source clips, the current fine-tuning mixes the effect of hybrid exposure with the effect of augmentation; training on clean hybrids and testing on degraded hybrids separately would isolate the two.
- Since every hybrid sample has known splice boundaries, HSAD can directly support a segment-level localization test—whether a frame-level detector marks exactly the synthetic spans—without requiring new data collection.
- The reported gains should be treated as provisional until the reliability-score mapping is published; once it is, the same protocol can be applied to any speech encoder, making HSAD a reusable stress test rather than a one-off benchmark.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces HSAD, a proposed four-class (human, cloned, AI-generated, hybrid) audio spoofing benchmark with clean and degraded versions, and evaluates several transformer-based models before and after fine-tuning. The central claims are that pretrained models collapse on hybrid and degraded audio, that fine-tuning on HSAD yields very high accuracy (AST models >97%, F1 ≈99%), and that hybrid-aware benchmarks such as HSAD are therefore essential. The paper also reports reliability-score statistics, confusion matrices, and false-positive/false-negative reductions to support these conclusions.
Significance. If the presented results were internally consistent and reproducible, HSAD would be a potentially useful contribution to audio anti-spoofing evaluation, particularly by emphasizing hybrid compositions and degradations that are underrepresented in ASVspoof-style benchmarks. The qualitative finding that fine-tuning on a domain-specific benchmark can substantially improve four-way classification is plausible and worth investigating. However, the manuscript currently contains multiple contradictory statements about the dataset size, undefined evaluation metrics, and quantitative results that cannot be reconciled with any single reading of the corpus. These issues prevent the reader from verifying the central empirical claims, so the contribution cannot be assessed in its present form.
major comments (4)
- [Abstract vs. §3.1, §4.6, §4.8, Table 4, §6.6.1] The dataset size is described inconsistently. The abstract states '1,248 clean and 41,044 degraded utterances,' while §3.1, §4.6, and §4.8 describe a corpus of 1,248 clean and 1,248 degraded utterances (312 per class). Table 4 reports test-set denominators of 8,549 and 8,459 for the HSAD evaluation, which exceed the entire stated corpus of 2,496 under any described split; these denominators are instead close to 20% of the abstract's 42,292 total (8,458.4). Likewise, §6.6.1 reports 14,117 false negatives for AI-generated content, which is larger than the total number of samples in either reading. Because the accuracies and F1 scores in Tables 3–6 and the fine-tuning gains in §6.6 depend on a corpus whose size is never fixed, the central quantitative claims cannot be verified.
- [§6.2, Eq. (1); Tables 3, 5, 6] The key scalar 'reliability_score' is never defined. The binary decision rule in §6.2 uses '|real_tag - reliability_score| < 0.5', but the manuscript does not specify how the four-class softmax outputs are converted to a scalar reliability score, nor what 'real_tag' denotes per class. Tables 3, 5, and 6 report mean, std, max, min, and mode of this score, and §6.6.1 attributes model-collapse and calibration conclusions to it. Without an explicit, reproducible definition, all reliability-score statistics and the associated binary FPR/FNR analyses are not computable by a reader, and cross-model comparisons based on this quantity are not well defined.
- [§3.2, §6.4, §6.5, §6.6] The paper is inconsistent about how many models are evaluated and what they are called. The Introduction and §6.4 say six models, §3.2 says five models, and the text in §6.6.1 discusses only two fine-tuned models (Model A and Model B) while §6.6.2 discusses three more (Wav2Vec2/HuBERT). The baseline list in Table 2 includes WpythonW with no corresponding HSAD fine-tuning result, and the naming shifts between 'MIT-AST,' 'AST-MIT,' 'MattyB95,' 'AST-MattyB95,' and model letters. This ambiguity makes it difficult to attribute any particular claim to a concrete architecture, and the claimed six-model benchmark is not consistently executed or reported.
- [§6.6.1 and §6.7] The reported false-positive and false-negative counts are incompatible with the described dataset. The text states that Model A reduced human-speech false positives from 1,207 to 197 and that Model B reduced AI-generated false negatives from 14,117 to 684, with an F1 of 99%. Even under the larger 42,292-sample reading, 14,117 false negatives for a single class would imply a class size that is not reported anywhere; under the stated 1,248+1,248 corpus, the number 1,207 already exceeds the total number of human utterances. These numbers cannot be derived from either dataset description, so the claimed reductions and F1 improvements are not grounded in any reproducible experimental setup.
minor comments (5)
- [§6.5] The text says 'Table 1 presents the number of correct predictions and accuracy' but the actual table is Table 2. Please fix the cross-reference.
- [§6.6.2 heading] The heading 'Fine-Trued Models' contains a typo; it should be 'Fine-Tuned Models.'
- [References [17], [18], [19], [27]] Reference [18] and [19] are identical duplicates of the same MIT model page, and [17] is listed as MattyB95's ASVspoof5 model while later text refers to MattyB95 as 'AST-ASVspoof2019'. Please reconcile the reference list and the Hugging Face identifiers with the actual models used.
- [§3.1, §4.4, §6.3] The paper alternates between describing four classes and six 'spoofing groups' (G1–G6). The relationship between the four-class taxonomy (Human, Cloned, AI-generated, Hybrid) and the G4–G6 groups (AI+Human, AI Clone+AI Generated, Human Recombined) is not clearly defined. Clarify how the six groups map to the four classes used for the softmax head.
- [§6.2] The symbol 'real_tag' appears in Eq. (1) without explanation. Even if 'reliability_score' were defined, 'real_tag' needs a formal definition or the equation should be rewritten in terms of class probabilities.
Circularity Check
No formal derivation circularity; mild self-referential validation of HSAD's utility as a benchmark.
-
other
[Abstract; §6.6.1 item 3; §7 Conclusion]
"These findings demonstrate that fine-tuning alone is not sufficient—robust hybrid-aware benchmarks like HSAD are essential to expose calibration failures, model biases, and factors affecting spoof detection in adversarial environments."
The paper's overall lesson is that hybrid-aware benchmarks like HSAD are essential, but the supporting 'findings' are measurements made on HSAD itself. Failure of pretrained models on hybrid groups and residual errors after fine-tuning are properties of the authors' own dataset and evaluation protocol, not of an independent or external hybrid corpus. Therefore the conclusion that HSAD is an essential benchmark is not derived from independent evidence; it is a self-referential validation: the benchmark's utility is asserted on the basis of observations that the same benchmark was constructed to produce. This is mild, not an equation-level input=output reduction, but it is a genuine validation-circularity in the paper's central 'lessons' claim.
full rationale
The paper contains no closed-form derivation whose output equals an input; its central numbers are empirical evaluations on a newly introduced dataset. I therefore find no equation-level, fitted-parameter, self-citation, or uniqueness-theorem circularity. The only quasi-circular element is the benchmark-utility claim: HSAD is constructed by the authors, and then the paper uses model behavior on HSAD to conclude that 'robust hybrid-aware benchmarks like HSAD are essential.' That is a mild self-validation, because the failure modes that motivate the conclusion are generated by the same dataset the conclusion endorses; no independent cross-corpus hybrid benchmark is used to establish necessity. I also flag two non-circularity but important reproducibility problems: the reliability-score mapping in §6.2 is undefined (the formula references 'real_tag' and 'reliability_score' but neither is defined), and dataset-size/test-denominator statements conflict (abstract: 41,044 degraded vs §3.1/§4.6/§4.8: 1,248; test counts 8,549/8,459 cannot arise from the stated 2,496-sample corpus). These are correctness/reproducibility concerns, not circular steps.
Assumptions & free parameters
free parameters (1)
- binary decision threshold =
0.5
assumptions (3)
- domain assumption The reliability score is computable from the 4-class softmax output
- domain assumption The speaker-disjoint 80/20 split of 12 speakers yields a representative generalization test
- ad hoc to paper The reported test-set denominators are consistent with the described HSAD size
Cite this review
Pith. "Pith review of When Fine-Tuning is Not Enough: Lessons from HSAD on Hybrid and Adversarial Audio Spoof Detection." pith.science (2026). https://pith.science/paper/KYXEJJLX
@misc{pith2026250907323,
author = {Pith},
title = {Pith review of: When Fine-Tuning is Not Enough: Lessons from HSAD on Hybrid and Adversarial Audio Spoof Detection},
year = {2026},
howpublished = {\url{https://pith.science/paper/KYXEJJLX}},
note = {Machine review of arXiv:2509.07323}
}
read the original abstract
The rapid advancement of AI has enabled highly realistic speech synthesis and voice cloning, posing serious risks to voice authentication, smart assistants, and telecom security. While most prior work frames spoof detection as a binary task, real-world attacks often involve hybrid utterances that mix genuine and synthetic speech, making detection substantially more challenging. To address this gap, we introduce the Hybrid Spoofed Audio Dataset (HSAD), a benchmark containing 1,248 clean and 41,044 degraded utterances across four classes: human, cloned, zero-shot AI-generated, and hybrid audio. Each sample is annotated with spoofing method, speaker identity, and degradation metadata to enable fine-grained analysis. We evaluate six transformer-based models, including spectrogram encoders (MIT-AST, MattyB95-AST) and self-supervised waveform models (Wav2Vec2, HuBERT). Results reveal critical lessons: pretrained models overgeneralize and collapse under hybrid conditions; spoof-specific fine-tuning improves separability but struggles with unseen compositions; and dataset-specific adaptation on HSAD yields large performance gains (AST greater than 97 percent and F1 score is approximately 99 percent), though residual errors persist for complex hybrids. These findings demonstrate that fine-tuning alone is not sufficient-robust hybrid-aware benchmarks like HSAD are essential to expose calibration failures, model biases, and factors affecting spoof detection in adversarial environments. HSAD thus provides both a dataset and an analytic framework for building resilient and trustworthy voice authentication systems.
Figures
Figures from the paper (7 more)
Reference graph
Works this paper leans on
-
[1]
Md Sahidullah Alam et al. 2022. Partial Spoof: A Dataset for Localized Audio Deepfake Detection. InInterspeech
work page 2022
-
[2]
Md Sahidullah Alam, Héctor Delgado, Massimiliano Todisco, Nicholas Evans, and Junichi Yamagishi. 2021. ASVspoof 2021: Challenge Evaluation Plan. In Proceedings of the 2021 ISCA Speaker Odyssey Workshop. 376–382
work page 2021
- [3]
-
[4]
S. Chen, L. Chen, J. Zhang, K. Lee, Z. Ling, and L. Dai. 2024. Adversarial Speech for Voice Privacy Protection from Personalized Speech Generation. InICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Seoul, Korea, Republic of, 11411 – 11415. https://doi.org/10.1109/ ICASSP48485.2024.10447699
arXiv 2024
-
[5]
Yafeng Chen, Siqi Zheng, Hui Wang, Luyao Cheng, , et al. 2024. ERes2NetV2: Boosting Short-Duration Speaker Verification Performance with Computational Efficiency. (2024)
work page 2024
-
[6]
Haowei Cheng, Candy Olivia Mawalim, et al. 2023. Analysis of Spectro-Temporal Modulation Representation for Deep-Fake Speech Detection. InAsia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC). 1822–1829
work page 2023
-
[7]
O. Chouchane, C. Busch, C. Galdi, N. Evans, and M. Todisco. 2024. A Comparison of Differential Performance Metrics for the Evaluation of Automatic Speaker Verification Fairness.arXiv preprint(2024). arXiv:arXiv:2404.17810
work page Pith review arXiv 2024
-
[8]
Nicholas Evans, Zhenhua Wu, Tomi Kinnunen, Junichi Yamagishi, and Ferrer Alegre. 2015. Spoofing and Countermeasures for Automatic Speaker Verification: A Survey.Speech Communication66 (2015), 130–153
work page 2015
Show all 36 references
-
[9]
Chirag Goel et al. 2024. Towards Attention-based Contrastive Learning for Audio Spoof Detection.arXiv preprint arXiv:2407.03514(2024)
2024 arXiv
-
[10]
Yuan Gong, Yu-An Chung, and James Glass. 2021. AST: Audio Spectrogram Transformer. InInterspeech 2021. ISCA, 571–575
2021
-
[11]
Ibrar, A
S. Ibrar, A. Javed, and H. Ilyas. 2023. Voice Presentation Attacks Detection using Acoustic MLTP Features and BiLSTM. In2023 International Conference on Communication, Computing and Digital Systems (C - CODE). Islamabad, Pakistan, 1 – 5. https://doi.org/10.1109/C-CODE58145.202...
2023
-
[12]
Anssi Kanervisto et al . 2022. Optimizing Tandem Speaker Verification and Anti-Spoofing Systems.IEEE/ACM Transactions on Audio, Speech, and Language Processing30 (2022), 477–488
2022
-
[13]
J. S. Kim. 2016. Development of a Customized User Control of Digital Audio in a Smart Home. In2016 Joint 8th International Conference on Soft Computing and Intelligent Systems (SCIS) and 17th International Symposium on Advanced Intelligent Systems (ISIS). Sapporo, Japan, 665 –...
2016
-
[14]
Z. Lei, H. Yan, C. Liu, M. Ma, and Y. Yang. 2022. Two - Path GMM - ResNet and GMM - SENet for ASV Spoofing Detection. InICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Singapore, Singapore, 6377 – 6381. https://doi.org/10.1...
2022
-
[15]
Menglu Li, Yasaman Ahmadiadli, and Xiao-Ping Zhang. 2024. Audio Anti- Spoofing Detection: A Survey.arXiv preprint arXiv:2404.13914(2024)
2024 arXiv
-
[16]
Y. Liu, J. Chen, J. Gao, and S. Gai. 2023. A Survey of Speech Recognition Based on Deep Learning. In2023 5th International Conference on Natural Language Process- ing (ICNLP). Guangzhou, China, 151 – 156. https://doi.org/10.1109/ICNLP58431. 2023.00034
2023
-
[17]
MattyB95. 2024. AST-ASVspoof5-Synthetic-Voice-Detection. https://huggingface. co/MattyB95/AST-ASVspoof5-Synthetic-Voice-Detection
2024
-
[19]
MIT. 2022. ast-finetuned-audioset-10-10-0.4593. https://huggingface.co/MIT/ast- finetuned-audioset-10-10-0.4593
2022
-
[20]
Andreas Nautsch, Xin Wang, Nicholas Evans, et al. 2021. ASVspoof 2019: Spoofing Countermeasures for the Detection of Synthesized, Converted and Replayed Speech.IEEE Transactions on Biometrics, Behavior, and Identity Science3, 2 (2021), 252–265
2021
-
[21]
Sushant Patil et al. 2022. Micro-Signatures: Physical Acoustic Cues for Deepfake Audio Detection. InInterspeech
2022
-
[22]
Ricardo Reimao and Vassilios Tzerpos. 2019. For: A Dataset for Synthetic Speech Detection. InInternational Conference on Speech Technology and Human-Computer Dialogue (SpeD)
2019
-
[23]
Massimiliano Todisco, Hector Delgado, and Nicholas Evans. 2016. Constant Q Cepstral Coefficients: A Spoofing Countermeasure for Automatic Speaker Verification. InProceedings of Odyssey: The Speaker and Language Recognition Workshop. ISCA, 283–290
2016
-
[24]
Ville Vestman, Massimiliano Todisco, Xin Wang, Md Sahidullah, Hector Delgado, Andreas Nautsch, and Kong Aik Lee. 2019. Future Horizons in Spoofed and Fake Audio Detection.arXiv preprint arXiv:1904.05441(2019)
2019 arXiv
-
[25]
K. V. VS and S. Naveed. 2024. A Review of Automatic Speaker Verification Systems with Feature Extractions and Spoofing Attacks. In2024 5th International Confer- ence on Electronics and Sustainable Communication Systems (ICESC). Coimbatore, India, 1999 – 2005. https://doi.org/1...
2024
-
[26]
Yan Wen, Zhenchun Lei, Yingen Yang, Changhong Liu, and Minglei Ma. 2022. Multi-Path GMM-MobileNet Based on Attack Algorithms and Codecs for Syn- thetic Speech and Deepfake Detection.Interspeech(2022)
2022
-
[27]
WpythonW. 2025. AST-ASVspoof2019-elevenlabs. https://huggingface.co/ WpythonW/AST_ASVspoof2019_elevenlabs
2025
-
[28]
Xugang Wu et al. 2020. Adversarial Examples for Automatic Speech Recognition: A Survey. InIEEE ICASSP
2020
-
[29]
Jiangyan Yi, Jianhua Tao, et al. 2021. Half-truth: A Partially Fake Audio Detection Dataset.arXiv preprint arXiv:2104.03617(2021)
2021 arXiv
-
[30]
Jiangyan Yi, Jianhua Tao, et al. 2023. ADD 2023: the Second Audio Deepfake Detection Challenge.arXiv preprint arXiv:2305.13774(2023)
2023 arXiv
-
[31]
Yan Zhang, Md Sahidullah, and Nicholas Evans. 2021. Spoofing Detection Goes Noisy: An Analysis of Synthetic Speech Detection under Additive Noises. IEEE/ACM Transactions on Audio, Speech, and Language Processing29 (2021), 2091–2106
2021
- [32]
-
[33]
Decoupled Weight Decay Regularization,
I. Loshchilov and F. Hutter, “Decoupled Weight Decay Regularization, ” inInter- national Conference on Learning Representations (ICLR), 2019. arXiv:1711.05101
2019 arXiv
-
[34]
SGDR: Stochastic Gradient Descent with Warm Restarts,
I. Loshchilov and F. Hutter, “SGDR: Stochastic Gradient Descent with Warm Restarts, ” inInternational Conference on Learning Representations (ICLR), 2017. arXiv:1608.03983
2017 arXiv
-
[35]
wav2vec 2.0: A framework for self-supervised learning of speech representations,
A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations, ” inAdvances in Neural Information Processing Systems (NeurIPS), vol. 33, pp. 12449–12460, 2020
2020
-
[36]
DeiT III: Revenge of the ViT,
H. Touvron, M. Cord, and H. Jégou, “DeiT III: Revenge of the ViT, ” inComputer Vision – ECCV 2022 (Lecture Notes in Computer Science, vol. 13684), pp. 516–533, Springer, Cham, 2022. doi: 10.1007/978-3-031-20053-3_30
2022 doi
-
[37]
HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,
W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “HuBERT: Self-supervised speech representation learning by masked prediction of hidden units, ”IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 3451–3460, 2021
2021
Reviewed August 4, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.