Pith. sign in

REVIEW 5 major objections 6 minor 30 references

Hybrid Audio Detection Using Fine-Tuned Audio Spectrogram Transformers: A Dataset-Driven Evaluation of Mixed AI-Human Speech

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fine-tuned transformers catch hybrid AI-human speech at 97%

desk verdict A promising dataset idea buried under an evaluation that does not add up; the splice-seam confound is real, and the reported numbers are internally impossible. read the letter →

arxiv 2505.15136 v1 pith:Q4YW35W7 submitted 2025-05-21 cs.SD cs.CReess.AS

Kunyang Huang , Bin Hu This is my paper
classification cs.SDcs.CReess.AS
keywords hybridaudiodetectionSpectrogramTransformervoicecloningAI-generatedspeechanti-spoofingpartialforgeryauthenticationHSADdataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that detecting 'hybrid' speech, utterances that splice genuine human audio with AI-generated or cloned segments, requires both a purpose-built dataset and models tuned on it. It introduces a hybrid spoofed audio dataset with four balanced classes across six composition groups, and fine-tunes two Audio Spectrogram Transformer variants on it. The paper reports that the fine-tuned models reach 97% test accuracy and 99% F1, with hybrid audio classified at 100% precision, whereas untuned baselines either misclassify all human speech or separate classes poorly. The aim is to move anti-spoofing benchmarks beyond binary genuine-versus-fake classification toward the composite attacks that occur in practice.

What carries the argument

The central object is the Audio Spectrogram Transformer (AST), a Vision-Transformer-style model that converts 128-bin log-Mel spectrograms into patches and applies self-attention over the full time-frequency grid. Two pretrained AST variants, one general audio model and one already tuned for synthetic-voice detection, are re-fine-tuned on the hybrid dataset with a four-way softmax head, 6-second padded inputs, Adam at a learning rate of 2e-5, and early stopping. The load-bearing dataset design is the hybrid construction: genuine and synthetic segments are concatenated in three patterns (human-to-synthetic, synthetic-to-human, interleaved) with 10 ms cross-fades, plus a human-recombined group of spliced genuine segments, and six annotated groups across clean and noisy or degraded partitions.

What would settle it

A reader could settle this by testing the fine-tuned models on genuinely mixed recordings, for example a real person speaking over a phone channel with an AI voice inserted in the same acoustic environment, or by ablating the seam: synthesize hybrid content with no cross-fade artifact versus a smooth continuous transition and compare accuracy. If accuracy collapses when seam artifacts are removed, the classifier was keying on the splice.

Watch

Extended reading notes

Core claim

The central claim is that a transformer trained on a corpus containing mixed human-synthetic utterances learns cues that let it separate four classes, human, cloned, AI-generated, and hybrid, at 97% accuracy and 99% F1, with all hybrid test samples correctly identified. The discovery is that composition itself is learnable: after fine-tuning, both models recognize spliced speech as its own category, even though the starting checkpoints were built for binary or general audio discrimination. The paper also demonstrates that fine-tuning fixes the baseline failure mode where a general-purpose audio transformer labeled every genuine human utterance as spoofed.

Load-bearing premise

The claim rests on hybrid samples made by stitching real and synthetic clips with a 10-millisecond fade standing in for realistic mixed human-AI speech; if the seam leaves an audible or machine-detectable artifact, the near-perfect scores may be detecting the splice rather than mixed-source speech.

Editorial extensions

If this is right

  • Four-way classification becomes feasible: a single model can flag human, cloned, AI-generated, and hybrid audio at 97% accuracy rather than only binary real-versus-fake.
  • Hybrid audio, the class no current benchmark models, receives 100% precision in this test, meaning spliced utterances are distinguishable as a category after fine-tuning.
  • Fine-tuning on the hybrid dataset fixes the calibration failure of a general audio transformer that misclassified all genuine human utterances, a necessary condition for usable anti-spoofing.
  • The reported reductions in false positives and false negatives imply that deployment metrics beyond raw accuracy improve materially on speaker-disjoint splits.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • I infer that the 10 ms cross-fade is the least-tested variable in the pipeline; a controlled experiment comparing no-fade splices, cross-faded splices, and naturally seamless mixed speech would reveal how much of the 97% is composition detection versus seam detection.
  • The human-recombined group is a step in this direction, but the paper does not report whether models trained with it reject seamless human-human splicing; that comparison would isolate the synthetic-content cue.
  • A transferable implication not stated in the paper is that the same fine-tuning recipe could be applied to spotting partial forgeries in long-form audio such as phone calls or meetings by sliding-window classification over utterance boundaries, since the model already learns hybrid segments as a distinct class.
  • Because the dataset is only 1,248 clean utterances from twelve speakers, the reported accuracy likely upper-bounds cross-speaker generalization; an external evaluation on unseen voices would test whether the learned cues are speaker-independent.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper constructs a new dataset, HSAD, containing human, cloned, AI-generated, and hybrid (mixed human-AI) speech, and proposes fine-tuned Audio Spectrogram Transformer (AST) models for four-way classification. The authors report that their fine-tuned models achieve about 97% accuracy and 99% F1-score on this dataset, claiming that this significantly outperforms existing baselines. The dataset includes six groups (G1-G6), with G6 a human-recombined control, and a noisy/codec-degraded version. The paper also evaluates two baseline AST models on ASVspoof 2019 LA and on HSAD.

Significance. The problem of detecting hybrid human-AI speech is timely and relevant for voice authentication security, and the dataset construction is systematic in several respects: it varies cloning context levels, creates multiple hybrid composition patterns, and includes a human-recombined control group (G6) to address seam artifacts. If the reported results were reproducible and the control analysis were provided, the dataset and fine-tuning approach could be a useful contribution to the anti-spoofing community. However, the central quantitative claims are undermined by internal numerical inconsistencies and by the absence of the very control experiment needed to distinguish seam detection from genuine mixed-source recognition. No code or dataset release is indicated, further limiting reproducibility. Because the principal evidence is not internally consistent, the significance of the claimed 97% accuracy cannot currently be assessed.

major comments (5)
  1. [5.6] The claim that the MIT-AST baseline achieved 93.67% accuracy on the HSAD dataset while 'failed to correctly identify any genuine human utterances' is numerically impossible on a balanced 1,248-utterance, four-class dataset. With 312 human samples, misclassifying all of them caps accuracy at 75% (936/1248 correct), so 93.67% cannot be reached. This internal contradiction makes the baseline results unreliable.
  2. [5.7] The reported false-negative reduction for AI-generated content, from 14,117 to 684, is incompatible with the dataset size described in Section 3.6, which contains only 312 AI-generated samples in the clean set and 1,248 total clean utterances. A false-negative count of 14,117 is larger than the entire dataset, indicating that these counts are not derived from the proposed HSAD evaluation and undermining the claimed performance improvements.
  3. [5.3 / 5.7] Section 5.3 introduces the G6 (Human Recombined) group as a control for hybrid-like flows without spoof artifacts, but Section 5.7 never reports performance of the fine-tuned models on G6. Without this control, the reported 100% precision on hybrid audio may reflect detection of the 10 ms cross-fade seam described in Section 3.4 rather than recognition of mixed human-AI content. This control experiment is essential to validate the central claim.
  4. [5.5-5.7] The abstract and conclusion claim that the approach 'significantly outperforms existing baselines,' but no comparison is made on any existing benchmark such as HAD or ADD2023-PF. The ASVspoof 2019 LA results in Table 2 are for baseline models only, not for the fine-tuned models. The fine-tuned models are evaluated exclusively on the self-constructed HSAD, so the claimed superiority over existing methods and the generalization of the approach are not established.
  5. [3.6 / 5.3] The dataset statistics are inconsistent between sections: Section 3.6 describes 1,248 clean utterances in four balanced classes, while Section 5.3 describes six groups (G1-G6) without stating how these map to the four classes or the sample size per group. This ambiguity prevents the reader from reconciling the reported reliability statistics and confusion matrices with the stated dataset design.
minor comments (6)
  1. [4.2.4] There is an incomplete citation '[ ? ]' for the DeiT model; the reference should be completed.
  2. [Title / metadata] The ACM template metadata still contains placeholder text ('Conference acronym ’XX, June 03–05, 2018' and 2018 copyright), which should be updated to the actual venue and year.
  3. [Table 3] The reported mode values of 5.96e-7 for reliability scores are implausibly small for a model output that is described as a reliability score; the score scale and the thresholding rule should be clarified.
  4. [5.2] The thresholding rule defined in Section 5.2 is for binary reliability-based classification, but the experiments use four-class classification; the authors should explain how the threshold is applied or how the four-class decision is derived.
  5. [Abstract / 5.7] The abstract reports '97% classification accuracy,' while Section 5.7 gives validation accuracies of 97.88% and 98.08% and test accuracy 'stabilized at 97%'; these figures should be reconciled for consistency.
  6. [4.1 / References] The model name 'MattyB95/AST-ASVspoof5-Synthetic-Voice-Detection' in the keywords and Section 4.1 differs from the reference [17] title 'AST-ASVspoof2019-Synthetic-Voice-Detection'; please reconcile the naming.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the pipeline is standard supervised training and held-out evaluation; the paper's validity issues (missing G6 control, inconsistent baseline numbers) are not circularity.

full rationale

I traced the claimed derivation chain: dataset construction (Section 3), AST fine-tuning (Section 4), and evaluation on an 80/20 speaker-disjoint held-out split (Section 5.1). Accuracy is defined in Section 5.2 as C/N, i.e., the fraction of correct predictions on the test set. Nothing in these definitions makes the reported 97% accuracy, the 99% F1-score, or the 100% hybrid precision equal to a training objective, a fitted parameter, or a label. The models are not evaluated on their own training utterances, so the central result is a standard held-out performance claim rather than a tautology. The comparison against unfine-tuned baselines is unfair but not circular: the baselines are not fitted to HSAD, so 'outperforming' them is not a by-construction consequence. I also checked for self-citation load-bearing arguments or imported uniqueness theorems: none appear; references are to external models and datasets. The paper does have serious non-circularity concerns that I flag for the correctness pass: Section 5.3 defines the G6 'Human Recombined' control but Section 5.7 never evaluates the fine-tuned models on it, leaving the hybrid-detection result potentially confounded by the 10 ms cross-fade seam; and Section 5.6 reports 93.67% accuracy on the balanced 1,248-utterance HSAD set while claiming MIT-AST failed to identify any genuine human utterance, which is arithmetically impossible (if all 312 human samples were wrong, accuracy could not exceed 75%). These are internal-consistency and experimental-design problems, not cases where a result reduces by definition or by self-citation to its own inputs. Therefore the circularity score is 0.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The paper introduces no theoretical constructs. The only fitted quantities are the neural network weights produced by fine-tuning, which are not released. The evaluation rests on the domain assumption that concatenated hybrid clips represent real-world mixed human-AI attacks; without external validation on existing benchmarks, this assumption remains untested.

free parameters (1)
  • Fine-tuned model weights = Not released (Model A approx. 86.2M parameters; Model B not specified)
    The reported 97% accuracy is an outcome of these learned weights. Since weights, training code, and data are not provided, the central result cannot be independently reproduced.
assumptions (4)
  • domain assumption Hybrid clips made by concatenating genuine and synthetic segments with 10 ms cross-fade are representative of real-world mixed AI-human speech attacks.
    The entire evaluation's validity depends on this. If the seam is detectable, the 97% accuracy reflects the splice artifact. Location: Section 3.4.
  • domain assumption Speaker-disjoint 80/20 split removes speaker identity leakage so that the reported test accuracy estimates generalization to new speakers.
    With only 12 speakers, the split size is tiny and the paper gives no details on how speakers were assigned. Location: Section 5.1.
  • domain assumption Pretrained AST representations from AudioSet and ASVspoof transfer to the new hybrid spoof detection task.
    Standard transfer learning assumption, not independently validated in this paper. Location: Section 4.2.3.
  • standard math The 128-bin log-Mel spectrogram with 25 ms window and 10 ms hop is treated as a sufficient representation for the four-way classification.
    Standard audio processing; the paper provides no ablation showing that this representation is sufficient for hybrid detection.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Hybrid Audio Detection Using Fine-Tuned Audio Spectrogram Transformers: A Dataset-Driven Evaluation of Mixed AI-Human Speech." pith.science (2026). https://pith.science/paper/Q4YW35W7

@misc{pith2026250515136,
  author       = {Pith},
  title        = {Pith review of: Hybrid Audio Detection Using Fine-Tuned Audio Spectrogram Transformers: A Dataset-Driven Evaluation of Mixed AI-Human Speech},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q4YW35W7}},
  note         = {Machine review of arXiv:2505.15136}
}
read the original abstract

The rapid advancement of artificial intelligence (AI) has enabled sophisticated audio generation and voice cloning technologies, posing significant security risks for applications reliant on voice authentication. While existing datasets and models primarily focus on distinguishing between human and fully synthetic speech, real-world attacks often involve audio that combines both genuine and cloned segments. To address this gap, we construct a novel hybrid audio dataset incorporating human, AI-generated, cloned, and mixed audio samples. We further propose fine-tuned Audio Spectrogram Transformer (AST)-based models tailored for detecting these complex acoustic patterns. Extensive experiments demonstrate that our approach significantly outperforms existing baselines in mixed-audio detection, achieving 97\% classification accuracy. Our findings highlight the importance of hybrid datasets and tailored models in advancing the robustness of speech-based authentication systems.

Figures

Figures reproduced from arXiv: 2505.15136 by the authors.

Figure 1
Figure 1. Our evaluation framework in a smart home audio token [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Construction process of the hybrid sentence dataset, en [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 4
Figure 4. Confusion matrix – MIT fine-tuned (Model A) [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Confusion matrix – MattyB95 fine-tuned (Model B) [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Comparison of false positives, false negatives, F1-score, [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

30 extracted references · 14 canonical work pages

  1. [1]

    Md Sahidullah Alam et al. 2022. Partial Spoof: A Dataset for Localized Audio Deepfake Detection. In Interspeech

    work page 2022

  2. [2]

    Md Sahidullah Alam, Héctor Delgado, Massimiliano Todisco, Nicholas Evans, and Junichi Yamagishi. 2021. ASVspoof 2021: Challenge Evaluation Plan. In Proceedings of the 2021 ISCA Speaker Odyssey Workshop . 376–382

    work page 2021

  3. [3]

    Brewster

    T. Brewster. 2021. Fraudsters cloned company director’s voice in $35 million bank heist, police find. https://www.forbes.com/sites/ thomasbrewster/2021/10/14/huge-bankfraud-uses-deep-fake-voice-tech- to-steal-millions/?sh=2d60538a7559. Online; accessed January 10, 2023

    2021

  4. [4]

    S. Chen, L. Chen, J. Zhang, K. Lee, Z. Ling, and L. Dai. 2024. Adversarial Speech for Voice Privacy Protection from Personalized Speech Generation. In ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Seoul, Korea, Republic of, 11411 – 11415. https://doi.org/10.1109/ ICASSP48485.2024.10447699

    arXiv 2024

  5. [5]

    Yafeng Chen, Siqi Zheng, Hui Wang, Luyao Cheng, , et al. 2024. ERes2NetV2: Boosting Short-Duration Speaker Verification Performance with Computational Hybrid Audio Detection Using Fine-Tuned Audio Spectrogram Transformers: A Dataset-Driven Evaluation of Mixed AI-Human Speech Conference acronym ’XX, June 03–05, 2018, Woodstock, NY Efficiency. (2024)

    work page 2024

  6. [6]

    Haowei Cheng, Candy Olivia Mawalim, et al. 2023. Analysis of Spectro-Temporal Modulation Representation for Deep-Fake Speech Detection. In Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC). 1822–1829

    work page 2023

  7. [7]

    Chouchane, C

    O. Chouchane, C. Busch, C. Galdi, N. Evans, and M. Todisco. 2024. A Comparison of Differential Performance Metrics for the Evaluation of Automatic Speaker Verification Fairness. arXiv preprint (2024). arXiv:arXiv:2404.17810

    arXiv 2024

  8. [8]

    Nicholas Evans, Zhenhua Wu, Tomi Kinnunen, Junichi Yamagishi, and Ferrer Alegre. 2015. Spoofing and Countermeasures for Automatic Speaker Verification: A Survey. Speech Communication 66 (2015), 130–153

    work page 2015

Show all 30 references
  1. [9]

    Chirag Goel et al. 2024. Towards Attention-based Contrastive Learning for Audio Spoof Detection. arXiv preprint arXiv:2407.03514 (2024)

    2024 arXiv

  2. [10]

    Yuan Gong, Yu-An Chung, and James Glass. 2021. AST: Audio Spectrogram Transformer. In Interspeech 2021. ISCA, 571–575

    2021

  3. [11]

    Ibrar, A

    S. Ibrar, A. Javed, and H. Ilyas. 2023. Voice Presentation Attacks Detection using Acoustic MLTP Features and BiLSTM. In 2023 International Conference on Communication, Computing and Digital Systems (C - CODE) . Islamabad, Pakistan, 1 – 5. https://doi.org/10.1109/C-CODE58145.2...

    2023

  4. [12]

    Anssi Kanervisto et al . 2022. Optimizing Tandem Speaker Verification and Anti-Spoofing Systems. IEEE/ACM Transactions on Audio, Speech, and Language Processing 30 (2022), 477–488

    2022

  5. [13]

    J. S. Kim. 2016. Development of a Customized User Control of Digital Audio in a Smart Home. In 2016 Joint 8th International Conference on Soft Computing and Intelligent Systems (SCIS) and 17th International Symposium on Advanced Intelligent Systems (ISIS). Sapporo, Japan, 665 ...

    2016

  6. [14]

    Z. Lei, H. Yan, C. Liu, M. Ma, and Y. Yang. 2022. Two - Path GMM - ResNet and GMM - SENet for ASV Spoofing Detection. In ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . Singapore, Singapore, 6377 – 6381. https://doi.org/10...

    2022

  7. [15]

    Menglu Li, Yasaman Ahmadiadli, and Xiao-Ping Zhang. 2024. Audio Anti- Spoofing Detection: A Survey. arXiv preprint arXiv:2404.13914 (2024)

    2024 arXiv

  8. [16]

    Y. Liu, J. Chen, J. Gao, and S. Gai. 2023. A Survey of Speech Recognition Based on Deep Learning. In 2023 5th International Conference on Natural Language Process- ing (ICNLP). Guangzhou, China, 151 – 156. https://doi.org/10.1109/ICNLP58431. 2023.00034

    2023

  9. [17]

    MattyB95. 2024. AST-ASVspoof5-Synthetic-Voice-Detection. https://huggingface. co/MattyB95/AST-ASVspoof5-Synthetic-Voice-Detection

    2024

  10. [19]

    MIT. 2022. ast-finetuned-audioset-10-10-0.4593. https://huggingface.co/MIT/ast- finetuned-audioset-10-10-0.4593

    2022

  11. [20]

    Andreas Nautsch, Xin Wang, Nicholas Evans, et al. 2021. ASVspoof 2019: Spoofing Countermeasures for the Detection of Synthesized, Converted and Replayed Speech. IEEE Transactions on Biometrics, Behavior, and Identity Science 3, 2 (2021), 252–265

    2021

  12. [21]

    Sushant Patil et al. 2022. Micro-Signatures: Physical Acoustic Cues for Deepfake Audio Detection. In Interspeech

    2022

  13. [22]

    Ricardo Reimao and Vassilios Tzerpos. 2019. For: A Dataset for Synthetic Speech Detection. In International Conference on Speech Technology and Human-Computer Dialogue (SpeD)

    2019

  14. [23]

    Massimiliano Todisco, Hector Delgado, and Nicholas Evans. 2016. Constant Q Cepstral Coefficients: A Spoofing Countermeasure for Automatic Speaker Verification. In Proceedings of Odyssey: The Speaker and Language Recognition Workshop. ISCA, 283–290

    2016

  15. [24]

    Ville Vestman, Massimiliano Todisco, Xin Wang, Md Sahidullah, Hector Delgado, Andreas Nautsch, and Kong Aik Lee. 2019. Future Horizons in Spoofed and Fake Audio Detection. arXiv preprint arXiv:1904.05441 (2019)

    2019 arXiv

  16. [25]

    K. V. VS and S. Naveed. 2024. A Review of Automatic Speaker Verification Systems with Feature Extractions and Spoofing Attacks. In 2024 5th International Confer- ence on Electronics and Sustainable Communication Systems (ICESC) . Coimbatore, India, 1999 – 2005. https://doi.org...

    2024

  17. [26]

    Yan Wen, Zhenchun Lei, Yingen Yang, Changhong Liu, and Minglei Ma. 2022. Multi-Path GMM-MobileNet Based on Attack Algorithms and Codecs for Syn- thetic Speech and Deepfake Detection. Interspeech (2022)

    2022

  18. [27]

    WpythonW. 2025. AST-ASVspoof2019-elevenlabs. https://huggingface.co/ WpythonW/AST_ASVspoof2019_elevenlabs

    2025

  19. [28]

    Xugang Wu et al. 2020. Adversarial Examples for Automatic Speech Recognition: A Survey. In IEEE ICASSP

    2020

  20. [29]

    Jiangyan Yi, Jianhua Tao, et al. 2021. Half-truth: A Partially Fake Audio Detection Dataset. arXiv preprint arXiv:2104.03617 (2021)

    2021 arXiv

  21. [30]

    Jiangyan Yi, Jianhua Tao, et al. 2023. ADD 2023: the Second Audio Deepfake Detection Challenge. arXiv preprint arXiv:2305.13774 (2023)

    2023 arXiv

  22. [31]

    Yan Zhang, Md Sahidullah, and Nicholas Evans. 2021. Spoofing Detection Goes Noisy: An Analysis of Synthetic Speech Detection under Additive Noises. IEEE/ACM Transactions on Audio, Speech, and Language Processing 29 (2021), 2091–2106

    2021

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.