REVIEW 3 major objections 4 minor 26 references
Explicit forensic reasoning, trained on a 100K-sample chain-of-thought dataset, lets a multimodal audio LLM detect and localize partial deepfakes on unseen audio, with reported cross-dataset mAP of 74.67%.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-01 13:28 UTC pith:LWVYACTW
load-bearing objection Strong systems paper with a genuinely useful reasoning dataset, but the headline cross-dataset numbers are compromised by test-set selection of loss weights. the 3 major comments →
ThinkOmni: A Reasoning-Driven Omni-Modal LLM Framework for Audio Forgery Detection and Localization
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The paper's central claim is that AFDL benefits from explicit, supervised reasoning over multi-modal forensic evidence. ThinkOmni generates a <think> rationale covering speaker consistency, spectral artifacts, prosodic features, and related dimensions, followed by a three-way detection label (fully real/fully fake/partially fake) and interval predictions. The authors assert that this explicit reasoning, trained via FACoT and aligned through FMIL with an omni-modal LLM backbone, yields strong cross-dataset generalization: 74.67% mAP on ADD and Speech-Forensics, 43.73 points over the best SSL-based method and 15.32 over the best audio-LLM method, plus detection gains of 4.57% ACC and 7.39% F1
What carries the argument
The load-bearing components are three named objects. FACoT (Forensic-Aware Chain-of-Thought) is a 100K-sample dataset of structured rationales linking manipulation cues to labels; it is what makes reasoning supervision possible. FMIL (Forensic-Aware Modality-Incremental Learning) is the training regime that progressively aligns semantic, acoustic (through the SAFE module), and spectrogram-visual encoders with the LLM backbone, freezing earlier stages to avoid interference. FCML (Forensic-Consistent Multi-task Loss) is the objective that weights reasoning tokens down and applies an adaptive IoU-based temporal localization loss, coordinating the three outputs. The argument depends on these pie
Load-bearing premise
The headline cross-dataset gains rest on the assumption that ADD 2023 Track 2 and Speech-Forensics were never used for any development decision; the paper's own Table 11 shows the final loss-weight configuration was chosen from those exact test sets, so without a proper held-out validation split the reported generalization advantage may be inflated.
What would settle it
Retrain the pipeline with hyperparameters (especially the token weights) chosen on a validation set that strictly excludes ADD and Speech-Forensics, then evaluate on both; if the margin over the best ALLM baseline collapses to near zero, the generalization claim is falsified.
If this is right
- If correct, reasoning supervision generalizes better than low-level artifact matching: the same FACoT-style rationales could be applied to other forensic tasks, such as deepfake video or image tampering localization.
- The large cross-dataset gain (mAP from ~31% to ~75% on unseen data) implies that partial-forgery localization is not hopeless under distribution shift, contrary to the near-zero performance of several SSL baselines.
- The progressive FMIL schedule, rather than joint training, is credited with improving cross-dataset mACC, mF1, and mAP by 9.52%, 9.99%, and 3.95%, suggesting that staged modality alignment is a reusable recipe for multimodal LLM adaptation.
- The FCML ablation (removing role-based token weighting drops cross-dataset mAP from 70.82% to 55.91%) indicates that sequence-loss balance is essential when reasoning tokens dominate the output.
- The reasoning outputs are human-inspectable: human and MLLM judges rated the generated rationales higher than shuffled rationales, suggesting the model's explanations track the audio rather than being generic text.
Where Pith is reading between the lines
- A direct evaluative follow-up: freeze the token weights (0.2, 0.2, 0.6) using a validation split that excludes ADD and Speech-Forensics, then re-measure cross-dataset margins; this would quantify how much of the reported generalization is due to the method versus test-set-aware tuning.
- The FACoT pipeline's CLAP-based filtering removes dimensions with low audio-text similarity, but the paper's own failure cases show rationales can still cite plausible-sounding cues (e.g., 'metallic ringing' on a genuine recording). A testable extension is to add a grounding or factuality reward that penalizes rationales referencing cues contradicted by the audio.
- The observed detection-localization trade-off at the SFA stage (CoT improves mAP but slightly lowers ACC/F1) hints that reasoning supervision biases the model toward boundary prediction; a multi-objective calibration or abstention mechanism might recover detection accuracy without sacrificing localization.
- Beyond audio, this design suggests that explicit reasoning supervision, progressive modality alignment, and role-weighted losses could be transferred to other multi-modal integrity tasks, such as audio-visual deepfake detection, where cross-modal consistency is a natural forensic cue.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes ThinkOmni, a Qwen2.5-Omni-based framework for audio forgery detection and localization (AFDL). In addition to a prediction label and temporal intervals, the model is trained to generate an explicit forensic reasoning chain. The authors construct FACoT, a 100K-sample chain-of-thought dataset with structured annotations, introduce FMIL, a three-stage progressive alignment of semantic, acoustic, and spectral-visual encoders, and propose FCML, a loss combining role-weighted cross-entropy with an adaptive localization loss. The main empirical claims are strong intra-dataset performance and, more importantly, cross-dataset generalization to ADD 2023 Track 2 and Speech-Forensics: the paper reports a cross-dataset mAP of 74.67%, beating the best SSL baseline by 43.73% and the best ALLM baseline by 15.32%, and detection gains of 4.57% ACC / 7.39% F1 over the best ALLM baseline. The paper includes ablations of the dataset construction, the FMIL stages, the loss components, and reasoning quality, and the authors state they will release code, models, and data.
Significance. If the evaluation were clean, this would be a substantial contribution to AFDL: it is one of the first attempts to supervise an omni-modal LLM with explicit forensic reasoning while jointly optimizing detection and localization, and the reported cross-dataset gains are large. The construction of a 100K-sample reasoning dataset, the careful ablation of the progressive training stages, and the candid observation that CoT supervision can reduce cross-dataset detection accuracy are notable strengths. The release of code, models, and data would support replication and follow-up work. However, the significance is currently capped by the evaluation-protocol concerns described below, especially the documented use of the cross-dataset test sets for hyperparameter selection.
major comments (3)
- [Supplementary Table 11 / Section F.3; Eq. (8)] The headline cross-dataset results are not clean held-out numbers. Table 11 reports mACC/mF1/mAP on the ADD 2023 Track 2 and Speech-Forensics test sets for seven token-weight configurations, and the final configuration (ω_think, α_det, ω_loc) = (0.2, 0.2, 0.6) is explicitly selected because it gives the highest mACC, mF1, and overall average. These weights enter the FCML objective in Eq. (8) and therefore determine the evaluated model. This is model selection on the cross-dataset test sets, so the 'unseen' status of ADD/SF is compromised and the reported margins (e.g., 74.67% mAP) are optimistically biased. The same concern applies to any other hyperparameters selected using cross-dataset metrics (Tables 3–5). Please re-select all hyperparameters on a validation split that does not include ADD/SF, e.g., a held-out subset of the FACoT pool, and report the final ADD/SF evaluation separatel
- [Appendix H.4 / Table 6] The reasoning-quality evaluation is partly circular. The MLLM judge prompt in H.4 explicitly provides the ground-truth detection and localization labels, and instructs the judge to assess whether the assistant's prediction aligns with those labels. The human evaluators are presumably given the same information. Thus the high reasoning scores in Table 6 partly measure the model's ability to restate label-conditional rationales rather than the forensic validity of the reasoning. This is compounded by the fact that FACoT annotations are generated from ground-truth labels/timestamps (H.2). Please re-run the reasoning evaluation without disclosing the labels to the judges, or clearly reinterpret the scores as consistency with the reference rather than independent evidence quality.
- [Section 4.3 and Section 5.4] Several loss weights are stated to be 'empirically set' (λ_fr = 0.3, λ_pf = 0.5, λ_loc = 0.5) and the CLAP filtering threshold (0.2) is given without sensitivity analysis. If any of these were tuned on the ADD/SF test sets, the same test-set contamination as in Table 11 applies. Even if they were fixed in advance, the paper should say so explicitly and report a sensitivity analysis or a validation-based selection. This is currently a gap in the reproducibility of the reported configuration.
minor comments (4)
- [Tables 1–2] Please clarify whether the 'Avg.' columns are unweighted averages over the datasets. ADD and Speech-Forensics have different sizes, so an unweighted average may not reflect the pooled test performance.
- [Table 2] The header 'SFAvg' appears to be a typo; it should be 'SF Avg.'.
- [Section 4.3] The class-prior weights ω_cls = (0.36, 0.24, 0.40) are said to follow FACoT class proportions. If the target test distribution is different, the benefit is not guaranteed; a sentence acknowledging this would be helpful.
- [Abstract / Section 4.1] The term 'omni-modal' is slightly misleading: the model uses audio, spectrogram, and text but no video modality, even though Qwen2.5-Omni is omni-modal. A more precise term such as 'tri-modal' would avoid overclaiming.
Circularity Check
Cross-dataset headline numbers are partly selected on the test sets (Table 11); reasoning-quality evaluation is label-conditioned, but the core detection/localization losses are not definitionally circular.
specific steps
-
fitted input called prediction
[Supplementary Sec. F.3, Table 11; Sec. 5.1 data setup; Eq. (8)]
"Ablation of token-weighting factors under cross-dataset evaluation. ... The configuration (ω_think, α_det, ω_loc)=(0.2, 0.2, 0.6) provides the highest mACC, mF1, and overall average."
The role-specific token weights entering Eq. (8) are selected by maximizing mACC/mF1/mAP on ADD and Speech-Forensics, the identical datasets later used to report the headline cross-dataset gains (80.74 ACC / 85.15 F1 / 74.67 mAP). The final configuration is therefore fitted to the evaluation target, so the reported 'unseen' generalization numbers are not a clean held-out estimate; the prediction is biased by the selection criterion.
-
other
[App. H.2 / H.4 (FACoT annotation and MLLM evaluation prompts); Sec. 3.2]
"The annotation prompt deliberately supplies the reference detection and localization metadata. ... The corresponding ground-truth label information includes the detection outcome ({gt_detection_result}) and the localization result ({gt_localization_result}). ... Your score should reflect the quality of the assistant’s reasoning and whether its prediction aligns with the ground-truth labels above."
The reference rationales in FACoT are generated by LLMs from the ground-truth labels/timestamps, and the reasoning-quality judges are given the same ground-truth labels and asked to score agreement with them. Consequently the Table 6 reasoning-quality scores partly measure label-conditional paraphrase fidelity rather than independent causal grounding of the rationales. This affects the 'explicit reasoning' sub-claim, but not the detection/localization metrics directly.
full rationale
The central detection/localization pipeline (FMIL, SAFE, FCML) is not definitionally circular: Eq. (3)-(8) define losses over outputs, and Tables 1-2 compare against retrained baselines on test data. No load-bearing uniqueness theorem or ansatz is imported from the authors' prior work; self-citations are contextual. The main circularity is test-set hyperparameter selection: Table 11 chooses (ω_think, α_det, ω_loc)=(0.2,0.2,0.6) using ADD/SF cross-dataset metrics, so the headline cross-dataset gains are partly fitted to the evaluation sets rather than predicted. A secondary label-conditioning issue affects the reasoning-quality evaluation (Table 6, App. H.4) because rationales and judges both receive ground-truth labels. These issues materially weaken, but do not fully erase, the independent content of the method; the score is therefore 5.
Axiom & Free-Parameter Ledger
free parameters (5)
- Role-specific token weights (ω_think, α_det, ω_loc) =
(0.2, 0.2, 0.6)
- Localization loss weights (λ_fr, λ_pf, λ_loc) =
λ_fr=0.3, λ_pf=0.5, λ_loc=0.5
- CLAP similarity threshold =
0.2
- Class-prior detection weights ω_cls =
(0.36, 0.24, 0.40)
- SAFE design hyperparameters (D_k, frequency scaling s, gate dims) =
D_k=256, s=1.5
axioms (5)
- domain assumption Pretrained semantic (Whisper-large-v3), acoustic (XLSR-300M), and vision (ViT) encoders carry transferable forensic cues into the LLM backbone.
- ad hoc to paper LLM-generated rationales, produced from ground-truth labels/timestamps, are valid enough to serve as supervision for generalizable reasoning.
- domain assumption CLAP similarity above 0.2 indicates that a reasoning dimension is audio-grounded.
- ad hoc to paper ADD 2023 Track 2 and Speech-Forensics are untouched by model development, including hyperparameter selection.
- domain assumption The condition-adaptive localization loss (Eq. 4) with zero loss for fully fake samples and zero-pushing for real samples is a faithful training target.
read the original abstract
Existing audio forgery detection and localization (AFDL) methods often overfit dataset-specific low-level artifacts, limiting their generalization to subtle, localized, and unseen manipulations. Recent audio large language model (ALLM)-based approaches cast AFDL as question answering but still model forensic evidence implicitly, without linking manipulation cues to predictions. To bridge this gap, we propose ThinkOmni, a reasoning-driven omni-modal large language model that jointly performs explicit forensic reasoning, spoofing detection, and temporal manipulation localization. To enable explicit reasoning supervision, we construct Forensic-Aware Chain-of-Thought (FACoT), a 100K-sample dataset with structured forensic evidence and reasoning annotations. Leveraging FACoT, we introduce Forensic-Aware Modality-Incremental Learning (FMIL), which progressively aligns semantic, acoustic, and spectral-visual representations with the LLM backbone to capture complementary forensic cues. We further propose Forensic-Consistent Multi-task Loss (FCML), which combines weighted cross-entropy with an adaptive localization loss to coordinate reasoning generation, spoofing detection, and temporal localization. Extensive experiments show that ThinkOmni achieves strong cross-dataset generalization in both detection and localization. Code, models, data, and inference examples are available at https://beyond0814.github.io/ThinkOmni/.
Figures
Reference graph
Works this paper leans on
-
[1]
Zhixi Cai, Shreya Ghosh, Abhinav Dhall, Tom Gedeon, Kalin Stefanov, and Munawar Hayat. 2023. Glitch in the matrix: A large scale benchmark for content driven audio–visual forgery detection and localization.Computer Vision and Image Understanding236 (2023), 103818
2023
-
[2]
Zhixi Cai, Kartik Kuckreja, Shreya Ghosh, Akanksha Chuchra, Muhammad Haris Khan, Usman Tariq, Tom Gedeon, and Abhinav Dhall. 2025. Av-deepfake1m++: A large-scale audio-visual deepfake benchmark with real-world perturbations. InProceedings of the 33rd ACM International Conference on Multimedia. 13686– 13691
2025
-
[3]
Hao Gu, Jiangyan Yi, Chenglong Wang, Jianhua Tao, Zheng Lian, Jiayi He, Yong Ren, Yujie Chen, and Zhengqi Wen. 2025. Allm4add: Unlocking the capabilities of audio large language models for audio deepfake detection. InProceedings of the 33rd ACM International Conference on Multimedia. 11736–11745
2025
-
[4]
Sung-Feng Huang, Heng-Cheng Kuo, Zhehuai Chen, Xuesong Yang, Chao- Han Huck Yang, Yu Tsao, Yu-Chiang Frank Wang, Hung-yi Lee, and Szu-Wei Fu. 2024. Detecting the undetectable: Assessing the efficacy of current spoof detection methods against seamless speech edits. In2024 IEEE Spoken Language Technology Workshop (SLT). IEEE, 652–659
2024
-
[5]
Zhoulin Ji, Chenhao Lin, Hang Wang, and Chao Shen. 2024. Speech-Forensics: Towards Comprehensive Synthetic Speech Dataset Establishment and Analysis. InProceedings of the 33rd International Joint Conference on Artificial Intelligence, IJCAI 2024. 413–421
2024
-
[6]
Kartik Kuckreja, Parul Gupta, Injy Hamed, Thamar Solorio, Muhammad Haris Khan, and Abhinav Dhall. 2025. Tell me Habibi, is it Real or Fake?arXiv preprint arXiv:2505.22581(2025)
Pith/arXiv arXiv 2025
-
[7]
Jindong Li, Yali Fu, Li Fan, Jiahong Liu, Yao Shu, Chengwei Qin, Menglin Yang, Irwin King, and Rex Ying. 2025. Implicit reasoning in large language models: A comprehensive survey.arXiv preprint arXiv:2509.02350(2025)
Pith/arXiv arXiv 2025
-
[8]
Tianchi Liu, Duc-Tuan Truong, Rohan Kumar Das, Kong Aik Lee, and Haizhou Li. 2025. Nes2net: A lightweight nested architecture for foundation model driven speech anti-spoofing.IEEE Transactions on Information Forensics and Security20 (2025), 12005–12018
2025
-
[9]
Hieu-Thi Luong, Haoyang Li, Lin Zhang, Kong Aik Lee, and Eng Siong Chng. 2025. Llamapartialspoof: An llm-driven fake speech dataset simulating disinformation generation. InICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 1–5
2025
-
[10]
Andreas Nautsch, Xin Wang, Nicholas Evans, Tomi H Kinnunen, Ville Vestman, Massimiliano Todisco, Héctor Delgado, Md Sahidullah, Junichi Yamagishi, and Kong Aik Lee. 2021. ASVspoof 2019: Spoofing countermeasures for the detection of synthesized, converted and replayed speech.IEEE Transactions on Biometrics, Behavior, and Identity Science3, 2 (2021), 252–265
2021
-
[11]
Eros Rosello, Alejandro Gómez Alanís, Angel M Gomez, Antonio M Peinado, N Harte, J Carson-Berndsen, and G Jones. 2023. A conformer-based classifier for variable-length utterance processing in anti-spoofing.. InInterspeech, Vol. 2023. 5281–5285
2023
-
[12]
Yao Shi, Hui Bu, Xin Xu, Shaoji Zhang, and Ming Li. 2021. AISHELL-3: A Multi- Speaker Mandarin TTS Corpus.Interspeech 2021(2021)
2021
-
[13]
Hemlata Tak, Massimiliano Todisco, Xin Wang, Jee-weon Jung, Junichi Yamag- ishi, and Nicholas Evans. 2022. Automatic Speaker Verification Spoofing and Deepfake Detection Using Wav2vec 2.0 and Data Augmentation. InThe Speaker and Language Recognition Workshop (Odyssey 2022). ISCA
2022
-
[14]
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. 2023. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805(2023)
Pith/arXiv arXiv 2023
-
[15]
Duc Tuan Truong, Ruijie Tao, Tuan Nguyen, Hieu Thi Luong, Kong Aik Lee, and Eng Siong Chng. 2024. Temporal-Channel Modeling in Multi-head Self-Attention for Synthetic Speech Detection. In25th Interspeech Conferece 2024. 537–541
2024
-
[16]
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems35 (2022), 24824–24837
2022
-
[17]
Junyan Wu, Wei Lu, Xiangyang Luo, Rui Yang, Qian Wang, and Xiaochun Cao
-
[18]
Yuankun Xie, Haonan Cheng, Yutian Wang, and Long Ye. 2024. An efficient temporary deepfake location approach based embeddings for partially spoofed audio detection. InICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 966–970
2024
-
[19]
Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Jun- yang Lin. 2025. Qwen2.5-Omni Technical Report.arXiv preprint arXiv:2503.20215 (2025)
Pith/arXiv arXiv 2025
-
[20]
Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yuxuan Wang, Xian Shi, Ting He, Xinfa Zhu, et al. 2025. Qwen3-omni technical report. arXiv preprint arXiv:2509.17765(2025)
Pith/arXiv arXiv 2025
-
[21]
Jiangyan Yi, Ye Bai, Jianhua Tao, Haoxin Ma, Zhengkun Tian, Chenglong Wang, Tao Wang, and Ruibo Fu. 2021. Half-truth: A partially fake audio detection dataset.arXiv preprint arXiv:2104.03617(2021)
Pith/arXiv arXiv 2021
-
[22]
Jiangyan Yi, Jianhua Tao, Ruibo Fu, Xinrui Yan, Chenglong Wang, Tao Wang, Chu Yuan Zhang, Xiaohui Zhang, Yan Zhao, Yong Ren, et al. 2023. ADD 2023: the Second Audio Deepfake Detection Challenge. InCEUR Workshop Proceedings, Vol. 3597. 125–130
2023
-
[23]
Lin Zhang, Xin Wang, Erica Cooper, Nicholas Evans, and Junichi Yamagishi. 2022. The partialspoof database and countermeasures for the detection of short fake speech segments embedded in an utterance.IEEE/ACM Transactions on Audio, Speech, and Language Processing31 (2022), 813–825
2022
-
[24]
Qishan Zhang, Shuangbing Wen, and Tao Hu. 2024. Audio deepfake detection with self-supervised xls-r and sls classifier. InProceedings of the 32nd ACM Inter- national Conference on Multimedia. 6765–6773
2024
-
[25]
Jiafeng Zhong, Bin Li, and Jiangyan Yi. 2024. Enhancing Partially Spoofed Audio Localization with Boundary-aware Attention Mechanism. In25th Annual Conference of the International Speech Communication Association, Interspeech 2024, Kos, Greece, September 1-5, 2024
2024
-
[2024]
InProceedings of the 32nd ACM International Conference on Multimedia
Coarse-to-fine proposal refinement framework for audio temporal forgery detection and localization. InProceedings of the 32nd ACM International Conference on Multimedia. 7395–7403
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.