REVIEW 4 major objections 4 minor 37 references
Combining predictions from four models—three visual and one audio—reaches 72.8% accuracy on a balanced deepfake test set, roughly three points above the best single model.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-04 04:26 UTC pith:FMAUXZ7T
load-bearing objection Honest engineering benchmark with a useful negative audio result, but the headline ensemble gain is not established because thresholds and fusion selection may have been tuned on the evaluation set. the 4 major comments →
Ensemble Deep Learning Approaches for AI-Altered Video Detection
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central discovery is that a weighted-voting ensemble of four detectors, processing video frames and audio separately, achieves 72.8% accuracy on a balanced sample from the FakeA VCeleb dataset, outperforming the best single model (XceptionNet at 69.8%). The gain is modest and tied to this test distribution: the audio model, AASIST, which scores 99% on its training benchmark, collapses to near-random (49–53%) on the same test videos, so the ensemble leans almost entirely on visual features. Voting-based fusion outperforms score-averaging, and performance collapses when only audio is manipulated, confirming that domain gap and unseen manipulation types, not fusion strategy, are the binding
What carries the argument
The load-bearing mechanism is the ensemble fusion layer: each base model emits a fake-probability score, and five fusion rules (mean, weighted average, majority voting, weighted voting, and stacking) combine them into a final decision. Weighted voting performs best by converting scores into hard votes before aggregation. The pipeline uses MTCNN for face cropping and a separate path for 16 kHz mono audio, with 0.5 used as a neutral fallback for missing scores. This design allows direct comparison of fusion strategies and isolates the contribution of each modality.
Load-bearing premise
The reported accuracy numbers assume that decision thresholds and the fusion-method choice were fixed before the balanced test set was inspected; the paper does not fully document this, and threshold tuning alone can swing one model's accuracy from 30% to 61%.
What would settle it
Re-run the evaluation with all thresholds fixed at 0.5 and the fusion method chosen before seeing any test labels; if weighted voting no longer beats the best single model by a meaningful margin, the reported ensemble gain is an artifact of test-set-informed tuning. Alternatively, train the same ensemble on only one manipulation type and test on a disjoint newer type; if accuracy falls to chance, generalization to unseen manipulations is not achieved.
If this is right
- Weighted voting, not stacking or averaging, is the most reliable fusion rule for this set of detectors, adding about three accuracy points over the best single model.
- A state-of-the-art audio anti-spoofing model trained on ASVspoof 2019 does not transfer to real-world deepfake video audio; audio-only detection on balanced real/fake audio is near chance (49–53%).
- Ensemble accuracy collapses to roughly 51% when only the audio is manipulated (real video plus fake audio), showing the system effectively ignores the audio channel and cannot catch audio-only fakes.
- Video models trained on FaceForensics++ and AIGVDBench retain 70–80% accuracy across different test-set compositions, making them far more robust than the audio model.
- Overall accuracy around 70% on a mixed test set indicates that unseen generation methods remain the core limitation, not the choice of ensemble strategy.
Where Pith is reading between the lines
- The audio branch would likely add real value only if retrained on deepfake video audio (or on the same manipulation types as the visual models) rather than on ASVspoof-style spoofed speech; a domain-adapted audio model could make the multimodal claim genuine.
- The superiority of voting over averaging suggests the models' raw confidence scores are poorly calibrated; calibrating scores before fusion could produce larger gains than adding more models.
- A direct test of the generalization claim would be to freeze the entire pipeline and evaluate on newer deepfake generators (e.g., diffusion-based video models); if the ~70% accuracy holds, the robustness claim is stronger, and if it drops sharply, the ensemble is tuned to training-era artifacts.
- Because the decision threshold was set around 0.2–0.4 to favor sensitivity, the system likely trades a higher false-positive rate for catching unseen manipulations; reporting precision-recall trade-offs would clarify the practical cost.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper describes a multimodal deepfake-detection pipeline combining one audio model (AASIST) and three visual models (EfficientNet-B1, XceptionNet, MesoNet), with scores fused by mean, majority voting, weighted voting, weighted averaging, and stacking. Models are trained on AIGVDBench, FaceForensics++, and ASVspoof 2019 LA, and evaluated on FakeA VCeleb. The main reported result is that weighted voting reaches 0.728 accuracy versus 0.698 for the best single model, while AASIST performs near chance on FakeA VCeleb. The authors conclude that ensembles improve robustness across manipulation types, that video models dominate, and that generalization to unseen manipulations remains the central open challenge.
Significance. If the evaluation protocol is sound, this is a useful empirical study: it compares five fusion strategies on an external benchmark, honestly documents the failure of an audio anti-spoofing model to transfer to in-the-wild data, and makes code publicly available. The negative result about AASIST and the per-manipulation-type breakdown are valuable. However, the central quantitative claim—that fusion yields a 3-point gain over the best single model—depends on selection details that are not reported, and the reported gain is within sampling error for the apparent test-set size. The paper's title-level claim about multimodal improvement is also weaker than the authors' own results: audio contributes little in every reported configuration.
major comments (4)
- [§5.2, §6.4, Tables 7–10] The evaluation protocol for thresholds and fusion hyperparameters is underspecified, and this is load-bearing for the headline gain. §6.4 states that the video decision threshold in each ensemble method is 'around 0.2–0.4', and §5.2 shows that EfficientNet's accuracy varies from 30% to 61% with threshold. The paper never states which data were used to select these thresholds, how weighted-voting weights were chosen, or how the stacking meta-model was trained (despite §2.5 describing a hold-out procedure). If thresholds, weights, or the fusion method were selected on the FakeA VCeleb evaluation split, then the reported 0.728 weighed-voting result is a fitted value, not a measured generalization. The authors must document the exact split used for every design choice, ideally with code logs, and report results on a held-out split not used for any selection.
- [Tables 7–11, §4.1] No measure of uncertainty is reported. The balanced test set appears to contain roughly 500–1000 videos (about 250 real videos, §4.1 and §5.5.1), so the standard error of an accuracy near 0.70 is approximately 0.015–0.021. The headline gain of 0.728 versus 0.698 is therefore within one to two standard errors and may be sampling noise. The differences among majority voting (0.712), weighted voting (0.728), and stacking (0.710) are similarly not distinguishable. Report confidence intervals, bootstrap estimates, or significance tests, and specify exact test-set sizes for every balanced category in Tables 12–14.
- [Abstract, §6.3, Table 14] The abstract's claim that 'using both audio and visual information together is a more reliable approach' is not supported by the paper's own evidence. In Table 14 (RvFa vs RvRa), where only audio is manipulated, AASIST achieves 0.532 and the ensemble 0.514, essentially chance; §6.3 explicitly says the audio modality is underused and the system relies heavily on video models. If the authors want to claim multimodal benefit, they must either show a setting where adding audio improves over the visual-only ensemble or temper the abstract to match the reported result that audio contributes little in this benchmark.
- [§4.1, §5.5.1] The dataset description is internally inconsistent. §4.1 states FakeA VCeleb contains 500 real videos and 19,500 deepfake videos (20,000 total), whereas §5.5.1 lists 21,544 total with 10,835 FvFa, 9,709 FvRa, 500 RvFa, and 500 RvRa. The magnitude of the imbalance and the construction of the balanced test set are also ambiguous: §4.1 says 'an equal number of videos from both the real and fake categories,' while §5.5.1 says 'the whole test set contains half the number of videos from each of the categories above.' Please reconcile the totals and specify exactly how many videos of each of the four types appear in the reported test sets.
minor comments (4)
- [§5.5.1] The heading 'FakeA VCeleb Validation Set result' conflicts with the surrounding text and with §4.1, which says the data were divided into validation and test sets. Clarify whether Tables 7–14 report validation, test, or combined results.
- [§5.2] The sentence 'By tuning the threshold, EfficientNet’s video accuracy can scale from 30% to 61%' should state the direction of the relationship and the threshold values used, since this is the same model family used in the ensemble.
- [§4.2 / Tables 1–5] The hyperparameter tables are useful, but the sampling rate for frame extraction (parameter n) is never specified. Also state whether the same extracted frames were used across all visual models.
- [§7] The limitations section correctly notes that no ablation study was performed, but an explicit table showing ensemble performance with each component removed would strengthen the paper and directly address the claim about audio contribution.
Circularity Check
No circular derivation; empirical evaluation on an external benchmark, with only a protocol ambiguity about threshold selection.
full rationale
The paper does not derive its main result from an assumption that already contains it. The ensemble accuracy (Tables 7-11) is measured by running independently trained models on the FakeA VCeleb benchmark, which is external to the training data (AIGVDBench, FaceForensics++, ASVspoof2019). No equation in the paper defines the target from a fitted parameter. The AASIST component is from the authors' prior work ([24], [29]), but this self-citation is not load-bearing: AASIST is an open-source, externally benchmarked model, and the paper actually shows it fails on FakeA VCeleb; the main conclusion rests on the video models and on measured fusion outputs. The only potentially circular-looking element is the decision-threshold selection: §5.2 reports that tuning the threshold moves EfficientNet accuracy from 30% to 61%, and §6.4 says the video decision threshold in each ensemble is 'around 0.2-0.4'. If thresholds or fusion choices had been selected on the same FakeA VCeleb partition whose accuracy is reported, the 0.728 vs 0.698 comparison would be a fitted number rather than an independent prediction. However, the paper does not state that the same partition was used for both selection and reporting, no concrete reduction can be quoted, and the results are presented as validation/test observations rather than as a theoretically forced outcome. Missing split/threshold specification is a reporting and generalizability concern, not demonstrated circularity. Therefore no circular step is established.
Axiom & Free-Parameter Ledger
free parameters (4)
- Per-model decision thresholds =
≈0.2–0.4 for video models; AASIST threshold undocumented
- Stacking meta-model parameters =
unspecified
- Test-set balancing ratio (real:fake) =
1:1 artificially balanced
- Fusion-method selection =
weighted voting / majority voting reported as best
axioms (3)
- domain assumption Scores from all four models are on a comparable fake-probability scale, making averaging, voting, and a 0.5 threshold meaningful.
- domain assumption Frame-level MTCNN face crops capture the manipulation artifacts of a video.
- domain assumption FakeA VCeleb is a fair proxy for in-the-wild multimodal deepfakes.
read the original abstract
The increasing accessibility of artificial intelligence has led to a rapid rise in AI-generated videos, making it more difficult to distinguish between real and manipulated content. Many existing detection methods rely on a single model and often struggle to generalize across different types of deepfakes. In this work, we developed a multimodal deepfake detection system that combines both audio and visual analysis using an ensemble of models. The system includes AASIST for audio-based detection, and EfficientNet, XceptionNet, and MesoNet for analyzing visual features in video frames. The pipeline takes a video as input, separates the audio, and extracts face frames using MTCNN. Each model produces a score indicating the likelihood of the input being fake. These scores are then combined using ensemble strategies, including mean averaging and stacking. Mean fusion provides a simple and stable baseline, while stacking uses a trained meta-model to learn how to combine predictions more effectively. Results show that while individual models perform well on the datasets they were trained on, their performance drops when tested on more diverse datasets. The ensemble approach helps improve overall robustness by combining predictions from multiple models, leading to more consistent performance across different types of deepfakes. This suggests that using both audio and visual information together is a more reliable approach for deepfake detection. Our results highlight generalization to unseen manipulations as the central open challenge, with average accuracy around 70%.
Reference graph
Works this paper leans on
-
[1]
Desi: Deepfake source identifier for social media,
K. Narayan et al., “Desi: Deepfake source identifier for social media,” in 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) , New Orleans, LA, USA, 2022, pp. 2857–2866. DOI: 10.1109/CVPRW56347.2022.00323
arXiv 2022
-
[2]
The spread of synthetic media on x,
G. Corsi, B. Marino, and W. Wong, “The spread of synthetic media on x,” Harvard Kennedy School (HKS) Misinformation Review, vol. 5, no. 3, Jun. 2024
2024
-
[3]
A brief history of deepfakes
G. Regan. “A brief history of deepfakes.” (Dec. 2025), [Online]. Available: https://www.realitydefender. com/insights/history-of-deepfakes
2025
-
[4]
Analysis of ai-generated content and deepfakes in social media,
B. To, “Analysis of ai-generated content and deepfakes in social media,” Metropolia University of Applied Sciences, Helsinki, Finland, 2024. [Online]. Available: https : / / www . theseus . fi / bitstream/handle/10024/857145/To_Billy.pdf
2024
-
[5]
Impact of deepfake technology on social media: Detection, misinformation and societal implications,
S. H. Al-khazraji, H. H. Saleh, A. I. Khalid, and I. A. Mishkhal, “Impact of deepfake technology on social media: Detection, misinformation and societal implications,”The Eurasia Proceedings of Science, Technology, Engineering and Mathematics, vol. 23, pp. 429–441, 2023. [Online]. Available: https: //www.researchgate.net/publication/374761496
arXiv 2023
-
[6]
E. R. Kismawadi, “The impact of deepfakes on trust and security in islamic banking: Emerging threats and mitigation strategies,” in Deepfakes and Their Impact on Business , G. Gupta et al., Eds., Hershey, PA, USA: IGI Global, 2025, pp. 99–122. DOI: 10.4018/979-8-3693-6890-9.ch005
-
[7]
Deepfake: Definitions, performance metrics and standards, datasets, and a meta-review,
E. Altuncu, V . N. L. Franqueira, and S. Li, “Deepfake: Definitions, performance metrics and standards, datasets, and a meta-review,” Frontiers in Big Data, vol. 7, p. 1 400 024, Sep. 2024. DOI: 10.3389/ fdata.2024.1400024
arXiv 2024
-
[8]
Deepfakes: A new disinformation threat?
M. Brady, “Deepfakes: A new disinformation threat?” Democracy Reporting International, Tech. Rep., Sep. 2020. [Online]. Available: https : / / democracyreporting . s3 . eu - central - 1 . amazonaws . com / images / 20842020 - 09 - 01 - DRI - deepfake - publication - no - 1.pdf
2020
-
[9]
Generative adversarial networks: Intro- duction and outlook,
K. Wang, C. Gou, Y . Duan, Y . Lin, X. Zheng, and F.-Y . Wang, “Generative adversarial networks: Intro- duction and outlook,” IEEE/CAA Journal of Automatica Sinica , vol. 4, no. 4, pp. 588–598, Oct. 2017. DOI: 10.1109/JAS.2017.7510583. 19
arXiv 2017
-
[10]
Generative adversarial network: An overview of theory and applications,
A. Aggarwal, M. Mittal, and G. Battineni, “Generative adversarial network: An overview of theory and applications,” International Journal of Information Management Data Insights, vol. 1, p. 100 004, 2021. DOI: 10.1016/j.jjimei.2020.100004
arXiv 2021
-
[11]
Diffusion models and generative artificial intelligence: Frameworks, applications and chal- lenges,
P. Kumar, “Diffusion models and generative artificial intelligence: Frameworks, applications and chal- lenges,” Archives of Computational Methods in Engineering , vol. 32, no. 7, 2025. DOI: 10 . 1007 / s11831-025-10266-z
2025
-
[12]
Diffusion models in vision: A survey,
F.-A. Croitoru et al., “Diffusion models in vision: A survey,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 9, pp. 10 850–10 869, Sep. 2023. DOI: 10.1109/TPAMI.2023. 3261988
-
[13]
How diffusion models work
Coding Tech. “How diffusion models work.” (Dec. 2025), [Online]. Available: https : / / www . youtube.com/watch?v=CZJgO7clruI (visited on 04/05/2026)
2025
-
[14]
A review of deepfake and its detection: From generative adversarial networks to diffusion models,
B. Liu, B. Liu, T. Zhu, and M. Ding, “A review of deepfake and its detection: From generative adversarial networks to diffusion models,” International Journal of Intelligent Systems , vol. 2025, p. 9 987 535,
2025
-
[15]
Ensemble classification and regression-recent developments, applications and future directions,
Y . Ren, L. Zhang, and P. N. Suganthan, “Ensemble classification and regression-recent developments, applications and future directions,” IEEE Computational Intelligence Magazine, vol. 11, no. 1, pp. 41– 53, Feb. 2016. DOI: 10.1109/MCI.2015.2471235
arXiv 2016
-
[16]
Stacked generalization: An introduction to super learning,
A. I. Naimi and L. B. Balzer, “Stacked generalization: An introduction to super learning,” European Journal of Epidemiology, vol. 33, no. 5, pp. 459–464, May 2018. DOI: 10.1007/s10654- 018- 0390-z
doi:10.1007/s10654- 2018
-
[17]
G ´eron, Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow, 2nd ed
A. G ´eron, Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow, 2nd ed. Sebastopol, CA, USA: O’Reilly Media, 2019
2019
-
[18]
Stacking ensemble machine learning with python
J. Brownlee. “Stacking ensemble machine learning with python.” (Apr. 2021), [Online]. Available: https://machinelearningmastery.com/stacking-ensemble-machine-learning- with-python/ (visited on 04/07/2026)
2021
-
[19]
Efficientnet: Rethinking model scaling for convolutional neural networks,
M. Tan and Q. V . Le, “Efficientnet: Rethinking model scaling for convolutional neural networks,” in Proceedings of the 36th International Conference on Machine Learning (ICML) , Long Beach, CA, USA, Jun. 2019, pp. 6105–6114. [Online]. Available: https://arxiv.org/abs/1905.11946
Pith/arXiv arXiv 2019
-
[20]
Efficientnet
Keras Team. “Efficientnet.” (), [Online]. Available: https://keras.io/2/api/applications/ efficientnet/
-
[21]
Chollet, Xception: Deep learning with depthwise separable convolutions, 2017
F. Chollet, Xception: Deep learning with depthwise separable convolutions, 2017. arXiv:1610.02357. [Online]. Available: https://arxiv.org/abs/1610.02357
Pith/arXiv arXiv 2017
-
[22]
Xception net & vision transformer: A comparative study for deepfake detection,
D. Shah, D. Shah, D. Jodhawat, J. Parekh, and K. Srivastava, “Xception net & vision transformer: A comparative study for deepfake detection,” in 2022 International Conference on Machine Learning, Computer Systems and Security (MLCSS), 2022, pp. 393–398. DOI: 10.1109/MLCSS57186.2022. 00077
arXiv 2022
-
[23]
Mesonet: A compact facial video forgery detection network,
D. Afchar, V . Nozick, J. Yamagishi, and I. Echizen, “Mesonet: A compact facial video forgery detection network,” in Proceedings of the IEEE International Workshop on Information Forensics and Security (WIFS), Hong Kong, China, Dec. 2018, pp. 1–7. DOI: 10.1109/WIFS.2018.8630761. 20
arXiv 2018
-
[24]
Aasist: Audio anti-spoofing using integrated spectro-temporal graph attention networks,
J. Jung, H.-S. Heo, H. Tak, et al., “Aasist: Audio anti-spoofing using integrated spectro-temporal graph attention networks,” inProceedings of the IEEE International Conference on Acoustics, Speech and Sig- nal Processing (ICASSP), 2022, pp. 2405–2409. DOI: 10.1109/ICASSP43922.2022.9747766
arXiv 2022
-
[25]
End-to-end spectro-temporal graph attention networks for speaker verification anti-spoofing and speech deepfake detection,
H. Tak, J. Jung, J. Patino, M. Kamble, M. Todisco, and N. Evans, “End-to-end spectro-temporal graph attention networks for speaker verification anti-spoofing and speech deepfake detection,” inProceedings of the ASVspoof Challenge, 2021, pp. 1–8
2021
-
[26]
Mtcnn documentation
I. de Paz Centeno. “Mtcnn documentation.” (2021), [Online]. Available: https://mtcnn.readthedocs. io/en/latest/introduction/ (visited on 04/25/2026)
2021
-
[27]
Joint face detection and alignment using multitask cascaded convolutional networks,
K. Zhang, Z. Zhang, Z. Li, and Y . Qiao, “Joint face detection and alignment using multitask cascaded convolutional networks,” IEEE Signal Processing Letters , vol. 23, no. 10, pp. 1499–1503, Oct. 2016. DOI: 10.1109/LSP.2016.2603342
arXiv 2016
-
[28]
The opencv library,
G. Bradski, “The opencv library,” Dr. Dobb’s Journal of Software Tools, vol. 25, no. 11, pp. 120–123, Nov. 2000
2000
-
[29]
Aasist: Audio anti-spoofing using integrated spectro-temporal graph attention networks
J. Jung et al. “Aasist: Audio anti-spoofing using integrated spectro-temporal graph attention networks.” (2021), [Online]. Available: https://github.com/clovaai/aasist (visited on 03/27/2026)
2021
-
[30]
Ensemble methods in machine learning,
T. G. Dietterich, “Ensemble methods in machine learning,” in Proceedings of the International Workshop on Multiple Classifier Systems, 2000, pp. 1–15
2000
-
[31]
Stacked generalization,
D. H. Wolpert, “Stacked generalization,” Neural Networks, vol. 5, no. 2, pp. 241–259, 1992. DOI: 10. 1016/S0893-6080(05)80023-1
1992
- [32]
-
[33]
Faceforensics++: Learning to detect manipulated facial images,
A. R ¨ossler, D. Cozzolino, L. Verdoliva, C. Riess, J. Thies, and M. Nießner, “Faceforensics++: Learning to detect manipulated facial images,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, South Korea, 2019, pp. 1–11. DOI: 10.1109/ICCV.2019.00009
arXiv 2019
-
[34]
Asvspoof 2019: Future horizons in spoofed and fake audio de- tection,
M. Todisco, X. Wang, V . Vestman,et al., “Asvspoof 2019: Future horizons in spoofed and fake audio de- tection,” in Proceedings of the 20th Annual Conference of the International Speech Communication As- sociation (INTERSPEECH), Graz, Austria, 2019, pp. 1008–1012. DOI: 10.21437/Interspeech. 2019-2249
-
[35]
Fakeavceleb: A novel audio-video multimodal deepfake dataset,
H. Khalid, S. Tariq, M. Kim, and S. S. Woo, “Fakeavceleb: A novel audio-video multimodal deepfake dataset,” in Proceedings of the 35th Conference on Neural Information Processing Systems (NeurIPS) , 2021, pp. 1–14. DOI: 10.48550/arXiv.2108.05080
-
[36]
Improving generalization of deepfake detection with domain adap- tive batch normalization,
Z. Yin, J. Wang, Y . Ding, et al., “Improving generalization of deepfake detection with domain adap- tive batch normalization,” in Proceedings of the International Workshop on Adversarial Learning for Multimedia (ADVM), New York, NY , USA, 2021, pp. 21–27.DOI: 10.1145/3475724.3483603. 21
arXiv 2021
-
[2025]
DOI: 10.1155/int/9987535
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.