Pith. sign in

REVIEW 5 major objections 5 minor 15 references

Ensemble-Based Deepfake Detection using State-of-the-Art Models with Robust Cross-Dataset Generalisation

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that combining the probability outputs of six state-of-the-art deepfake detectors yields stable, near-top performance across datasets where no single model stays reliable, making ensembles a safer choice for real-world…

desk verdict A clean, modest empirical note: averaging six published detectors keeps you near the top on two out-of-domain datasets; the 'all scenarios' framing overreaches, but the numbers are useful and reproducible. read the letter →

arxiv 2507.05996 v1 pith:KJXNXK7X submitted 2025-07-08 cs.CV

Haroon Wahab , Hassan Ugail , Lujain Jaleel This is my paper · ORCID
classification cs.CV
keywords ensemblelearningdeepfakedetectioncross-datasetgeneralizationprobabilityaveragingfaceforgeryout-of-distributionevaluationAUROC
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether any single deepfake detector can be trusted when tested on data it was not trained on. It answers no: across two out-of-domain datasets, the six state-of-the-art models reorder themselves. The paper then shows that combining their probability outputs, whether by simple averaging or by skill weighting, keeps performance near the best single model on both datasets, and the ensemble never ranks last. This matters for real deployment, where the forgery type and quality are unknown in advance.

What carries the argument

The core mechanism is late-fusion probability averaging: each model outputs $p_i(x)$, the probability an input is fake, and the ensemble combines them either with uniform weights, $P_{\mathrm{avg}}(x) = \frac{1}{6}\sum_{i=1}^6 p_i(x)$, or with skill weights $w_i$ calibrated on validation performance. The design principle, borrowed from work on asymmetric deep ensembles, is that structurally diverse models make errors that average out. The paper does not measure the diversity directly; it relies on architectural asymmetry as a proxy.

What would settle it

Run the same protocol on a third out-of-distribution dataset such as DFDC. If the ensemble's AUROC falls to or below the median of the six individual models, or if per-sample error correlation among the six is high on forged frames, the claimed stability is an artifact of these two datasets.

Watch

Extended reading notes

Core claim

On models trained with FaceForensics++ and evaluated on UADFV and Celeb-DF-v2, no individual model dominates: Core and FFD lead on UADFV (AUROC 0.961 and 0.950) but fall behind on Celeb-DF-v2 (0.741 and 0.687), while UCF rises to the top there (0.772). The paper's central finding is that probability-level ensembling of all six models yields AUROC 0.958 on UADFV and 0.768/0.769 on Celeb-DF-v2, close to the best individual model in each setting, and never ranks lowest in either. The authors interpret this as evidence that asymmetric deep ensembles are a robust fallback when the data distribution is unknown.

Load-bearing premise

The claim depends on the six models making errors that are sufficiently independent so averaging cancels individual failures; the paper does not measure that diversity or failure correlation.

Editorial extensions

If this is right

  • Both ensemble variants, unweighted and skill-weighted, perform within 0.004 AUROC of the best single model on UADFV and remain competitive on Celeb-DF-v2.
  • Because ensembling never ranks lowest, it offers a safe default choice when prior knowledge of the forgery type is unavailable.
  • The ensemble's stability undercuts any expectation that a single top model, such as Core or UCF, will generalize across domains.
  • The authors note that further gains could come from dynamic ensemble weighting or model selection conditioned on content characteristics.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The robustness claim is not guaranteed beyond these two datasets: if the six models share the same systematic bias on a third distribution, averaging will preserve rather than cancel that bias.
  • A testable extension is to measure pairwise error correlation among the models on out-of-distribution samples; ensemble gain should track disagreement rate.
  • The skill-weighted ensemble was weighted on in-domain validation; with a validation split from the target domain, weighted averaging might outperform simple averaging by a larger margin.
  • For real-time applications, the six-model ensemble carries six-fold inference cost; the results imply a smaller asymmetric subset could be selected while retaining stability.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes an ensemble-based approach to cross-dataset deepfake detection. Six state-of-the-art detectors (MesoInception-4, Xception, CORE, FFD, SRM, UCF), all trained on FaceForensics++, are combined at the probability level using two variants: a simple average and a skill-weighted average. Evaluation is carried out on two out-of-domain datasets, UADFV and Celeb-DF-v2, with AUROC and AUPRC metrics. Table I reports that the two ensemble variants achieve AUROC 0.958 on UADFV and 0.768/0.769 on Celeb-DF-v2, close to the best individual models (0.961 and 0.772, respectively). The paper concludes that no single model is consistently best and that ensembles provide more stable and reliable performance across scenarios.

Significance. If read narrowly, the paper makes a modest but useful empirical observation: an unweighted average of diverse pretrained deepfake detectors is competitive with, and more stable in rank than, the best individual model on two out-of-domain benchmarks. The unweighted ensemble is a parameter-free procedure that can be reproduced from official implementations, which is a strength. However, the broader claim of robustness 'in all scenarios' is not supported by the evidence: only two test domains are used, no confidence intervals or repeated runs are provided, the skill-weighted ensemble's validation set is unspecified, and the diversity of ensemble members is not measured. These limitations directly affect the loading of the paper's central generalization claim.

major comments (5)
  1. [Abstract, §V, Table I] The central claim of 'more stable and reliable performance in all scenarios' rests on exactly two out-of-domain datasets, UADFV and Celeb-DF-v2, with a single evaluation run. The gaps between the ensembles and the best individual model are 0.003 in AUROC on UADFV (0.958 vs. 0.961) and 0.003-0.004 on Celeb-DF-v2 (0.768/0.769 vs. 0.772), which is well within the range that could arise from random variation in face sampling or model initialization. Without confidence intervals, repeated runs, or a significance test, the evidence supports only a statement that the ensembles are competitive with the best model, not that they are more reliable. Additional out-of-domain datasets (e.g., DFDC, WildDeepfake, or a diffusion-generated benchmark) and repeated evaluation runs are needed to support the generalisation claim.
  2. [§III-A, Eq. (1)] The skill-weighted ensemble is defined by weights w_i that are based on 'validation performance,' but the paper never states which validation set was used to compute these weights. If the weights were chosen using UADFV and/or Celeb-DF-v2, the weighted ensemble results in Table I are optimistically biased and partially circular. The authors must specify the exact validation split and the performance metric used to derive w_i, or remove the weighted variant from the main claim. Without this information, the unweighted ensemble is the only interpretable, non-circular result.
  3. [§III, §V] The paper motivates ensembling by citing the benefit of asymmetric deep ensembles, but it never measures the diversity or failure correlation among the six selected models. The robustness argument depends on the assumption that the models' errors are sufficiently decorrelated so that averaging cancels them. If most models share the same systematic bias on a genuinely new domain, the ensemble will inherit that bias. Reporting a diversity metric (e.g., pairwise error correlation, disagreement rate, or complementary failure cases) or adding a third domain that stresses a different type of forgery would directly test this load-bearing assumption.
  4. [§V, Conclusion] The observation that 'the ensembles never ranked the lowest in either dataset' is a weak criterion for robustness. A middle-of-the-pack average can satisfy this condition without being a strong detector, and the criterion says nothing about performance on unseen datasets where the common bias could push the entire ensemble down. The conclusion that ensembles 'are a more reliable choice when facing unknown or shifting data distributions' should be scaled back to the two tested domains unless additional evidence is provided.
  5. [§IV] The experimental setup omits several details needed to assess the result: the number of videos/frames per dataset, the face sampling procedure, whether the same aligned-face preprocessing was applied to all models, and the exact source and version of the pretrained weights from the deepfake benchmark [5]. These details matter because small preprocessing differences can change cross-dataset AUROC by several points, which is comparable to the observed ensemble gains.
minor comments (5)
  1. [Throughout] The model name is inconsistent: Section II-C and Table I use 'CORE' and 'Core' interchangeably. Please use one canonical spelling.
  2. [Section II-A] MesoInception-4 is described as a 'naive detector,' but the meaning of 'naive' in this context is not explained. Clarify whether this refers to the original Mesonet naming or to architectural simplicity.
  3. [Section III-A] Equation (1) defines w_i as skill-based weights, but the paper does not state whether the weights are derived from accuracy, AUROC, or AUPRC on the validation set. Please specify this explicitly.
  4. [Figures 1 and 2] The captions do not indicate the number of points or the uncertainty in the plotted values. Since the underlying table has no error bars, consider adding error bars or at least stating that each value comes from a single evaluation pass.
  5. [References] Reference [12] is a February 2025 arXiv preprint; if possible, cite a peer-reviewed version or additional work on asymmetric deep ensembles to strengthen the motivation.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the ensemble evaluations are fixed averages of independently pretrained model outputs, with no fitted parameter being renamed as a prediction.

full rationale

The paper's central claim is an empirical comparison, not a derivation. The unweighted ensemble is a fixed six-model probability average (Eq. 2) with no free parameters, and the weighted ensemble (Eq. 1) assigns weights from validation performance before evaluation on held-out out-of-domain test sets; the text never states that the validation set is the same as the reported test sets, so the AUROC/AUPRC results cannot be shown to be forced by construction. The six component models and their pretrained weights come from an external open-source benchmark (DeepfakeBench), not from this paper, so the individual predictions are independent inputs rather than outputs of the ensemble claim. The motivation for asymmetric ensembling cites an external study ([12]) and is not a self-citation. The observed near-best ensemble performance on only two datasets is narrow evidence for the phrase 'all scenarios,' but narrowness is an empirical-generalization concern, not circularity. The only related reporting gap is that the validation protocol for the weighted ensemble is not fully specified, so one cannot rule out that weights were tuned on evaluation data; this is an under-reporting issue, not a demonstrated circular step. Because no equation reduces a claimed prediction to its own input and no load-bearing self-citation is present, the circularity score is 0.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The weighted ensemble introduces one set of free parameters, the skill weights, whose values and validation source are not reported. The evaluation relies on the assumption that pretrained DeepfakeBench weights are faithful, that two test datasets represent out-of-domain conditions, and that probability averaging is an appropriate fusion rule. No new entities are postulated.

free parameters (1)
  • Skill weights w_i for weighted ensemble = Not disclosed
    Equation (1) defines the weighted ensemble with weights normalized to sum to one. The paper says weights are based on validation performance (Section III-A) but does not report the weight values or which dataset was used for validation.
assumptions (3)
  • domain assumption Pretrained weights and implementations from DeepfakeBench [5] faithfully reproduce the original models' behavior.
    Section IV relies on 'model implementations and pretrained weights provided by the deepfake benchmark study' without further verification.
  • domain assumption The two test datasets (Celeb-DF-v2 and UADFV) are representative out-of-domain distributions for real-world deepfake detection.
    Section IV selects these two datasets; the conclusion generalizes to 'real-world scenarios' on this basis.
  • domain assumption Probability-level averaging is an appropriate fusion method for these models.
    Section III-B justifies via bias-variance decomposition assuming independent models.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Ensemble-Based Deepfake Detection using State-of-the-Art Models with Robust Cross-Dataset Generalisation." pith.science (2026). https://pith.science/paper/KJXNXK7X

@misc{pith2026250705996,
  author       = {Pith},
  title        = {Pith review of: Ensemble-Based Deepfake Detection using State-of-the-Art Models with Robust Cross-Dataset Generalisation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KJXNXK7X}},
  note         = {Machine review of arXiv:2507.05996}
}
read the original abstract

Machine learning-based Deepfake detection models have achieved impressive results on benchmark datasets, yet their performance often deteriorates significantly when evaluated on out-of-distribution data. In this work, we investigate an ensemble-based approach for improving the generalization of deepfake detection systems across diverse datasets. Building on a recent open-source benchmark, we combine prediction probabilities from several state-of-the-art asymmetric models proposed at top venues. Our experiments span two distinct out-of-domain datasets and demonstrate that no single model consistently outperforms others across settings. In contrast, ensemble-based predictions provide more stable and reliable performance in all scenarios. Our results suggest that asymmetric ensembling offers a robust and scalable solution for real-world deepfake detection where prior knowledge of forgery type or quality is often unavailable.

Figures

Figures reproduced from arXiv: 2507.05996 by the authors.

Figure 1
Figure 1. Comparison of AUROC (left) and AUPRC (right) for individual deepfake detectors and their ensemble combinations, evaluated on UADFV and [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. AUROC-based rank shifts across datasets. A lower position (1) [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 5 canonical work pages

  1. [12]

    The Empirical Impact of Reducing Symmetries on the Performance of Deep Ensembles and MoE

    A. Chernov and O. Novitskij, “The empirical impact of reducing symmetries on the performance of deep ensembles and moe,” arXiv preprint arXiv:2502.17391, 2025

    work page Pith review arXiv 2025

  2. [5]

    Deepfakebench: A comprehensive benchmark of deepfake detection,

    Z. Yan, Y . Zhang, X. Yuan, S. Lyu, and B. Wu, “Deepfakebench: A comprehensive benchmark of deepfake detection,” arXiv preprint arXiv:2307.01426, 2023

    arXiv 2023

  3. [1]

    Deepfakes: Deceptions, mitigations, and opportunities,

    M. Mustak, J. Salminen, M. M ¨antym¨aki, A. Rahman, and Y . K. Dwivedi, “Deepfakes: Deceptions, mitigations, and opportunities,” Journal of Business Research , vol. 154, p. 113368, 2023. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0148296322008335

    work page 2023

  4. [2]

    Unmasking deepfakes: A systematic review of deepfake detection and generation techniques using artificial intelligence,

    F. Abbas and A. Taeihagh, “Unmasking deepfakes: A systematic review of deepfake detection and generation techniques using artificial intelligence,” Expert Systems with Applications, vol. 252, p. 124260, 2024. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0957417424011266

    work page 2024

  5. [3]

    The cat and mouse game: The ongoing arms race between diffusion models and detection methods,

    L. Laurier, A. Giulietta, A. Octavia, and M. Cleti, “The cat and mouse game: The ongoing arms race between diffusion models and detection methods,” arXiv preprint arXiv:2410.18866 , 2024

    arXiv 2024

  6. [4]

    Deepfake generation and detection: A benchmark and survey,

    G. Pei, J. Zhang, M. Hu, Z. Zhang, C. Wang, Y . Wu, G. Zhai, J. Yang, C. Shen, and D. Tao, “Deepfake generation and detection: A benchmark and survey,” arXiv preprint arXiv:2403.17881 , 2024

    arXiv 2024

  7. [6]

    Mesonet: a compact facial video forgery detection network,

    D. Afchar, V . Nozick, J. Yamagishi, and I. Echizen, “Mesonet: a compact facial video forgery detection network,” in 2018 IEEE international workshop on information forensics and security (WIFS) . IEEE, 2018, pp. 1–7

    2018

  8. [7]

    Faceforensics++: Learning to detect manipulated facial images,

    A. Rossler, D. Cozzolino, L. Verdoliva, C. Riess, J. Thies, and M. Nießner, “Faceforensics++: Learning to detect manipulated facial images,” in Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 1–11

    2019

Show all 15 references
  1. [8]

    Core: Consistent representation learning for face forgery detection,

    Y . Ni, D. Meng, C. Yu, C. Quan, D. Ren, and Y . Zhao, “Core: Consistent representation learning for face forgery detection,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 12–21

    2022

  2. [9]

    On the detection of digital face manipulation,

    H. Dang, F. Liu, J. Stehouwer, X. Liu, and A. K. Jain, “On the detection of digital face manipulation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern recognition , 2020, pp. 5781–5790

    2020

  3. [10]

    Generalizing face forgery detec- tion with high-frequency features,

    Y . Luo, Y . Zhang, J. Yan, and W. Liu, “Generalizing face forgery detec- tion with high-frequency features,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2021, pp. 16 317–16 326

    2021

  4. [11]

    Ucf: Uncovering common features for generalizable deepfake detection,

    Z. Yan, Y . Zhang, Y . Fan, and B. Wu, “Ucf: Uncovering common features for generalizable deepfake detection,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 22 412–22 423

    2023

  5. [13]

    Face2face: Real-time face capture and reenactment of rgb videos,

    J. Thies, M. Zollhofer, M. Stamminger, C. Theobalt, and M. Nießner, “Face2face: Real-time face capture and reenactment of rgb videos,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 2387–2395

    2016

  6. [14]

    Deferred neural rendering: Image synthesis using neural textures,

    J. Thies, M. Zollh ¨ofer, and M. Nießner, “Deferred neural rendering: Image synthesis using neural textures,” Acm Transactions on Graphics (TOG), vol. 38, no. 4, pp. 1–12, 2019

    2019

  7. [15]

    Dlib-ml: A machine learning toolkit,

    D. E. King, “Dlib-ml: A machine learning toolkit,” Journal of Machine Learning Research, vol. 10, pp. 1755–1758, 2009

    2009

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.