{"id":"e8fc52c1-6149-413d-b60a-3244ee131f58","arxiv_id":"2507.05996","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Averaging the outputs of six pretrained deepfake detectors achieves stable near-best accuracy on two out-of-domain face forgery datasets.","lead":"This paper tests whether averaging the predictions of six existing deepfake detectors improves performance on unseen datasets. It finds that the ensembles track the best single model on two out-of-domain benchmarks and never rank last.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central robustness claim rests on only two datasets; a third domain could break the ensemble's near-best performance.","rationale":"The reader's weakest_assumption focuses on the lack of measured model diversity, which is a plausible mechanism for ensemble failure. My concern is closely related but more directly empirical: the paper's evidence base consists of only two out-of-domain datasets, and the central claim explicitly generalizes to 'all scenarios.' The near-best ensemble performance on two datasets, without error bars or significance tests, does not establish that the ensemble will remain robust on a third domain with a different shift. The diversity concern is one reason this could fail, but the broader issue is that the paper does not provide enough test conditions to justify the claim. The reader's rationale already mentions the limited datasets and lack of error bars, so we partially agree. My proposed concrete test directly addresses this by adding two more diverse datasets; if the ensemble remains consistently near the best, the concern is resolved. Since the reader's verdict is already CONDITIONAL and this concern reinforces that condition, no verdict change is needed.","tokens_in":6185,"tokens_out":2811,"duration_ms":31407,"concrete_test":"Re-run the exact same evaluation pipeline on at least two additional out-of-domain datasets with different manipulation types and compression levels, e.g., DFDC and a diffusion-generated deepfake dataset, using the same six pretrained models and both ensemble variants. Check whether each ensemble stays within 0.01 AUROC of the best individual model and never ranks lowest on these new datasets. If the ensemble ranks lowest or falls well below the best model on any new dataset, the 'all scenarios' claim is falsified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim that ensembles are 'more stable and reliable performance in all scenarios' is supported by exactly two out-of-domain datasets, UADFV and Celeb-DF-v2. Table I shows the ensemble variants actually trail the best single model on both datasets (0.958 vs 0.961 on UADFV; 0.768/0.769 vs 0.772 on Celeb-DF-v2), with no error bars or significance testing. The 'never ranked lowest' observation is a weak criterion, since a middle-of-the-pack average can satisfy it without demonstrating robustness. More importantly, Section III motivates ensembling by citing the benefit of asymmetric diversity but never measures diversity or failure correlation among the six models. If all models share the same systematic bias on a genuinely new domain, the ensemble will inherit that bias, and the near-best performance seen here may not generalize. Thus the central claim is empirically underdetermined: the evidence base is too narrow to support a claim about 'all scenarios.'","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an ensemble-based approach to cross-dataset deepfake detection. Six state-of-the-art detectors (MesoInception-4, Xception, CORE, FFD, SRM, UCF), all trained on FaceForensics++, are combined at the probability level using two variants: a simple average and a skill-weighted average. Evaluation is carried out on two out-of-domain datasets, UADFV and Celeb-DF-v2, with AUROC and AUPRC metrics. Table I reports that the two ensemble variants achieve AUROC 0.958 on UADFV and 0.768/0.769 on Celeb-DF-v2, close to the best individual models (0.961 and 0.772, respectively). The paper concludes that no single model is consistently best and that ensembles provide more stable and reliable performance across scenarios.","tokens_in":6421,"tokens_out":4221,"duration_ms":50631,"significance":"If read narrowly, the paper makes a modest but useful empirical observation: an unweighted average of diverse pretrained deepfake detectors is competitive with, and more stable in rank than, the best individual model on two out-of-domain benchmarks. The unweighted ensemble is a parameter-free procedure that can be reproduced from official implementations, which is a strength. However, the broader claim of robustness 'in all scenarios' is not supported by the evidence: only two test domains are used, no confidence intervals or repeated runs are provided, the skill-weighted ensemble's validation set is unspecified, and the diversity of ensemble members is not measured. These limitations directly affect the loading of the paper's central generalization claim.","major_comments":[{"comment":"The central claim of 'more stable and reliable performance in all scenarios' rests on exactly two out-of-domain datasets, UADFV and Celeb-DF-v2, with a single evaluation run. The gaps between the ensembles and the best individual model are 0.003 in AUROC on UADFV (0.958 vs. 0.961) and 0.003-0.004 on Celeb-DF-v2 (0.768/0.769 vs. 0.772), which is well within the range that could arise from random variation in face sampling or model initialization. Without confidence intervals, repeated runs, or a significance test, the evidence supports only a statement that the ensembles are competitive with the best model, not that they are more reliable. Additional out-of-domain datasets (e.g., DFDC, WildDeepfake, or a diffusion-generated benchmark) and repeated evaluation runs are needed to support the generalisation claim.","section":"Abstract, §V, Table I"},{"comment":"The skill-weighted ensemble is defined by weights w_i that are based on 'validation performance,' but the paper never states which validation set was used to compute these weights. If the weights were chosen using UADFV and/or Celeb-DF-v2, the weighted ensemble results in Table I are optimistically biased and partially circular. The authors must specify the exact validation split and the performance metric used to derive w_i, or remove the weighted variant from the main claim. Without this information, the unweighted ensemble is the only interpretable, non-circular result.","section":"§III-A, Eq. (1)"},{"comment":"The paper motivates ensembling by citing the benefit of asymmetric deep ensembles, but it never measures the diversity or failure correlation among the six selected models. The robustness argument depends on the assumption that the models' errors are sufficiently decorrelated so that averaging cancels them. If most models share the same systematic bias on a genuinely new domain, the ensemble will inherit that bias. Reporting a diversity metric (e.g., pairwise error correlation, disagreement rate, or complementary failure cases) or adding a third domain that stresses a different type of forgery would directly test this load-bearing assumption.","section":"§III, §V"},{"comment":"The observation that 'the ensembles never ranked the lowest in either dataset' is a weak criterion for robustness. A middle-of-the-pack average can satisfy this condition without being a strong detector, and the criterion says nothing about performance on unseen datasets where the common bias could push the entire ensemble down. The conclusion that ensembles 'are a more reliable choice when facing unknown or shifting data distributions' should be scaled back to the two tested domains unless additional evidence is provided.","section":"§V, Conclusion"},{"comment":"The experimental setup omits several details needed to assess the result: the number of videos/frames per dataset, the face sampling procedure, whether the same aligned-face preprocessing was applied to all models, and the exact source and version of the pretrained weights from the deepfake benchmark [5]. These details matter because small preprocessing differences can change cross-dataset AUROC by several points, which is comparable to the observed ensemble gains.","section":"§IV"}],"minor_comments":[{"comment":"The model name is inconsistent: Section II-C and Table I use 'CORE' and 'Core' interchangeably. Please use one canonical spelling.","section":"Throughout"},{"comment":"MesoInception-4 is described as a 'naive detector,' but the meaning of 'naive' in this context is not explained. Clarify whether this refers to the original Mesonet naming or to architectural simplicity.","section":"Section II-A"},{"comment":"Equation (1) defines w_i as skill-based weights, but the paper does not state whether the weights are derived from accuracy, AUROC, or AUPRC on the validation set. Please specify this explicitly.","section":"Section III-A"},{"comment":"The captions do not indicate the number of points or the uncertainty in the plotted values. Since the underlying table has no error bars, consider adding error bars or at least stating that each value comes from a single evaluation pass.","section":"Figures 1 and 2"},{"comment":"Reference [12] is a February 2025 arXiv preprint; if possible, cite a peer-reviewed version or additional work on asymmetric deep ensembles to strengthen the motivation.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"This is a short empirical note whose core observation—that an unweighted ensemble of diverse pretrained detectors is competitive with the best single model on two out-of-domain tests—is plausible and easily reproducible. The main problem is the gap between the evidence and the 'all scenarios' claim, which can be fixed by narrowing the claims and supplying the missing validation details and uncertainty quantification. Given the simplicity of the method, the novelty is incremental, but the paper may still be a useful contribution if revised along the lines above."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper does one thing and does it honestly. It takes six published detectors, averages their probabilities, and shows the average stays near the top on two unseen datasets. That is a useful data point, not a breakthrough. The paper mostly says so, except the abstract's \"all scenarios\" overreaches.\n\nWhat's new: new evaluation numbers on Celeb-DF-v2 and UADFV using official DeepfakeBench weights. The fusion equations are standard, and the paper correctly cites [12] for asymmetric ensembles. The bump chart and rank-shift analysis are a nice touch. I trust the reported table; there's no sign of cherry-picking, and the individual model numbers are in line with the literature.\n\nSoft spots, in order: (1) The evidence base is two datasets. That is enough to show a trend, not enough for \"all scenarios.\" A third out-of-domain set would have made the claim proportional. (2) The weighted ensemble's validation set is never described. If it was the test set, the weighted numbers are partially circular; if it was a held-out split, say so. (3) No error bars or repeated runs. For a small table like this, I'd want standard deviations across runs. (4) The ensembles actually trail the best single model on both datasets (0.958 vs 0.961 on UADFV; 0.768/0.769 vs 0.772 on Celeb-DF-v2). So \"superior\" in the conclusion is unsupported; \"near-best\" is accurate. (5) The paper motivates diversity by citing asymmetric ensembles but never measures diversity or failure correlation. That's a gap, but a minor one; the empirical result stands on its own.\n\nThe stress-test note is right: the central claim is underdetermined. But I would not call it a load-bearing flaw. The paper's actual contribution is the observation that simple averaging is a safe default in exactly the two scenarios tested. That is reproducible and useful for practitioners.\n\nI would send this to peer review. It is a short, honest empirical note; a good referee will ask for a third dataset and disclosure of the validation set, and the authors can probably supply both. It is not a desk reject, and it is not a big deal either. Worth engaging with if you work on deepfake detection or model ensembling.","headline":"A clean, modest empirical note: averaging six published detectors keeps you near the top on two out-of-domain datasets; the 'all scenarios' framing overreaches, but the numbers are useful and reproducible.","tokens_in":6851,"tokens_out":2110,"would_cite":true,"duration_ms":22893,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that combining the probability outputs of six state-of-the-art deepfake detectors yields stable, near-top performance across datasets where no single model stays reliable, making ensembles a safer choice for real-world…","keywords":["ensemble learning","deepfake detection","cross-dataset generalization","probability averaging","face forgery","out-of-distribution evaluation","AUROC"],"falsifier":"Run the same protocol on a third out-of-distribution dataset such as DFDC. If the ensemble's AUROC falls to or below the median of the six individual models, or if per-sample error correlation among the six is high on forged frames, the claimed stability is an artifact of these two datasets.","tokens_in":5945,"feed_emoji":"🎭","tokens_out":3588,"duration_ms":37813,"temperature":0.7,"pith_summary":"The paper asks whether any single deepfake detector can be trusted when tested on data it was not trained on. It answers no: across two out-of-domain datasets, the six state-of-the-art models reorder themselves. The paper then shows that combining their probability outputs, whether by simple averaging or by skill weighting, keeps performance near the best single model on both datasets, and the ensemble never ranks last. This matters for real deployment, where the forgery type and quality are unknown in advance.","feed_headline":"Six-model ensemble keeps deepfake detection stable across datasets","feed_subtitle":"In two out-of-domain tests, no individual model dominates and the ensemble never ranks last.","key_machinery":"The core mechanism is late-fusion probability averaging: each model outputs $p_i(x)$, the probability an input is fake, and the ensemble combines them either with uniform weights, $P_{\\mathrm{avg}}(x) = \\frac{1}{6}\\sum_{i=1}^6 p_i(x)$, or with skill weights $w_i$ calibrated on validation performance. The design principle, borrowed from work on asymmetric deep ensembles, is that structurally diverse models make errors that average out. The paper does not measure the diversity directly; it relies on architectural asymmetry as a proxy.","core_discovery":"On models trained with FaceForensics++ and evaluated on UADFV and Celeb-DF-v2, no individual model dominates: Core and FFD lead on UADFV (AUROC 0.961 and 0.950) but fall behind on Celeb-DF-v2 (0.741 and 0.687), while UCF rises to the top there (0.772). The paper's central finding is that probability-level ensembling of all six models yields AUROC 0.958 on UADFV and 0.768/0.769 on Celeb-DF-v2, close to the best individual model in each setting, and never ranks lowest in either. The authors interpret this as evidence that asymmetric deep ensembles are a robust fallback when the data distribution is unknown.","pith_inferences":["The robustness claim is not guaranteed beyond these two datasets: if the six models share the same systematic bias on a third distribution, averaging will preserve rather than cancel that bias.","A testable extension is to measure pairwise error correlation among the models on out-of-distribution samples; ensemble gain should track disagreement rate.","The skill-weighted ensemble was weighted on in-domain validation; with a validation split from the target domain, weighted averaging might outperform simple averaging by a larger margin.","For real-time applications, the six-model ensemble carries six-fold inference cost; the results imply a smaller asymmetric subset could be selected while retaining stability."],"forward_implications":["Both ensemble variants, unweighted and skill-weighted, perform within 0.004 AUROC of the best single model on UADFV and remain competitive on Celeb-DF-v2.","Because ensembling never ranks lowest, it offers a safe default choice when prior knowledge of the forgery type is unavailable.","The ensemble's stability undercuts any expectation that a single top model, such as Core or UCF, will generalize across domains.","The authors note that further gains could come from dynamic ensemble weighting or model selection conditioned on content characteristics."],"supporting_citations":[{"why":"Supplies the six model implementations and pretrained weights from the DeepfakeBench suite that the experiments run on.","marker":"[5]"},{"why":"Provides the empirical basis for expecting asymmetric deep ensembles to improve with size, motivating the ensemble design.","marker":"[12]"},{"why":"Defines FaceForensics++, the training dataset whose manipulations the models are fitted on.","marker":"[7]"}],"fun_headline_variants":["Ensemble of six deepfake detectors stays reliable on new data","Six-model blend tops deepfake detection across unseen datasets","No single model wins, so this six-model ensemble never loses","Probability ensemble keeps deepfake detection steady out-of-domain","Robust deepfake detection: six models beat any single one"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim depends on the six models making errors that are sufficiently independent so averaging cancels individual failures; the paper does not measure that diversity or failure correlation.","fun_headline_variants_meta":{"raw":{"variants":["Ensemble of six deepfake detectors stays reliable on new data","Six-model blend tops deepfake detection across unseen datasets","No single model wins, so this six-model ensemble never loses","Probability ensemble keeps deepfake detection steady out-of-domain","Robust deepfake detection: six models beat any single one"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000681,"raw_usage":{"total_tokens":3031,"prompt_tokens":824,"completion_tokens":2207,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":440,"completion_tokens_details":{"reasoning_tokens":2125}},"tokens_in":440,"tokens_out":2207,"duration_ms":15748,"temperature":1.0,"reasoning_tokens":2125,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:12:36.708599+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same protocol on a third out-of-distribution dataset such as DFDC. If the ensemble's AUROC falls to or below the median of the six individual models, or if per-sample error correlation among the six is high on forged frames, the claimed stability is an artifact of these two datasets.","supporting_citations":[{"cited_title":"The Empirical Impact of Reducing Symmetries on the Performance of Deep Ensembles and MoE","cited_arxiv_id":"2502.17391","evidence_quote":"Provides the empirical basis for expecting asymmetric deep ensembles to improve with size, motivating the ensemble design."}],"review_version":1}