{"id":"8a4d6a68-b29c-4ebb-a6dc-bf85c1138f2c","arxiv_id":"2505.00312","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A two-tier ensemble that averages three instances of Xception, Res2Net101, and EfficientNet-B7 and learns a softmax-weighted fusion reports near-perfect intra-dataset scores, but its headline cross-dataset AUC of 88.20% is mathematically inconsistent with its own method.","lead":"AWARE-NET combines three copies each of three standard deep networks and learns a weighted average of their outputs for deepfake detection. The paper reports near-perfect scores on two benchmark datasets and a huge cross-dataset jump that is likely an artifact of its own evaluation.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 3's reported 88.20% cross-dataset AUC exceeds a provable upper bound (~79.3%) for any positive-weight ensemble of the listed below-chance base models, so the central generalization claim is internally inconsistent as reported.","rationale":"The reader's rejection is supported by a decisive quantitative check. The method in Eq. 7 is a convex combination of base sigmoid outputs with strictly positive softmax weights; therefore, whenever all three base models rank a fake below a real on a given pair, the ensemble must also rank it below. The base-model AUCs in Table 3 imply that the probability all three are wrong is at least 1 − (0.2437 + 0.2461 + 0.3031) = 0.2071, so the ensemble AUC is bounded above by 79.29%. The reported 88.20% exceeds this bound by 8.91 points. This is not a matter of missing baselines or unstated hyperparameters; the headline cross-dataset result cannot be produced by the described pipeline unless evaluation conditions differ across rows. The paper also lacks code, ablations, error bars, and consistent table captions, but the bound alone is sufficient to reject the central claim. I agree with the reader's verdict, and the specific upper-bound argument strengthens the weakest assumption from a general concern about convex combinations into a precise, falsifiable inconsistency.","tokens_in":12059,"tokens_out":9894,"duration_ms":108818,"concrete_test":"Re-run the frozen FF++-trained Xception, Res2Net101, and EfficientNetB7 models on the exact CelebDF-v2 test videos used for Table 3, keeping one label convention. Record the per-pair differences D_i and compute y = Σ α_i p_i with the learned weights. If the three individual AUCs reproduce as 24–30%, then y's AUC cannot exceed their sum (79.29%); check whether 88.20% is reproducible or whether it arises only under a different test split or flipped label orientation.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim in Section 5.2 is not merely surprising; it is inconsistent with Eq. 7 and Table 3 on the same test set. For FF++ to CelebDF-v2, the individual AUCs are 24.37%, 24.61%, and 30.31%. For any real/fake pair, let D_i = p_i(fake) − p_i(real). Then AUC_i = P(D_i > 0). Because the final score is y = Σ α_i p_i with α_i > 0 and Σ α_i = 1, if every D_i ≤ 0 then y(fake) − y(real) ≤ 0, so the ensemble cannot rank that fake above that real. Hence AUC_ens ≤ P(∪_i {D_i > 0}) ≤ Σ_i AUC_i = 79.29%. The reported 88.20% is therefore impossible for any fixed positive weights evaluated with the same label orientation and test set. The only resolutions are that the base-model AUCs, the ensemble AUC, or the columns are evaluated under different conventions, or that Eq. 7 is not what produced the table. The conclusion's own parenthetical ('CelebDF-v2 to FF++ transfer') further mislabels the 57.89-point gap, reinforcing the evaluation-inconsistency concern.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes AWARE-NET, a two-tier ensemble for deepfake video detection. Three instances each of Xception, Res2Net101, and EfficientNet-B7 are trained; their sigmoid outputs are averaged within each architecture family in the first tier, and a softmax-normalized learnable weight vector combines the three family-level scores in the second tier. Experiments on FaceForensics++ and CelebDF-v2 report near-saturated intra-dataset AUCs of 99.22% and 100.00%, and cross-dataset AUCs of 88.20% and 72.52%. The paper's central claim is that the learned-weight two-tier fusion substantially improves cross-dataset generalization, especially in the FF++ to CelebDF-v2 direction.","tokens_in":12353,"tokens_out":5778,"duration_ms":57791,"significance":"If the cross-dataset numbers were correct, the result would be remarkable, since it would imply that a positive-weight convex combination of three below-chance base detectors (AUCs 24.37%, 24.61%, 30.31%) becomes a highly accurate 88.20% AUC detector. That implication is impossible under the paper's own Eq. (7), as the stress-test upper bound shows. The intra-dataset results are plausible but near saturation and add little over the best base model, so the cross-dataset generalization claim is the main contribution of the paper, and that claim is internally inconsistent. On the positive side, the framework is clearly described, the intra-dataset evaluation uses standard datasets and metrics, and the inconsistency is precisely diagnosable from the provided equations and tables, which gives the authors a concrete path to locate the evaluation error. No code or model weights are released, so the discrepancy cannot currently be resolved by artifact inspection.","major_comments":[{"comment":"The reported cross-dataset AUC of 88.20% for FF++ to CelebDF-v2 is impossible under Eq. (7) given the individual AUCs in Table 3. Eq. (7) defines y(x) = sum_i alpha_i p_i(x) with alpha_i > 0 and sum_i alpha_i = 1. For any fake/real pair, define D_i = p_i(fake) - p_i(real); the ensemble ranks the pair correctly only if sum_i alpha_i D_i > 0. This cannot happen when all three D_i <= 0, so the ensemble can correctly rank a pair only if at least one base model does. Hence AUC_ens <= P(union_i {D_i > 0}) <= sum_i AUC_i = 24.37% + 24.61% + 30.31% = 79.29%, which is strictly less than 88.20%. The reported ensemble AUC therefore requires either a different label orientation, a different test set, or a different fusion formula than Eq. (7) for the AWARE-NET row. This is a load-bearing inconsistency in the central generalization claim.","section":"Section 5.2, Eq. (7), Table 3"},{"comment":"The augmentation results in Table 4 are mutually inconsistent with Table 3 and with the text. Table 4 reports an AWARE-NET AUC of 22.43% for FF++ to CelebDF-v2 with augmentation, whereas Table 3 reports 88.20% for the same direction without augmentation; this 65.77-point drop is not a plausible augmentation effect. The surrounding text adds further confusion by stating 'FF++ to CelebDF-v2: 69.66% vs. 72.52% without augmentation,' although the 72.52% value in Table 3 appears in the column that the text elsewhere associates with CelebDF-v2 to FF++. The column labels in Tables 3 and 4 must be reconciled with the transfer directions stated in the text before any cross-dataset conclusion can be evaluated.","section":"Section 5.2, Tables 3 and 4"},{"comment":"The concluding paragraph attributes the '57.89% improvement' to the CelebDF-v2 to FF++ transfer. From Table 3, 57.89 = 88.20 - 30.31 belongs to the FF++ to CelebDF-v2 direction, while the CelebDF-v2 to FF++ improvement over the best base model is 72.52 - 71.76 = 0.76%. This mislabeled direction is consistent with the column-orientation problem in Tables 3 and 4 and should be corrected in any revision.","section":"Section 6"}],"minor_comments":[{"comment":"The contribution list claims that the framework 'jointly optimizes model parameters and architectural weights,' but the implementation section states that Phase 2 freezes the pre-trained base models and trains only the weight vector w; the wording should be aligned with the actual training procedure.","section":"Section 4"},{"comment":"The paper does not state whether AUC and F1 are computed at frame level or video level, nor how many test videos or frames are used in the cross-dataset evaluation; this information is needed to interpret the absolute numbers.","section":"Section 5.2"},{"comment":"There are typographical errors, such as 'pochs' for 'epochs' in Section 4, and Figure 9's axes and color key are not self-explanatory.","section":"Section 4"},{"comment":"References [29] and [30] appear to duplicate the same survey, and reference [35] is a GitHub repository link rather than a formal citation; the reference formatting should be normalized.","section":"References"}],"recommendation":"reject","confidential_remarks":"The paper's main selling point is the cross-dataset generalization result, and that result fails an internal consistency check under the paper's own equations. Because the reported 88.20% AUC exceeds a provable upper bound for any positive-weight ensemble of the three listed base models, the central claim cannot be repaired by additional analysis within the current experimental claims; the authors would need to rerun the cross-dataset evaluation and likely find that the contribution reduces to a marginal intra-dataset gain. I see no ethical concerns beyond the consistency issue, but I recommend rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: the headline cross-dataset result is not just surprising, it is inconsistent with the paper's own method. Table 3 reports AWARE-NET at 88.20% AUC on FF++ to CelebDF-v2 while every base model scores below 31% AUC on that same direction. Since Eq. 7 forms the ensemble output as a convex combination (positive softmax weights) of the base sigmoid outputs, the ensemble score is a weighted average of the base scores. If each base model ranks a fake below a real, the ensemble must do the same. The ensemble AUC therefore cannot exceed the sum of the base AUCs, which here is about 79%, and with all bases below chance it would actually be far lower. The only ways to get 88.20% are different label orientation, a different test set, or a different equation than the one reported. The paper does not acknowledge any of these. This kills the central generalization claim.\n\nWhat the paper does well: the two-tier structure—averaging three instances within each architecture family, then fusing with learned softmax weights—is a reasonable recombination of standard ensemble ideas. The intra-dataset tables look plausible: near-saturated AUCs on FF++ and CelebDF-v2 with small but consistent gains from the ensemble. The method section is straightforwardly written.\n\nSoft spots beyond the fatal one: no code, no error bars, no ablations on the number of instances, and no comparison against the cited ensemble baselines. The augmentation cross-dataset text also mislabels the direction: the 69.66% figure it attributes to FF++ to CelebDF-v2 is actually the reverse transfer, and the actual FF++ to CelebDF-v2 number with augmentation is 22.43%. That kind of sloppiness reinforces the concern that the 88.20% comes from an evaluation inconsistency rather than a real effect. The learned weights are three scalars fit on validation data, which is standard supervised learning, not circular, but also not the dramatic mechanism the paper claims.\n\nWho is this for? Not many people, in its current form. If the authors can release code and produce a single consistent evaluation protocol, the intra-dataset ensemble idea might be a minor contribution. As submitted, the main result is arithmetically impossible, and the paper should not consume referee time until that is resolved.","headline":"The cross-dataset AUC of 88.20% is arithmetically impossible given Table 3 and Eq. 7; the paper's only plausible result is a small intra-dataset gain near saturation.","tokens_in":743,"tokens_out":771,"would_cite":false,"duration_ms":35610,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AWARE-NET claims a two-tier ensemble with learnable weights lifts cross-dataset deepfake detection from 30.31% to 88.20% AUC.","keywords":["deepfake detection","ensemble learning","learnable weights","cross-dataset generalization","FaceForensics++","CelebDF-v2","two-tier fusion","Xception"],"falsifier":"Recompute the ensemble's score on the CelebDF-v2 test set as the weighted average of the three architecture-family scores using the learned weights, and verify that the resulting AUC is approximately 88.20% while each family's individual AUC is below 31%. If no positive weighting of those scores yields an AUC above 50%, the reported jump is an artifact of inconsistent evaluation rather than a property of the ensemble.","tokens_in":11879,"feed_emoji":"🎭","tokens_out":7294,"duration_ms":65265,"temperature":0.7,"pith_summary":"The paper introduces AWARE-NET, a two-tier ensemble for deepfake detection that combines three instances each of Xception, Res2Net101, and EfficientNet-B7: the first tier averages predictions within each architecture family, and the second tier learns a softmax-weighted combination of the three family outputs through backpropagation. The authors report near-perfect intra-dataset results (99.22% AUC on FaceForensics++, 100% on CelebDF-v2) and, as the central finding, a cross-dataset AUC of 88.20% when models trained on FaceForensics++ are tested on CelebDF-v2, where the best individual architecture reaches only 30.31%. If correct, this shows that a learnably weighted ensemble can convert individually failing detectors into a strong generalizing one. The framework is fully differentiable and yields interpretable per-architecture weights.","feed_headline":"Ensemble lifts deepfake AUC from 30% to 88% across datasets","feed_subtitle":"Nine CNN instances plus learned weights beat every single network on the FaceForensics++ to CelebDF-v2 transfer.","key_machinery":"The central object is the two-tier fusion rule. First-tier: for each architecture family $A \\in \\{\\text{Xception}, \\text{Res2Net101}, \\text{EfficientNet-B7}\\}$, three independently initialized instances produce sigmoid outputs $\\sigma(M_i^A(x))$ whose mean is $p_A(x)$ (Eq. 5). Second-tier: a learnable parameter vector $w=(w_1,w_2,w_3)$ is passed through softmax to give $\\alpha=\\mathrm{softmax}(w)$, and the final prediction is $y(x)=\\sum_i \\alpha_i p_i(x)$ (Eq. 7). The machinery is fully differentiable: the gradient of the loss with respect to $w_i$ (Eq. 8) flows through $\\partial y/\\partial \\alpha_i = p_i(x)$ and the softmax Jacobian, so the ensemble discovers which architecture to trust during training.","core_discovery":"The paper's central claim is that a two-tier ensemble can turn individually poor cross-dataset detectors into a strong one. Trained on FaceForensics++ and tested on CelebDF-v2, AWARE-NET reports an AUC of 88.20% and an F1 score of 93.16%, while every individual architecture — Xception, Res2Net101, and EfficientNet-B7 — achieves an AUC below 31%, i.e., below chance. The authors attribute this to diversity across the three families and to the adaptive weighting mechanism, which learns to emphasize reliable architectures through backpropagation. On the reverse transfer the advantage is smaller but positive (72.52% AUC versus 71.76% for the best single model), and on intra-dataset tests the ensemble reaches 99.22% AUC on FF++ and 100% on CelebDF-v2.","pith_inferences":["The 57.89-point AUC jump from below-chance individuals to an 88% ensemble is extraordinary for a convex combination with fixed positive weights; a straightforward check is whether the base-model AUCs in Table 3 share the same label orientation as the ensemble row.","If the result reproduces under a consistent evaluation protocol, it would suggest that model diversity alone can act as a form of unsupervised domain adaptation, with the learned weights effectively suppressing unreliable architectures on the source domain.","A natural extension is to record the learned weights $\\alpha$ on source data and inspect whether they shift when a small labeled target-domain sample is introduced, turning the frozen ensemble into a few-shot domain-adaptation tool."],"forward_implications":["AWARE-NET reaches essentially ceiling performance on intra-dataset tests: 99.22% AUC on FF++ and 100% on CelebDF-v2 without augmentation, with F1 scores above 98% in both cases.","On the FF++ to CelebDF-v2 transfer, the ensemble reports 88.20% AUC versus 30.31% for the best individual network, implying that the learned combination exploits failure complementarity.","On the reverse transfer (CelebDF-v2 to FF++), the ensemble improves AUC to 72.52% from the best single model's 71.76%, a positive but much smaller gain.","Data augmentation helps intra-dataset performance but slightly hurts cross-dataset transfer (69.66% vs 72.52% AUC in one direction), indicating that source-domain augmentation strategies do not transfer."],"supporting_citations":[{"why":"Supplies the Xception backbone architecture used as one of the three ensemble families.","marker":"[26]"},{"why":"Supplies the Res2Net101 multi-scale backbone architecture used in the ensemble.","marker":"[27]"},{"why":"Supplies the EfficientNet-B7 backbone architecture used in the ensemble.","marker":"[28]"},{"why":"Defines the CelebDF-v2 dataset used for cross-dataset testing.","marker":"[31]"},{"why":"Defines the FaceForensics++ dataset used for training and intra-dataset evaluation.","marker":"[32]"},{"why":"DeepfakeStack is a deep ensemble baseline that the proposed two-tier approach extends and compares against.","marker":"[23]"},{"why":"Provides an ensemble-of-ViT-and-CNN baseline that motivates the architecture-diversity design.","marker":"[24]"}],"fun_headline_variants":["Adaptive ensemble takes deepfake AUC from 31% to 88%","Two-tier ensemble learns to weight models, far beats best single model","From below-chance to 88% AUC: AWARE-NET's learned ensemble","Deepfake ensemble with learned weights hits 88% AUC cross-dataset"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim stands on the assumption that the base-model AUCs and the ensemble AUC in Table 3 come from the same test set with identical label orientation; otherwise the convex combination in Eq. 7 could not turn several below-chance scores into an 88% AUC.","fun_headline_variants_meta":{"raw":{"variants":["Adaptive ensemble takes deepfake AUC from 31% to 88%","Two-tier ensemble learns to weight models, far beats best single model","From below-chance to 88% AUC: AWARE-NET's learned ensemble","Deepfake ensemble with learned weights hits 88% AUC cross-dataset"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000732,"raw_usage":{"total_tokens":3326,"prompt_tokens":1045,"completion_tokens":2281,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":661,"completion_tokens_details":{"reasoning_tokens":2199}},"tokens_in":661,"tokens_out":2281,"duration_ms":15315,"temperature":1.0,"reasoning_tokens":2199,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:46:09.920111+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the ensemble's score on the CelebDF-v2 test set as the weighted average of the three architecture-family scores using the learned weights, and verify that the resulting AUC is approximately 88.20% while each family's individual AUC is below 31%. If no positive weighting of those scores yields an AUC above 50%, the reported jump is an artifact of inconsistent evaluation rather than a property of the ensemble.","supporting_citations":[],"review_version":1}