{"id":"8e810cd9-f2d2-4315-837e-490219592513","arxiv_id":"2501.16704","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"An ensemble of three pretrained vision transformers trained with supervised contrastive loss and majority voting reports 95.83% validation accuracy on the DFWild-Cup 2025 deepfake detection dataset.","lead":"A student team built a deepfake detector by combining three large pretrained image models trained with a contrastive loss that separates real and fake faces, then combining their outputs by majority vote. The system reports 95.83% validation accuracy on the IEEE SP Cup 2025 deepfake detection benchmark, beating standard backbones by about 8 points.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 95.83% accuracy is measured on the same validation set used for backbone selection, learning-rate scheduling, and ensemble choice, so it is not an unbiased estimate of generalization to unseen data.","rationale":"The reader's weakest assumption correctly identifies the load-bearing issue: the validation set is used for both model selection and final reporting, so the headline accuracy is positively biased as an estimate of generalization. My read does not change the conditional verdict. The paper is a competition report with no code or data release, and the robustness claims in Sections III-D and IV go beyond what a single validation set can support. The numerical result appears internally consistent, and the ablation study is a useful sanity check, but the central generalization claim remains unverified without an independent test set or a properly held-out partition.","tokens_in":9439,"tokens_out":4001,"duration_ms":41745,"concrete_test":"Before any training or model selection, randomly hold out 20% (about 614 images) of the DFWild validation set as an untouched test set. Run the full pipeline—secondary fake generation, augmentation, SupCon backbone training, classifier training, and majority-voting ensemble—using only the remaining 80% for validation-based scheduling and backbone/ensemble selection. Report accuracy, AUC, and F1 on the held-out 20% with 95% confidence intervals across at least three random seeds. If the held-out accuracy is not significantly above the best single backbone, or drops materially below 95.83%, the claimed ensemble and generalization advantage is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—95.83% validation accuracy and the resulting claim of robustness/generalization—rests on the DFWild validation set being a fair estimate of unseen performance. The paper does not provide such an estimate. In Section III-B the authors evaluated advanced backbones on the validation set and selected MaxViT, CoAtNet, and EVA-02 based on accuracy/AUC; in Section III-C the same validation set is used to report the ensemble result; and in Section II-E ReduceLROnPlateau adjusts the learning rate using validation loss. Thus the validation set is used for model selection, hyperparameter scheduling, and final reporting. With five candidate architectures and only 3,072 validation images, selection on the validation set inflates the reported accuracy relative to a truly held-out test (winner's curse), and the gap to the ImageNet baselines may be overstated. Section III-D supports generalization and fairness claims with 'we expect' rather than with measurements, and no independent test set, confidence intervals, or repeated-seed results are reported. Even if the 95.83% number is internally consistent, it only supports a descriptive statement about this validation partition, not the abstract's and conclusion's broader claims about 'unseen scenarios' and 'real-world conditions.'","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper, a technical report for the IEEE SP Cup 2025, proposes DFCon, a deepfake detection system that ensembles three pretrained vision backbones (MaxViT, CoAtNet, and EVA-02). The method first fine-tunes each backbone with supervised contrastive loss on a balanced subset of the DFWild training data, then freezes the backbones and trains MLP classifiers with BCE loss, and finally combines the three classifiers by majority voting. On the DFWild validation set, the ensemble is reported to achieve 95.83% accuracy, 0.9586 F1 score, and 0.9807 AUC, outperforming the individual backbones and several ResNet/Inception baselines. The paper also includes an ablation study on CoAtNet and t-SNE visualizations of the learned embeddings.","tokens_in":9695,"tokens_out":5569,"duration_ms":50984,"significance":"The engineering recipe is sensible and the paper is transparent about many training details, including model variants, hyperparameters, per-epoch training times, and an ablation study. If the reported validation accuracy were confirmed on an independent test set, the system would be a competitive solution for the DFWild-Cup. However, as presented, the evidence supports only a descriptive statement about this particular validation partition. The absence of a held-out test set, the use of the validation set for model selection and learning-rate scheduling, and the lack of uncertainty estimates mean that the paper's broader generalization and robustness claims are not yet substantiated. The strengths of the paper are its clear pipeline description, the comparison of five backbones, and the explicit reporting of the validation-based selection process; these are useful for practitioners who want to reproduce the result.","major_comments":[{"comment":"The validation set is used for model selection (Table V: choosing MaxViT, CoAtNet, and EVA-02 from five candidates), for learning-rate scheduling (ReduceLROnPlateau in Section II-E), and for the final accuracy claim (Table VI). Consequently, the reported 95.83% accuracy is an in-sample, selection-biased estimate and cannot be interpreted as a measure of generalization to unseen deepfakes. The abstract and conclusion nevertheless claim generalization to 'unseen scenarios' and 'real-world conditions', but no independent test set, cross-validation, or repeated-seed results are provided. This overclaim is load-bearing because the reported performance is the paper's main contribution.","section":"III-B, III-C, II-E"},{"comment":"The paragraph on diverse scenarios states that the authors 'expect that these strategies will make our model both generalizable and fair across diverse scenarios' and contains no quantitative evaluation on age, ethnicity, or other scenario shifts. This is an explicit limitation in the text, yet the conclusion asserts that ensembling 'has improved accuracy, ensuring better performance in diverse and unseen datasets.' The paper needs either actual evaluations on held-out scenario splits or a substantial moderation of the generalization claims.","section":"III-D"},{"comment":"The supervised contrastive loss is described only in prose; the exact loss formula, the temperature parameter, and any projection head are not specified. Since the SupCon temperature is a known critical hyperparameter and is not listed in Table II, the reported results are not reproducible from the manuscript. This is a technical omission in the core training procedure and should be corrected with the loss equation and the temperature value.","section":"II-F and Table II"}],"minor_comments":[{"comment":"The dataset description is inconsistent: Section II-A reports 42,690 real and 219,470 fake training images, while Section II-G says three subsets of 73,157 fake images are sampled (which sums to 219,471) and refers to the 12,200 generated fake images as 'from the provided dataset' rather than from the secondary generation described in Section II-B. Please reconcile these numbers and the wording.","section":"II-A and II-G"},{"comment":"The validation set contains only 3,072 images, so accuracy differences of less than about one percentage point may not be statistically meaningful; no confidence intervals or repeated-seed variability are reported, making the ranking of backbones less definitive than the table suggests.","section":"III-B, Table V"},{"comment":"The labels 'CoatNet' and 'EV A-02' in Figure 1 are inconsistent with the text's 'CoAtNet' and 'EVA-02', and 'V oting' in Section II-I should be 'Voting'.","section":"Figure 1 and II-I"},{"comment":"The t-SNE visualizations are generated from 2,000 real and 2,000 fake images from the training dataset; the caption should state this explicitly and the paper should avoid implying that the plots show validation-set behavior.","section":"Figure 2 caption"},{"comment":"The phrase 'robustness and generalization capability' overstates what a single validation-set accuracy can demonstrate; 'performance on the DFWild validation set' would be a more accurate description of the reported baseline numbers.","section":"III-A"}],"recommendation":"major_revision","confidential_remarks":"This is a competition technical report; its main value is as a documented engineering solution. The novelty is low relative to a standard research paper, but the paper is well-structured and the method is described in enough detail to be reproduced once the SupCon temperature and projection-head details are added. The key scientific weakness is the validation-set double-use, which is common in competition write-ups; the authors should be asked to either provide an independent evaluation (e.g., a hold-out split of the training set or public deepfake benchmarks) or to explicitly restrict all claims to the validation partition. The fit with the journal depends on whether competition reports are in scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hi [Colleague],\n\nQuick take: this is a well-executed competition report, not a research paper. The headline 95.83% on DFWild validation is probably a few points optimistic because the same validation set was used to pick these three backbones out of five, schedule the learning rate, and choose the ensemble. So the number is not an unbiased estimate of how it will do on unseen data. Nothing here is conceptually new—SupCon, MaxViT/CoAtNet/EVA-02, and majority voting are all published—but the engineering is clean and the ablation study gives the components some support.\n\nWhat I like: the two-stage freeze-then-classify scheme is described precisely enough to reproduce, and the ablation showing that offline augmentation, online augmentation, and SupCon each matter on CoAtNet is useful evidence. Generating 12,200 extra fake images from seven modern forgery methods is a reasonable move for a competition and nicely documented. The tables are internally consistent.\n\nThe soft spots are the usual ones for competition papers. No independent test set, no repeated runs or error bars, no code or data release. The baseline comparison is single-run, so we don't know if the 8-point gap over InceptionResNetV2 is meaningful. Claims about fairness and generalization to 'unseen scenarios' rely on 'we expect' rather than measurements (Section III-D). The ablation only covers CoAtNet, so we don't get a read on whether SupCon helps the ensemble as a whole. The selection itself—five backbones evaluated on the validation set, then top three kept—inflates the reported accuracy, and with only ~3,072 validation images, the winner's curse is likely non-trivial.\n\nWho is this for? Teams entering similar competitions, and people building deepfake detection pipelines who want a set of components that work together. It is not a methodological contribution.\n\nIf this lands on my desk, I'd send it to review, because the empirical content is real and the flaws are fixable. But I'd require a proper held-out evaluation or at least confidence intervals, artifact release, and toned-down generalization claims before accepting. As a preprint, it's a useful data point for the competition.","headline":"A clean competition pipeline whose headline accuracy is likely inflated by tuning on the same validation set; worth a look for practitioners, not a research advance.","tokens_in":10285,"tokens_out":3019,"would_cite":false,"duration_ms":28819,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Three pretrained vision models, fine-tuned with supervised contrastive loss and combined by majority voting, detect deepfakes on the DFWild validation set at 95.83% accuracy, outperforming standard backbones and each model alone.","keywords":["deepfake detection","supervised contrastive learning","ensemble learning","vision transformer","MaxViT","CoAtNet","EVA-02","DFWild validation"],"falsifier":"Run the trained ensemble on the competition's held-out test set or on deepfakes produced by generation methods absent from the eight training datasets, and compare accuracy with the 95.83% validation number; a substantial drop would show the reported robustness was specific to the validation distribution.","tokens_in":9264,"feed_emoji":"🎭","tokens_out":4741,"duration_ms":42091,"temperature":0.7,"pith_summary":"The paper sets out to build a deepfake detector that generalizes across diverse face-manipulation methods, using the DFWild-Cup validation set as its benchmark. Its recipe is two-stage: first fine-tune three complementary pretrained vision models (MaxViT, CoAtNet, EVA-02) with supervised contrastive loss so real and fake embeddings form separate clusters; then freeze the backbones, train small classifiers, and combine their votes by majority. The reported payoff is 95.83% validation accuracy, clearly above every conventional baseline and above each backbone alone. A sympathetic reader should care because the claim is that architecture complementarity plus contrastive embedding separation, not a single new network, drives robustness to unseen deepfake types.","feed_headline":"Three-model ensemble spots deepfakes with 95.83% accuracy","feed_subtitle":"Contrastive training plus majority voting beats every single model and all ImageNet baselines on DFWild.","key_machinery":"The central mechanism is supervised contrastive (SupCon) loss computed on backbone embeddings during the first training stage, followed by frozen-backbone MLP classifiers trained with BCE loss and a majority-voting fusion of three models. SupCon normalizes the embeddings, builds pairwise similarity, pulls same-label (real-real, fake-fake) embeddings together, and pushes different-label embeddings apart, producing distinct clusters that make the later classifiers' job easier. The voting step combines MaxViT's local-artifact sensitivity, CoAtNet's multi-scale hybrid features, and EVA-02's global contextual understanding.","core_discovery":"The paper's central claim is that a two-stage pipeline—fine-tuning three complementary pretrained vision backbones (MaxViT, CoAtNet, EVA-02) with supervised contrastive loss, freezing them, training lightweight MLP classifiers, and fusing predictions by majority voting—detects deepfakes more reliably than any single component or standard ImageNet backbone. On the DFWild validation set the ensemble reaches 95.83% accuracy, 0.9586 F1, and 0.9807 AUC, against a best baseline of 0.8724 (InceptionResNetV2) and a best single ensemble member of 0.9508 (MaxViT). The authors attribute the gain to complementary feature extraction and to supervised contrastive loss creating well-separated real/fake embedding clusters, visualized with t-SNE showing overlapping distributions before training and separated clusters afterward.","pith_inferences":["The paper selects its three backbones, schedules learning rates, and reports its final number all on the same DFWild validation set, so the 95.83% figure is likely an optimistic estimate of performance on a truly novel deepfake distribution; a held-out competition test set would be needed to quantify the gap.","The same two-stage recipe of contrastive fine-tuning, frozen features, and a lightweight head could transfer to other binary image-forensics tasks such as GAN-image detection or copy-move detection, provided the augmentation diversity covers those artifact types.","A testable extension would be to replace majority voting with soft voting or a learned fusion layer, since the paper's rule only keeps the highest or lowest probability and discards some confidence information from the middle model.","Because all three backbones carry ImageNet-scale pretraining, part of the gain may come from pretraining scale rather than the deepfake-specific objective; ablating SupCon with a randomly initialized small backbone could separate these effects."],"forward_implications":["If the ensemble generalizes as claimed, large-scale ImageNet pretraining plus supervised contrastive fine-tuning can yield deepfake detectors that work across generators not seen during training.","The ablation indicates that online augmentation is the most influential single component, since removing it drops CoAtNet accuracy from 0.9447 to 0.8659, so data diversity deserves as much attention as architecture choice.","Majority voting improves on the best individual model by about one percentage point (0.9508 to 0.9583), showing complementary vision transformers can be combined without retraining the backbones.","Replacing SupCon with BCE loss lowers CoAtNet accuracy to 0.9163, suggesting the contrastive objective contributes beyond what the final classifier alone provides."],"supporting_citations":[{"why":"Supplies the MaxViT backbone and its multi-axis self-attention mechanism for local-global feature extraction.","marker":"[1]"},{"why":"Supplies the CoAtNet backbone, whose hybrid convolution-attention design provides multi-scale feature capture.","marker":"[2]"},{"why":"Supplies the EVA-02 backbone and its masked-image-modeling pretraining for global contextual features.","marker":"[3]"},{"why":"Defines supervised contrastive loss, the objective used to separate real and fake embeddings in stage one.","marker":"[5]"},{"why":"Provides the DeepfakeBench framework and DFWild dataset/evaluation protocol on which the accuracy and AUC claims are measured.","marker":"[6]"}],"fun_headline_variants":["Contrastive ensemble beats single models on deepfakes","Three backbones, one vote: 95.83% deepfake accuracy","Supervised contrastive fusion tops deepfake detection","Ensemble of three vision models nails deepfakes at 95.83%","Freeze, fine-tune, vote: robust deepfake detector hits 95.83%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The DFWild validation set is used to choose the three backbones, to schedule the learning rate, and to report the 95.83% accuracy, so the central claim assumes this validation set faithfully represents performance on truly unseen deepfake data.","fun_headline_variants_meta":{"raw":{"variants":["Contrastive ensemble beats single models on deepfakes","Three backbones, one vote: 95.83% deepfake accuracy","Supervised contrastive fusion tops deepfake detection","Ensemble of three vision models nails deepfakes at 95.83%","Freeze, fine-tune, vote: robust deepfake detector hits 95.83%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000879,"raw_usage":{"total_tokens":3781,"prompt_tokens":909,"completion_tokens":2872,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":525,"completion_tokens_details":{"reasoning_tokens":2776}},"tokens_in":525,"tokens_out":2872,"duration_ms":20562,"temperature":1.0,"reasoning_tokens":2776,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T11:16:03.072913+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the trained ensemble on the competition's held-out test set or on deepfakes produced by generation methods absent from the eight training datasets, and compare accuracy with the 95.83% validation number; a substantial drop would show the reported robustness was specific to the validation distribution.","supporting_citations":[{"cited_title":"Eva-02: A visual representation for neon genesis,","cited_arxiv_id":null,"evidence_quote":"Supplies the EVA-02 backbone and its masked-image-modeling pretraining for global contextual features."},{"cited_title":"Deepfakebench: A comprehensive benchmark of deepfake detection,","cited_arxiv_id":null,"evidence_quote":"Provides the DeepfakeBench framework and DFWild dataset/evaluation protocol on which the accuracy and AUC claims are measured."}],"review_version":1}