{"id":"961279cd-fe69-4a55-9fbc-e8539486a8d1","arxiv_id":"2505.04460","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Trained on FF++ only, RealID uses real-face prototypes and an independent dual-decision classifier to reach 91.06 average AUC across five deepfake test sets, 1.74 points above SBI.","lead":"This paper introduces RealID, a deepfake detector that learns a set of 'real face' prototypes and makes separate decisions about whether a face looks real or fake. It reports gains on five test datasets after training only on FaceForensics++, with the headline result being a 1.74 point average AUC improvement over the previous best method.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported 1.74% average AUC gain is not trustworthy as a cross-dataset result because λ1–λ3 were selected using Celeb-DF and DFDC, two of the five target test sets.","rationale":"I read the paper as proposing two complementary mechanisms: RealC2 uses a multi-real memory to learn diverse real-face prototypes, and IDC uses a four-output softmax with a regularizer to decouple, at least partially, the real and fake evidence. The mechanism is coherent, and the cross-backbone results in Table 2 show consistent improvements, which is genuine supporting evidence. The central quantitative claim, however, rests entirely on the cross-dataset comparison in Table 1. The paper's own parameter analysis reveals that the three loss weights were tuned by looking at AUC on Celeb-DF and DFDC, two of the five test datasets. Since the same weights are used to produce every row of the final evaluation, the average AUC is contaminated by test-set information. Given that the reported advantage over SBI is small (1.74 points) and no variance estimates are given, this leakage is the most load-bearing weakness. A clean validation-based retuning could either confirm or refute the claim, so the appropriate verdict remains conditional rather than rejection. The reader's weakest assumption identified the same issue, and I agree with that assessment.","tokens_in":13043,"tokens_out":4206,"duration_ms":43922,"concrete_test":"Re-run the full pipeline on FF++ while holding out a validation split (e.g., 20% of FF++ videos) and grid-search λ1, λ2, and λ3 on that split only. Freeze the best configuration, then evaluate once on Celeb-DF, DFD, DFDC, DFDCp, and UADFV. If the resulting average AUC falls below SBI's 89.32 or drops by more than 1.74 points, the headline improvement is explained by test-set tuning. In addition, run the frozen configuration for at least 5 seeds and report mean ± std to check whether the 1.74-point gap exceeds seed-level noise.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that RealID, trained only on FF++, achieves state-of-the-art cross-dataset generalization with an average AUC of 91.06 versus 89.32 for SBI (Table 1). That comparison is only valid if the model configuration is fixed before any target test set is examined. The paper explicitly documents otherwise: Section 4.1 says λ1, λ2, λ3 are selected via grid search, and Section 4.3/Figure 4 varies each λ while evaluating on DFDC and Celeb-DF, then chooses the final configuration from those curves: \"Based on the combined performance of the three parameters, we empirically selected λ1 = 0.6, λ2 = 1.0, and λ3 = 1.0.\" Celeb-DF and DFDC are two of the five datasets whose average defines the headline number, so the reported hyperparameters are partially chosen to maximize performance on the evaluation set. Because the gain over SBI is only 1.74 average AUC points and no error bars or multi-seed results are provided, the observed advantage may be an in-sample artifact of test-set tuning rather than a genuine improvement in generalization to unseen domains. The method itself is plausible and the backbone experiments provide some supporting evidence, but the central empirical claim is under-supported by the current protocol.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RealID, a deepfake detection method trained on FF++ only, with two new components: a Real Concept Capture Module (RealC2) that maintains multiple real-face prototypes in a memory and uses distinction and diversity losses, and an Independent Dual-Decision Classifier (IDC) that outputs four logits with a regularization term intended to decouple the real and fake probability estimates. The authors report cross-dataset AUC on Celeb-DF, DFD, DFDC, DFDCp, and UADFV, claiming an average AUC of 91.06 versus 89.32 for the strongest prior baseline SBI, a 1.74-point improvement, along with ablations and backbone-transfer experiments.","tokens_in":13353,"tokens_out":4764,"duration_ms":45668,"significance":"If the empirical claim were established, the work would be a useful and modular contribution to generalizable deepfake detection: the method is simple, backbone-agnostic, and the paper includes experiments on five datasets, three backbones, and module/loss ablations. The proposed mechanism of learning diverse real-face prototypes rather than only forgery artifacts is well motivated and the qualitative t-SNE and heatmap evidence supports the authors' interpretation. However, the central quantitative claim is currently not supported because the hyperparameters were selected using two of the five target test sets, and no uncertainty quantification is provided; the 1.74-point average gain could shrink or disappear under a properly held-out validation protocol and multi-seed evaluation.","major_comments":[{"comment":"The hyperparameters λ1, λ2, and λ3 in Eq. (15) are selected via grid search using performance on Celeb-DF and DFDC, which are two of the five datasets that define the headline average AUC in Table 1. The text states that \"based on the combined performance of the three parameters, we empirically selected λ1 = 0.6, λ2 = 1.0, and λ3 = 1.0.\" This makes the reported 1.74-point average improvement over SBI an in-sample estimate on target test sets, not a cross-dataset generalization result. Please re-run the selection procedure on a validation set disjoint from all target test datasets (for example, a held-out split of FF++ or an additional dataset not used in the final evaluation), or explicitly reframe the reported numbers as test-set-tuned rather than generalization numbers.","section":"Section 4.1 and 4.3 (Figure 4)"},{"comment":"All reported AUC values are single-run numbers without error bars, standard deviations, or numbers of trials. The headline contribution is a 1.74-average-AUC gain over SBI; in deepfake detection this margin can easily be within run-to-run variance, especially with small batch sizes and random prototype initialization. The phrase \"significantly outperforms\" is therefore not statistically supported. Please provide at least three to five independent runs per configuration and report mean ± std, and clearly state whether baseline numbers are taken from the original papers or from re-implementations under the same protocol as RealID.","section":"Tables 1, 2, 3, 4"},{"comment":"Several free parameters of the method are never specified: the number of real prototypes K, the diversity margin α in Eq. (9), and the regularization intensity β in Eq. (13). Only λ1, λ2, λ3 are reported. Since these parameters control the core RealC2 and IDC mechanisms, the experiments are not fully reproducible and the sensitivity of the method to these choices is unknown. Please report all hyperparameter values and include ablations on K, α, and β (or justify why they are fixed constants).","section":"Section 3.1 and Section 4.1"}],"minor_comments":[{"comment":"The abstract reports a \"1.74% improvement in average accuracy,\" but the evaluation metric throughout the paper is AUC; please use \"average AUC\" for consistency and precision.","section":"Abstract"},{"comment":"The denominator in the Prototype Distinction Loss is typeset as \"2KP k=K exp(·)\", which is garbled and does not parse as a sum over the K real prototypes plus the fake mean prototype; please rewrite it with proper summation notation.","section":"Eq. (7)"},{"comment":"The Independent Dual-Decision Regularization as written is discontinuous at |d| = 1 when β = 1.0, since the first branch gives β and the second branch gives 0 at that point; please clarify the intended functional form or correct the typo.","section":"Eq. (13)"},{"comment":"The caption reads \"For λ1, λ1, and λ1\" where it should read \"λ1, λ2, and λ3.\"","section":"Figure 4 caption"},{"comment":"The conclusion refers to \"Independent Decision Classification Module\" instead of the earlier \"Independent Dual-Decision Classifier\" (IDC); please make the terminology consistent.","section":"Section 5"},{"comment":"Training details such as optimizer, learning rate, number of epochs, number of training frames, and data augmentation are not reported; these are needed for reproducibility.","section":"Section 4.1"},{"comment":"The phrase \"no-cherry-pick heatmaps\" should be rephrased as \"non-cherry-picked heatmaps\" or \"randomly selected heatmaps\" for clarity.","section":"Section 4.4"}],"recommendation":"major_revision","confidential_remarks":"The test-set hyperparameter tuning issue is the main obstacle to publication: Figure 4 explicitly evaluates on Celeb-DF and DFDC, both of which are part of the final average. This is a data-leakage problem in the empirical protocol rather than a purely presentational weakness. The proposed architecture is plausible, but the quantitive claim will need to be re-established with a clean validation procedure and multi-seed statistics before the paper can be accepted. I would also ask the authors to specify K, α, and β, since these parameters directly control the novelty of the method."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The punchline: RealID is a thoughtful combination of memory-guided real-face prototypes and an independent dual-decision classifier, but its central empirical claim—the 1.74 average AUC gain over SBI—is weakened by a protocol problem. The λ1–λ3 hyperparameters were chosen via grid search, and Figure 4 explicitly evaluates candidate settings on Celeb-DF and DFDC, two of the five datasets that define the headline average. That makes the reported gain in-sample on those datasets, not a clean cross-dataset result.\n\nWhat is genuinely new: the multi-prototype real-face memory with the distinction and diversity losses, plus the four-way independent decision classifier, is a sensible new combination. The authors document the machinery carefully—prototype updates, the two-path regularization—and the ablations show both modules contribute. The cross-backbone results (Xception, ViT-L, ViT-B) are a plus: the improvement is consistent, even if smaller on stronger backbones, which is what you'd expect.\n\nWhere it falls short: the test-set tuning is the main issue. Without a held-out validation split for hyperparameter selection, the 1.74% could shrink or vanish on a fresh target dataset. There are also no error bars or multi-seed results anywhere in the paper, so we have no sense of the variance. The differences in Table 1 between RealID and SBI on some datasets are sizable (e.g., Celeb-DF +1.98, UADFV +1.06), so the method may well be genuinely better—but the current protocol doesn't support the claim at face value. Minor points: the abstract says \"average accuracy\" when the metric is AUC, and Section 4.4 has a typo (\"Qualititive\"). These are cosmetic.\n\nThe citation pattern looks fine: the self-citations are background, and Park et al. 2020 for memory guidance is properly credited. No signs of invented entities or circularity beyond the tuning issue.\n\nWho this is for: someone working on generalizable deepfake detection will find the architecture worth reading and probably worth building on. But the evaluation protocol needs fixing before the headline number is taken seriously.\n\nMy recommendation: send it to peer review. The idea is solid and the experiments are extensive, but the revision should require a validation-based tuning protocol, multi-seed statistics, and released code. Without those, the 1.74% claim is provisional.","headline":"RealID is a plausible new architecture for cross-dataset deepfake detection, but the headline 1.74% average AUC gain over SBI is compromised by test-set hyperparameter tuning.","tokens_in":13840,"tokens_out":2401,"would_cite":false,"duration_ms":20039,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RealID, trained only on FaceForensics++, achieves 91.06 average AUC across five unseen deepfake datasets by learning diverse real-face prototypes and deciding real versus fake independently.","keywords":["deepfake detection","cross-dataset generalization","real facial concepts","prototype learning","memory networks","independent dual-decision classifier","FaceForensics++","AUC"],"falsifier":"Run RealID and SBI on FF++ exactly as reported, then evaluate both on a curated deepfake dataset that was not touched during development and not derived from any dataset used for tuning or ablations; if the average AUC gap over SBI is smaller than 1.74 or reverses, the reported advantage was tuned to the five test sets rather than a general property.","tokens_in":12862,"feed_emoji":"🎭","tokens_out":6608,"duration_ms":55988,"temperature":0.7,"pith_summary":"RealID claims that deepfake detectors fail on unseen datasets because they learn a single, narrow notion of real faces and lean too heavily on forgery artifacts, which leads them to label genuine faces as fake. To fix this, it stores several prototypes of real faces in a memory bank and trains a classifier that scores the real and fake hypotheses independently instead of as two sides of one coin. Trained only on FaceForensics++, RealID raises the average cross-dataset AUC across Celeb-DF, DFD, DFDC, DFDCp, and UADFV to 91.06, beating the strongest prior method by 1.74 average AUC. The implied payoff is a detector whose decisions transfer to forgery types and image conditions it has never seen.","feed_headline":"RealID: modeling real faces boosts cross-dataset deepfake detection","feed_subtitle":"Training on FF++ alone, RealID averages 91.06 AUC across five unseen datasets, 1.74 points above the prior best","key_machinery":"The load-bearing object is the pair (RealC2, IDC). RealC2 is a memory bank of $K$ real-face prototypes, updated with a softmax-weighted combination of the real features assigned to each prototype, plus two losses: $\\mathcal{L}_{\\text{Distinction}}$ pulls each real feature toward its nearest prototype and the mean real prototype while pushing it away from fake prototypes, and $\\mathcal{L}_{\\text{Diversity}}$ keeps the second-nearest prototype at least a margin away. IDC is a classifier with four outputs and regularization $R$: for a real sample the term pushes the fake probability toward the auxiliary output $\\hat f_j^3$, and for a fake sample it pushes the real probability toward $\\hat f_j^2$, making the two decisions independent. The total loss is $\\mathcal{L}_{\\text{CE}}+\\lambda_1\\mathcal{L}_{\\text{Diversity}}+\\lambda_2\\mathcal{L}_{\\text{Distinction}}+\\lambda_3 R$.","core_discovery":"The paper's central claim is that generalization in deepfake detection improves when the model builds a comprehensive concept of real faces and applies it independently of forgery cues. The Real Concept Capture module (RealC2) maintains $K$ real prototypes in a Multi-Real Memory, updating each prototype from the real samples assigned to it and enforcing both intra-subclass tightness (Prototype Distinction Loss) and spread between prototypes (Prototype Diversity Loss). The Independent Dual-Decision Classifier (IDC) replaces the binary output with a four-dimensional softmax, $\\left(\\hat f_j^0, \\hat f_j^1, \\hat f_j^2, \\hat f_j^3\\right)$, whose first two entries are still real and fake probabilities but whose auxiliary entries absorb error through a regularization term, so a rise in the fake score no longer forces a fall in the real score. With both modules, the model can reject a sample because it lacks the learned real-face concept rather than because some local artifact resembles a forgery, and the reported cross-dataset results support that mechanism.","pith_inferences":["The paper does not test this, but a natural extension is applying RealID to video-level detection by aggregating frame-level independent real/fake scores, which should reduce false positives on real videos since misclassified frames would no longer share a single artifact-driven decision.","The independent-dual-decision trick could be borrowed by any binary classifier that suffers from asymmetric base rates, e.g., anomaly detection where the normal class is over-sampled, as a cheap way to avoid complementarity bias.","The prototype-update rule assumes the real class has meaningful subclusters; an easy test is to vary $K$ on a dataset with one dominant ethnicity or lighting condition and check whether the memory collapses."],"forward_implications":["If the two-module design is responsible for the gain, then any detector that learns a multi-prototype model of the real class and decouples the real/fake decisions should transfer better, not just this architecture.","The 1.74 average AUC gain over SBI on five datasets suggests the main remaining generalization bottleneck is the real class distribution, not richer forgery augmentation.","Backbone-agnostic improvements (Xception, ViT-L, ViT-B all gain from RealID) mean the module can be grafted onto stronger feature extractors as they appear.","Heatmaps that spread attention across lips, nose, and eyes, rather than local traces, imply the detector is using a holistic real-face criterion; video-level aggregation may amplify this advantage."],"supporting_citations":[{"why":"Supplies the FaceForensics++ training set and the Xception baseline used in the backbone comparison.","marker":"[Rössler et al., 2019]"},{"why":"Self-blended images (SBI) is the strongest prior baseline; its 89.32 average AUC is the number RealID must beat.","marker":"[Shiohara and Yamasaki, 2022]"},{"why":"Provides the memory-guided normality mechanism that Multi-Real Memory adapts for storing real-face prototypes.","marker":"[Park et al., 2020]"},{"why":"Supplies the Celeb-DF test set, the hardest cross-dataset target where RealID gains about 20 AUC over FoCus.","marker":"[Li et al., 2020b]"},{"why":"Supplies the DFDC and DFDCp test sets used in cross-dataset evaluation.","marker":"[Dolhansky et al., 2020]"},{"why":"Supplies the UADFV test set where RealID reaches 98.34 AUC.","marker":"[Li et al., 2018]"},{"why":"Supplies the DFD test set used in the five-dataset evaluation.","marker":"[Dufour and Gully, 2020]"},{"why":"EfficientNet is the default backbone; RealID must improve on its 73.16 average AUC.","marker":"[Tan and Le, 2019]"}],"fun_headline_variants":["RealID learns real-face concepts to boost deepfake detection","Independent real-face concept improves cross-dataset deepfake detection","RealID: separate real and fake scores for better deepfake generalization","Modeling real faces independently lifts deepfake detection accuracy","RealID gains 1.74% by learning real-face prototypes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The head-to-head claim stands on the assumption that the three loss weights $\\lambda_1,\\lambda_2,\\lambda_3$ were fixed without consulting the five test sets; the paper grid-searches them and reports Figure 4 varying each weight while evaluating on Celeb-DF and DFDC, two of the five evaluation datasets, so the 1.74 average AUC margin is an in-sample number unless a separate validation split was used.","fun_headline_variants_meta":{"raw":{"variants":["RealID learns real-face concepts to boost deepfake detection","Independent real-face concept improves cross-dataset deepfake detection","RealID: separate real and fake scores for better deepfake generalization","Modeling real faces independently lifts deepfake detection accuracy","RealID gains 1.74% by learning real-face prototypes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000395,"raw_usage":{"total_tokens":2082,"prompt_tokens":965,"completion_tokens":1117,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":581,"completion_tokens_details":{"reasoning_tokens":1033}},"tokens_in":581,"tokens_out":1117,"duration_ms":8574,"temperature":1.0,"reasoning_tokens":1033,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:27:20.008484+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run RealID and SBI on FF++ exactly as reported, then evaluate both on a curated deepfake dataset that was not touched during development and not derived from any dataset used for tuning or ablations; if the average AUC gap over SBI is smaller than 1.74 or reverses, the reported advantage was tuned to the five test sets rather than a general property.","supporting_citations":[{"cited_title":"Detecting deepfakes with self-blended images","cited_arxiv_id":null,"evidence_quote":"Self-blended images (SBI) is the strongest prior baseline; its 89.32 average AUC is the number RealID must beat."},{"cited_title":"Learning memory-guided normality for anomaly detection","cited_arxiv_id":null,"evidence_quote":"Provides the memory-guided normality mechanism that Multi-Real Memory adapts for storing real-face prototypes."},{"cited_title":"The deepfake detection challenge dataset","cited_arxiv_id":null,"evidence_quote":"Supplies the DFDC and DFDCp test sets used in cross-dataset evaluation."},{"cited_title":"Deepfake detection dataset, 2020","cited_arxiv_id":null,"evidence_quote":"Supplies the DFD test set used in the five-dataset evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"EfficientNet is the default backbone; RealID must improve on its 73.16 average AUC."}],"review_version":1}