{"id":"fca968cc-25ed-43c8-a992-71b90530469a","arxiv_id":"2507.18015","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Celeb-DF++ is a more diverse deepfake video benchmark, spanning 22 generation methods and three manipulation scenarios, on which 24 published detectors drop to about 70 to 72 percent average AUC in cross-method tests.","lead":"This paper releases Celeb-DF++, a video benchmark with 53,196 AI-generated fake face videos built from 22 different generation methods across face-swapping, face-reenactment, and talking-head scenarios. It then runs 24 deepfake detectors on the new data and shows they lose roughly 5 to 7 AUC points versus older benchmarks, evidence that generalizable deepfake detection remains unsolved.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Talking-face construction from a single first frame with mismatched VoxCeleb2 audio may manufacture the cross-scenario AUC drop, so the benchmark's 'generalizable forensics' interpretation is not yet secured.","rationale":"The reader's weakest assumption was that the 22 generation pipelines produce artifacts representative of real-world deepfakes, with the Talking-face first-frame/VoxCeleb2 protocol singled out as the main confound. That is the same load-bearing concern I identify. I do not see an internal contradiction that overturns the paper's numeric claims: the tables are broadly consistent, the release makes the resource useful, and the headline drop is large across most detectors. However, the central interpretation—that the drop demonstrates failure of generalizable forensics in the wild—depends on the unvalidated assumption about pipeline realism. The proposed controlled TF ablation would settle whether the single-source-frame and mismatched-audio construction is doing the work. Because the reader already conditioned acceptance on exactly these caveats (and on release of evaluation code and splits), I would not move the verdict; it should remain CONDITIONAL/UNCHANGED pending that evidence.","tokens_in":28397,"tokens_out":14986,"duration_ms":175338,"concrete_test":"Run a controlled TF ablation: for 100 real videos, re-generate each of the 7 TF methods' videos using a randomly selected mid-video frame as the source image and the original interview audio instead of random VoxCeleb2 clips, keeping all other generation settings identical; then re-run the 8 GF-eval detectors under the same Celeb-DF training and compare per-method AUC with Tables VI/VII. If SadTalker/EchoMimic AUC rises by more than about 5 points, the reported TF difficulty is substantially a construction artifact rather than evidence about generalizable real-world forensics.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central numeric result—an average 5.2/7.1 point drop and GF-eval AUCs of about 71.7/72.1—is not the load-bearing point; the interpretation that this drop measures difficulty of detecting generalizable forgeries is. Section III-B constructs all 20,279 Talking-face videos from the first frame of each of the 590 real videos, driven by five random VoxCeleb2 audio segments. This gives every TF fake a static source-frame/background, a single fixed identity image, and audio that never matches the target speaker. Such pipeline-level properties are shared by all seven TF generators but are not properties of real-world talking-face forgeries. The lowest GF-eval per-method AUCs are concentrated in this scenario (EchoMimic 51.2/46.3, SadTalker 53.8/45.7 in Tables VI/VII), so the cross-scenario failure could be driven by the construction protocol rather than by manipulation subtlety or genuine diversity. The paper provides no human realism evaluation, no per-method quality or failure rates, and no ablation separating method artifacts from source-frame/audio artifacts, so the central claim survives numerically but its stated meaning—benchmarking generalizable forensics in the wild—is unsecured.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Celeb-DF++, a large-scale video deepfake benchmark extending the authors' earlier Celeb-DF dataset. It contains 53,196 (sic) fake videos generated by 22 methods spanning face-swap (FS), face-reenactment (FR), and talking-face (TF) scenarios, and evaluates 24 detection methods under three new protocols: GF-eval (cross-method generalization), GFQ-eval (cross-compression robustness), and GFD-eval (cross-dataset generalization). The reported results show that all assessed detectors drop on Celeb-DF++ relative to Celeb-DF, with average decreases of approximately 5.2% frame-level AUC and 7.1% video-level AUC, and that the eight detectors retrained under GF-eval average about 71.7% frame-level and 72.1% video-level AUC. The dataset is released publicly.","tokens_in":28465,"tokens_out":12256,"duration_ms":111358,"significance":"If the construct-validity issues are addressed, this is a potentially valuable community resource: it is one of the largest and most method-diverse video deepfake benchmarks, with 22 generation pipelines and 24 assessed detectors, and the proposed per-method evaluation tables are detailed enough to support future comparisons. The authors are transparent about protocol details and release the dataset, which will enable reproducibility. However, the benchmark's interpretation as a faithful measure of generalizable forensics for real-world forgeries is not yet secured: the TF construction protocol and the GFQ-eval compression design may confound the measured difficulty with artifacts of the synthetic pipeline. Since the paper's central claim is that the benchmark reflects the diversity of forgeries 'witnessed in the wild,' the current evidence is incomplete without human evaluation, quality metrics, or ablations that separate method-specific artifacts from shared construction choices.","major_comments":[{"comment":"The Talking-face scenario is constructed from the first frame of each real video as a single source image, driven by audio segments from VoxCeleb2 that do not match the target speaker (Section III-B, Data Generation Details). Because all seven TF generators share this protocol, the uniformly low per-method AUCs in the TF columns of Tables VI and VII (e.g., EchoMimic 51.2/46.3, SadTalker 53.8/45.7) may be driven by these common pipeline artifacts—static background, frozen identity image, audio-visual mismatch—rather than by the subtlety of the manipulations or by genuine diversity of real-world talking-face forgeries. The manuscript provides no human realism evaluation, no per-method quality or failure-rate statistics, and no ablation separating method-specific artifacts from the shared source-frame/audio protocol. As the paper's stated contribution is a benchmark for generalizable forensics against forgeries 'witnessed in the wild,' this construct-validity gap leaves the central interpretation of the cross-scenario drops unsecured. Please add a limitations discussion and supporting analyses (e.g., full-video sources, matched audio, or human perceptual ratings).","section":"III-B, Tables VI-VII"},{"comment":"In the GFQ-eval protocol, the text states that FFmpeg is used to compress 'DeepFake videos,' and the table captions refer to compression of the fake methods; there is no statement that the real videos are subjected to the same H.264 (c35/c45) re-encoding. If the real test videos retain their original quality while only the forged videos are re-compressed, the AUC drops in Tables VIII–XI conflate the obscuration of forgery traces with a class-conditional quality artifact that detectors can exploit (e.g., recognizing H.264-induced artifacts as a fake cue). Please clarify whether real videos are compressed in the same way; if they are not, the protocol should be re-run with matched compression of both classes, or the paper should explicitly justify why the current procedure is a valid measure of compression robustness.","section":"III-E, Protocol #2"}],"minor_comments":[{"comment":"The phrase 'randomly select 5 audio segments from VoxCeleb2 for each video frame' is ambiguous and likely meant 'for each video' rather than 'for each video frame.' Please clarify the audio-selection procedure.","section":"III-B"},{"comment":"The total fake-video count of 53,196 does not match the sum of the per-method counts shown in Fig. 2 (5,639 + 13,646 + 13,631 + 20,279 = 53,195). In addition, the sentence 'For Face-swap and Face-reenactment scenario, we randomly select 2,000 identity pairs ... resulting in 13,646 videos' misattributes the FS-only count to both scenarios; the seven additional FS methods sum to 13,646, while the seven FR methods sum to 13,631. Please correct the counts and the wording.","section":"III-B, Abstract"},{"comment":"The statement that all detectors are run 'using their default settings' is not fully specified; it is unclear whether the default settings in DeepfakeBench match the original papers' recommended hyperparameters, and whether the recent methods (CFM, ProDet, ForAda, Effort) are trained with identical data splits. Please provide implementation details or a pointer to the code repository.","section":"III-C"},{"comment":"The selection of eight detectors for retraining under GF-eval should be justified; please specify which of the 24 detectors had released training code and how the chosen eight are representative of the design-space categories in Table III.","section":"III-E, GF-eval"},{"comment":"The reported average decreases of 'approximately 5.2% in frame-level AUC and 7.1% in video-level AUC' do not specify the baseline version of Celeb-DF; from Tables IV and V, the drops are relative to the v2 (official) version. Please state this explicitly.","section":"Section III-D"},{"comment":"There are several minor typos and grammatical issues, including 'developement' (Introduction), 'larges-scale' (Related Works), 'boarder range' (Conclusion), and inconsistent use of singular/plural in 'Face-swap and Face-reenactment scenario.' A careful proofread is recommended.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is authored by the creators of Celeb-DF, and the new benchmark is an extension of that work; the self-citation is appropriate. The two main risks are the construct validity of the talking-face scenario and the GFQ-eval compression confound, both of which are addressable in revision. I would support publication if these issues are resolved and the count errors are corrected."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the resource is real: 22 generators across face-swap, face-reenactment, and talking-face scenarios, 53k fake videos, 24 detectors evaluated, with recent 2024-2025 tools like EchoMimic, FLOAT, and AniTalker included, and the dataset is released. That is a step up from every existing video benchmark, which mostly cover one scenario and a handful of methods. Second, the headline interpretation - that detectors fail because forgeries are diverse - is not secured. The talking-face videos are all generated from the first frame of the real video, driven by VoxCeleb2 audio that never matches the speaker. That gives every TF fake a static background, a single identity frame, and mismatched audio. Those are pipeline artifacts, not properties of real-world talking-face forgeries, and the worst detector numbers concentrate in that scenario (EchoMimic 51.2/46.3, SadTalker 53.8/45.7 in Tables VI/VII). So the measured drop is real, but the reason is underdetermined: it could be method subtlety, or it could be the shared source-frame/audio confound. The paper provides no human realism evaluation, no per-method quality or failure rates, and no ablation separating these. That is the load-bearing soft spot, and it is fixable. Credit where earned: the tables are extensive and internally consistent aside from an off-by-one in the video count (per-method counts sum to 53,195 vs. claimed 53,196). Using 200-300 test videos per method is reasonable for a first pass. Including 2024-2025 generators addresses a real gap, and the dataset link works. The paper is honest about which detectors could be retrained for GF-eval (8 of 24), though that limit weakens the protocol. Other soft spots, in proportion: no error bars anywhere, so AUC differences of a few points may or may not be meaningful; GF-eval's 8 detectors are a convenience sample, not a balanced one; ForAda, from the authors' own group, is included without disclosed data separation; and the evaluation infrastructure descends from the group's DeepfakeBench, which is fine but should be stated more explicitly. None of these are fatal. Who is this for? Anyone working on deepfake detection who needs a harder, more method-diverse testbed than Celeb-DF or DFDC. The paper deserves a serious referee and a conditional accept: ask the authors to release the full evaluation pipeline and split lists, add variance or confidence intervals, and add an ablation that separates method-specific artifacts from the talking-face source-frame/audio confound. Without that ablation, the 'generalizable forensics' claim remains plausible but unproven. Desk rejection would be wrong.","headline":"A genuinely large and method-diverse deepfake video benchmark that deserves a serious referee, but its central 'generalizable forensics' interpretation is undercut by a talking-face construction confound that the paper never ablates.","tokens_in":776,"tokens_out":915,"would_cite":true,"duration_ms":27886,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"All 24 tested deepfake detectors lose accuracy on a new, more diverse benchmark.","keywords":["deepfake detection","benchmark","generalizable forensics","face-swap","face-reenactment","talking-face","video forensics","AUC evaluation"],"falsifier":"A detector that generalizes well on Celeb-DF++ but fails on an independent collection of real-world deepfakes (or vice versa) would disconfirm the benchmark's role as a proxy for real-world difficulty. Concretely, one could train a detector on Celeb-DF++ face-swap methods and evaluate it on a held-out set of verified in-the-wild deepfake videos; if its AUC there is close to chance while its Celeb-DF++ AUC is high, the benchmark's cross-method scores overstate real-world generalizability.","tokens_in":28000,"feed_emoji":"🎭","tokens_out":7881,"duration_ms":68969,"temperature":0.7,"pith_summary":"The paper introduces Celeb-DF++, a large-scale video deepfake benchmark built to test whether detectors can generalize to unseen forgery types. It expands the earlier Celeb-DF dataset with 53,196 fake videos generated by 22 recent methods spanning face-swap, face-reenactment, and talking-face scenarios. Evaluated on this benchmark, all 24 recent detection methods drop in AUC compared with Celeb-DF, with average falls of about 5.2 points at frame level and 7.1 points at video level. In a cross-method protocol, the detectors average roughly 72 percent AUC, indicating that none of them reliably identifies forgeries produced by methods they were not trained on. The paper argues that current detectors are not yet ready for generalizable forensics and offers the benchmark as a shared testbed to drive progress.","feed_headline":"All 24 deepfake detectors drop on a new 53,196-video benchmark","feed_subtitle":"Celeb-DF++ spans 22 generation methods and three forgery scenarios; average AUC falls by up to 7.1 points.","key_machinery":"The central object is the benchmark itself: 53,196 forged videos produced by 22 methods (8 face-swap, 7 face-reenactment, 7 talking-face), built on 590 real videos of 59 celebrities. The evaluation machinery is a set of three protocols: GF-eval trains detectors on Celeb-DF in the face-swap scenario and tests across all other methods and scenarios; GFQ-eval adds H.264 compression at two strengths; GFD-eval trains on FaceForensics++ (HQ) and tests across all Celeb-DF++ methods. These protocols convert the question 'can detectors generalize?' into measurable AUC scores at frame and video level.","core_discovery":"The central discovery claimed in this paper is that the difficulty of deepfake detection is substantially underestimated when benchmarks contain only a small number of forgery methods. By extending Celeb-DF to include 22 generation pipelines across three scenarios, the authors show that every one of 24 detectors performs worse on Celeb-DF++ than on the original Celeb-DF, and that cross-method generalization remains low even for the latest detectors. The work further claims that compressed versions of the benchmark degrade performance further, and that detectors trained on FaceForensics++ lose additional accuracy under a cross-dataset protocol. The result is presented as evidence that forgery diversity, not just scale, is the key missing ingredient in existing benchmarks.","pith_inferences":["The talking-face videos are generated from a single first frame and five VoxCeleb2 audio clips, so their motion and background are simpler than in-the-wild talking-head videos; a detector that exploits this pipeline's constraints might score well on the benchmark while failing on real social-media forgeries.","The benchmark stops at 2024-generation methods; diffusion-based video generators that appeared later could behave differently, so the measured gap may be a floor or a ceiling for future detectors.","A productive next test would be to use the benchmark to measure per-method difficulty and correlate it with model architecture, which would tell practitioners which kinds of forgery artifacts are the hardest to model."],"forward_implications":["State-of-the-art detectors overstate their real-world readiness: their published AUCs on single-method benchmarks do not transfer to diverse forgeries.","Cross-method generalization, not just within-method accuracy, becomes the primary evaluation target for new detectors.","Compression is a first-order obstacle: stronger H.264 compression (c45) cuts average frame-level AUC by an additional 4.4 points over already-low cross-method scores.","Training domain matters: detectors trained on FaceForensics++ do worse on Celeb-DF++ than detectors retrained on Celeb-DF, confirming a domain-gap penalty in GFD-eval.","The best-performing detector in the benchmark is not a solution: it leads within face-swap but degrades sharply on talking-face methods."],"supporting_citations":[{"why":"Celeb-DF is the base dataset that Celeb-DF++ extends, providing the 590 real videos and the original face-swap fakes used as the training set in GF-eval.","marker":"[21]"},{"why":"FaceForensics++ (HQ) supplies the training set for the detectors in the initial comparison against Celeb-DF and in the GFD-eval protocol.","marker":"[18]"},{"why":"DeepfakeBench provides the default settings and preprocessing used to implement most of the 24 assessed detectors.","marker":"[94]"},{"why":"FFmpeg is the tool used to compress videos under the two H.264 levels in the GFQ-eval protocol.","marker":"[95]"}],"fun_headline_variants":["All 24 deepfake detectors drop on 53,196-video benchmark","Deepfake detectors stumble on 22-method, 53,196-video set","New benchmark: 24 detectors all drop on diverse forgeries","Celeb-DF++ exposes deepfake detectors' generalization gap","Deepfake detectors drop up to 7.1 points on new benchmark"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that the benchmark measures generalizable forensics rests on the assumption that the artifacts left by these 22 generation pipelines are representative of the deepfakes a detector would meet in the wild, so that a detector's failure on them predicts failure on real-world forgeries.","fun_headline_variants_meta":{"raw":{"variants":["All 24 deepfake detectors drop on 53,196-video benchmark","Deepfake detectors stumble on 22-method, 53,196-video set","New benchmark: 24 detectors all drop on diverse forgeries","Celeb-DF++ exposes deepfake detectors' generalization gap","Deepfake detectors drop up to 7.1 points on new benchmark"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001072,"raw_usage":{"total_tokens":4482,"prompt_tokens":929,"completion_tokens":3553,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":545,"completion_tokens_details":{"reasoning_tokens":3457}},"tokens_in":545,"tokens_out":3553,"duration_ms":30306,"temperature":1.0,"reasoning_tokens":3457,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T14:39:45.215697+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A detector that generalizes well on Celeb-DF++ but fails on an independent collection of real-world deepfakes (or vice versa) would disconfirm the benchmark's role as a proxy for real-world difficulty. Concretely, one could train a detector on Celeb-DF++ face-swap methods and evaluate it on a held-out set of verified in-the-wild deepfake videos; if its AUC there is close to chance while its Celeb-DF++ AUC is high, the benchmark's cross-method scores overstate real-world generalizability.","supporting_citations":[{"cited_title":"Celeb-df: A large-scale challenging dataset for deepfake forensics,","cited_arxiv_id":null,"evidence_quote":"Celeb-DF is the base dataset that Celeb-DF++ extends, providing the 590 real videos and the original face-swap fakes used as the training set in GF-eval."},{"cited_title":"Faceforensics++: Learning to detect manipulated facial images,","cited_arxiv_id":null,"evidence_quote":"FaceForensics++ (HQ) supplies the training set for the detectors in the initial comparison against Celeb-DF and in the GFD-eval protocol."},{"cited_title":"Deepfakebench: A comprehensive benchmark of deepfake detection,","cited_arxiv_id":null,"evidence_quote":"DeepfakeBench provides the default settings and preprocessing used to implement most of the 24 assessed detectors."},{"cited_title":"Converting video formats with ffmpeg,","cited_arxiv_id":null,"evidence_quote":"FFmpeg is the tool used to compress videos under the two H.264 levels in the GFQ-eval protocol."}],"review_version":1}