{"id":"49d798f3-b6c7-4744-b32a-32c4017039f6","arxiv_id":"2507.20579","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"AV-Deepfake1M++ is a 2.05-million-clip audio-visual deepfake benchmark spanning three source corpora, nine generators, and 36 perturbations, with challenge results showing large generalization gaps.","lead":"The paper builds a two-million-clip audio-visual deepfake benchmark, adding more voices, faces, manipulation tools, and realistic distortions than earlier datasets. It ran a public detection challenge on the benchmark and reports that even strong prior detectors fail on the new test conditions.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unverified generation labels are the load-bearing risk: without quality control on LLM/TTS/lip-sync outputs, benchmark labels and localization annotations could be wrong.","rationale":"The reader's weakest assumption is exactly the right one: the automatic generation pipeline produces labels without verification, and every downstream benchmark number depends on those labels being correct. I considered other candidate concerns. The scale arithmetic is internally consistent and the split design is sensible, so the headline size claim is credible from the text. The perturbation table has placeholder rows and mixes generation methods into a perturbation list, but this is a documentation defect rather than a fatal flaw; the perturbation count can be reconstructed and the perturbations themselves are synthetic. The claim of 'nine generation models' is contradicted by the seven models described in Section 3.2 and Table 2, but even seven models would still support the diversity claim. The leaderboard results lack error bars, but they are illustrative challenge results, not the central contribution. By contrast, the label and localization correctness issue is load-bearing in a direct way: if the planned edit is not realized, or the generated audio/video is misaligned, then both the classification labels and the temporal localization ground truth are wrong. That would undermine not only the dataset's usability but also the validity of all reported benchmark numbers and the challenge conclusions. The paper gives no evidence that this failure mode is absent, and the internal inconsistencies in the manuscript suggest the pipeline documentation is not yet reliable enough to assume correctness. The proposed test is feasible: the dataset and scripts are claimed to be publicly released, so a sampled manual and automated audit can directly measure the failure rate. If the failure rate is low, the central claim holds; if it is high, the dataset needs reannotation or filtering. Because the paper's verdict was already conditional and this concern is the basis for that conditionality, the appropriate outcome is unchanged conditional acceptance, with the quality-control audit as an explicit acceptance condition.","tokens_in":12523,"tokens_out":7772,"duration_ms":84855,"concrete_test":"Sample 100-200 clips per manipulation type (insert, replace, delete) across the three source datasets from the released dataset. For each clip, run Whisper forced alignment on the fake audio against the manipulated transcript and compare the planned old_word/new_word boundaries with the actual edited span; run SyncNet (or an equivalent lip-sync confidence measure) on the generated mouth region; and have at least three annotators judge whether the intended semantic edit is realized and whether audio and video are synchronized. Compute: (a) the fraction of clips where the planned edit is absent or misaligned beyond a tolerance (e.g., >2 frames or >100 ms), and (b) the IoU between the planned and actual manipulated segment.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that AV-Deepfake1M++ is a reliable large-scale benchmark for classification and temporal localization. That requires the per-clip labels and the manipulated-segment boundaries to be correct. Section 3.2 describes an automatic pipeline (LLM plan, Demucs separation, TTS pool, lip-sync pool, assembly) but never reports a quality-control step. Labels are assigned by provenance: any clip that went through the pipeline is 'fake', and localization ground truth is inherited from the LLM's token-level plan {operation, old_word, new_word, index}. If ASR timestamps, TTS duration, or lip-sync output drift by more than a few frames, the clip is still labeled fake but the annotated segment boundary is wrong. For delete operations the pipeline 'keep[s] only background noise' from Demucs, with no check that the result is natural or synchronized. This is not a minor annotation issue: every AUC/AP/AR in Tables 4-6 is computed against these labels. The paper reports no failure-rate measurement, no human verification, and no ASR/SyncNet consistency check. The lack of QC is compounded by documentation inconsistencies: Section 1 claims nine generation models while Section 3.2 and Table 2 enumerate seven, and Table 2 itself contains four blank placeholder rows and lists generation methods under a perturbation table. These do not by themselves falsify the central claim, but they underscore that the pipeline is under-specified and its outputs unverified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces AV-Deepfake1M++, a large-scale audio-visual deepfake benchmark containing 2,051,154 clips (about 4,655.9 hours) built from VoxCeleb2, LRS3, and EngageNet. The generation pipeline uses an LLM to plan token-level insert/replace/delete edits, TTS models to synthesize modified speech, and lip-sync models to synthesize modified video; 36 audio-visual perturbations are then applied. The paper defines train/validation/testA/testB splits, reports dataset statistics, and benchmarks classification and temporal-localization methods, including results from the 2025 1M-Deepfakes Detection Challenge.","tokens_in":12756,"tokens_out":7545,"duration_ms":72090,"significance":"If the labels and manipulated-segment boundaries are correct, AV-Deepfake1M++ is a substantial community resource: it roughly doubles the scale of AV-Deepfake1M, adds source diversity, expands generation and perturbation diversity, and provides externally evaluated held-out test sets. The arithmetic in Table 3 is internally consistent, and the use of an independent challenge leaderboard is a practical strength. The main open question is whether the fully automatic generation pipeline produces reliable ground truth, since the paper does not report any verification or failure-rate analysis.","major_comments":[{"comment":"The dataset labels clips by provenance: every clip that passes through the pipeline is 'fake', and temporal localization annotations are inherited from the LLM's token-level plan without verification. The paper reports no quality-control step and no failure-rate measurement for the TTS, lip-sync, or assembly stages. If an LLM edit is not realized, if TTS duration drifts, or if lip-sync output is misaligned, the fake/real label or the segment boundary is wrong, and every AUC/AP/AR in Tables 4-6 is affected. Please add a validation protocol, such as ASR transcript verification, SyncNet consistency scoring, a human sample study, and per-pipeline failure rates, and report the results.","section":"Section 3.2, Tables 4-6"},{"comment":"The contribution bullet states that the pipeline includes 'nine state-of-the-art models' but Section 3.2 and Table 2 enumerate only seven (TalkLip, LatentSync, Diff2Lip; VITS, F5TTS, XTTSv2, YourTTS). The introduction also names 'visual-LipSync' and 'V ALLEX', which do not appear in Table 2. Please correct the count or list all models, and remove the four blank placeholder rows from Table 2; as written, the headline diversity claim is not auditable.","section":"Section 1 and Table 2"},{"comment":"The abstract and introduction claim '15 video-level and 11 audio-level distortions', but Table 2 lists 21 visual and 15 audio perturbation methods (36 unique rows, with PAD_SIMULATION duplicated). Please clarify whether the counts refer to a subset of perturbations, remove duplicate rows, and align the text with the table. The '36 perturbations' claim is a central contribution and must be precisely documented.","section":"Section 4.3 and Table 2"}],"minor_comments":[{"comment":"The 'Score' column in Tables 5 and 6 is not defined; specify how the averaged AP and AR values are combined.","section":"Section 5.1, Tables 5-6"},{"comment":"TestB AUC is reported only for the Xception baseline; state whether the challenge teams were evaluated on TestB and, if so, provide the results.","section":"Table 4"},{"comment":"The sentence 'Pindrop Labs surpasses BA-TFD+ by ≈ 0.52 of the localization score' is ambiguous; the actual difference is 52.49 points (67.20 vs 14.71).","section":"Section 5.2"},{"comment":"The claim that test videos are 'encoded with various codecs' is not supported by any details; list the codecs and encoding parameters or remove the claim.","section":"Section 3.1"},{"comment":"There are several language and formatting issues (e.g., 'we require datasets that rich in type' in the abstract, and the broken running title 'A V-Deepfake1M++'); a careful copyedit is needed.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is effectively a challenge report with substantial self-citation to the authors' prior dataset and challenge. The editor may wish to confirm that the leaderboard teams had no access to test labels and that the reported results are complete. The central claim is plausible, but the absence of any label-verification protocol is the main risk to the dataset's value."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a real dataset contribution, and the scale claim holds up. The stress-test note is right about the main risk: the generation pipeline has no reported quality control, and the labels are load-bearing for every number in the benchmark.\n\nWhat's actually new: 2,051,154 clips from three source corpora (VoxCeleb2, LRS3, EngageNet), three lip-sync generators, five TTS methods, and 36 perturbations. That's a genuine step up from AV-Deepfake1M, LAV-DF, and FakeAVCeleb, and the TestA/TestB split with different perturbation schedules is sensible. The challenge gives external validation, which is more than most dataset papers offer.\n\nThe arithmetic checks out; Table 3 sums to the claimed totals. The pipeline is described at a high level but is reproducible in principle. Self-citations are appropriate for an extension paper.\n\nThe soft spots are real. The absence of any QC step—human or automated—for the LLM/TTS/lip-sync outputs is the biggest one. If TTS durations drift or lip-sync frames misalign, the clip still gets a fake label, but the temporal boundaries won't be right. Delete operations keep 'only background noise' with no check that the result is natural or synchronized. That doesn't kill the classification claim, but it makes the localization annotations uncertain. The paper reports no failure rate, no ASR/SyncNet consistency check. That needs to be answered.\n\nThe documentation is also sloppier than it should be: Section 1 says nine generation models, but Section 3.2 and Table 2 list seven; Table 2 mixes generation methods with perturbations and has four blank rows. Perturbation realism is assumed, not measured, and the leaderboard scores have no error bars. None of these are fatal, but they reinforce the impression of a rushed manuscript.\n\nMy take: this deserves a serious referee. The benchmark is large and the challenge is public, so the community will use it. But the authors should be required to release the data, document perturbation parameters, and report at least a basic QC audit before the benchmark is treated as ground truth. I'd bring it to a reading group and would cite it once the data is accessible and the QC question is addressed.\n\nRecommendation: accept with major revision, contingent on QC evidence and a cleaned-up Table 2.","headline":"Large, genuinely useful deepfake benchmark, but the missing quality control on the generation pipeline is the load-bearing issue.","tokens_in":13384,"tokens_out":3946,"would_cite":true,"duration_ms":35755,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces AV-Deepfake1M++, a two-million-clip audio-visual deepfake benchmark combining nine generation models and 36 real-world perturbations, and reports that prior temporal localization methods collapse on it.","keywords":["audio-visual deepfake","temporal forgery localization","benchmark dataset","real-world perturbations","text-to-speech","lip-sync generation","LLM-driven semantic manipulation","deepfake detection"],"falsifier":"Take a random sample of, say, 1,000 clips from the fake split of AV-Deepfake1M++, and have human annotators or an independently trained audio-visual synchronization model verify that each clip actually contains the intended semantic change and that the audio and visual manipulations are aligned in time. If a non-trivial fraction of clips are mislabeled or misaligned, the benchmark's labels and the leaderboard numbers derived from them would need to be recomputed.","tokens_in":12308,"feed_emoji":"🎭","tokens_out":8571,"duration_ms":76687,"temperature":0.7,"pith_summary":"The paper introduces AV-Deepfake1M++, a benchmark of just over two million audio-visual video clips, about 4,656 hours, built from three real-video sources and designed for training and testing deepfake detectors under conditions closer to online distribution. It extends an earlier dataset by adding two more source datasets, nine generation models spanning speech synthesis and lip-sync, and 36 audio and visual perturbations such as compression, noise, frame jitter, and reverberation. The dataset carries frame-level and video-level annotations for both binary classification and temporal forgery localization, with held-out test sets whose identities, sources, generation methods, and perturbation schedules differ from training. Challenge results reported in the paper show strong classification performance near 0.98 AUC, while a previous state-of-the-art temporal localization method falls from 96.30 to 14.71 AP@0.5, illustrating the difficulty the new perturbations introduce.","feed_headline":"2M deepfake clips expose a blind spot in forgery detection","feed_subtitle":"Nine generators and 36 real-world distortions make locating fakes in time the hard part.","key_machinery":"The load-bearing machinery is the automated generation pipeline: an LLM plans word-level edits, text-to-speech models synthesize the manipulated audio, lip-sync models re-render the mouth region, and post-processing assembles edited segments into untouched video and applies audio and visual perturbations. The four manipulation types separate audio and visual contributions, and the split design isolates perturbations between training and test so that testA and testB measure cross-domain generalization. Dataset statistics—2,051,154 clips, 7,109 subjects, 4,655.9 hours—and a held-out identity/source/generator separation carry the claim that this is a large, diverse benchmark rather than a simple collection of fakes.","core_discovery":"The central claim is that a deepfake benchmark needs scale, generation diversity, and real-world perturbations together, and that AV-Deepfake1M++ provides all three. Real clips come from VoxCeleb2, LRS3, and EngageNet, covering interviews, TED talks, and conversational meetings. Fake clips are produced by an LLM that plans token-level insert, replace, and delete edits to invert the meaning of spoken sentences, followed by one of four text-to-speech models for speech and one of three lip-sync models for video, yielding four manipulation types: real, fake audio with real video, real audio with fake video, and fully fake. After assembly, clips may receive zero to several of 36 perturbations. On this material the best challenge entry reaches 97.83 AUC on testA classification, while the strongest prior temporal-localization baseline collapses to 14.71 AP@0.5, which the paper attributes to the new perturbations and unseen synthesis pipelines.","pith_inferences":["Because every clip is labeled by an automatic pipeline with no reported verification step, a human audit of a random sample would be the natural check on label quality; all benchmark numbers inherit that pipeline's error rate until then.","Since perturbations are separated between training and test sets, test scores measure a mixture of forgery detection and robustness to distribution shift; a detector that keys on perturbation artifacts rather than semantic manipulation could still score well without truly understanding forgeries.","An extension the paper does not test is whether training on AV-Deepfake1M++ transfers to other benchmarks such as FaceForensics++ or DFDC; a positive transfer result would strengthen the claim that the dataset captures general forgery cues.","The choice of LLM-planned meaning inversion points to a future in which fakes act at the level of content semantics, not low-level pixels or waveforms, which would push detection toward checking audio-visual semantic consistency across modalities."],"forward_implications":["Detectors trained on AV-Deepfake1M++ must face unseen generation methods and perturbation schedules in testA and testB, so leaderboard scores reflect cross-domain robustness rather than in-domain memorization.","The reported collapse of BA-TFD+ from 96.30 AP@0.5 on LAV-DF to 14.71 AP@0.5 on AV-Deepfake1M++ indicates that perturbation diversity can invalidate methods tuned to artifacts of a single generator.","The four manipulation types and frame-level annotations allow separate evaluation of audio-only, video-only, and fully cross-modal forgery detection and localization.","With best classification AUC at 97.83 but top localization score near 67, classification appears close to saturated while temporal localization remains the open problem."],"supporting_citations":[{"why":"The prior AV-Deepfake1M dataset whose pipeline this work extends and whose scale is roughly doubled.","marker":"[3]"},{"why":"Defines content-driven audio-visual forgery localization and supplies the LAV-DF benchmark and BA-TFD baseline.","marker":"[5]"},{"why":"Supplies the BA-TFD+ baseline whose large performance drop is the paper's key evidence that new perturbations break existing methods.","marker":"[4]"},{"why":"VoxCeleb2 is one of the three real-video source datasets.","marker":"[10]"},{"why":"LRS3-TED is a second real-video source, adding TED-talk style footage.","marker":"[1]"},{"why":"EngageNet is the third real-video source, adding conversational meeting footage.","marker":"[37]"},{"why":"Xception is the classification baseline whose 55.09 AUC contrasts with the top team's 97.83.","marker":"[9]"},{"why":"Demucs separates speech from background noise in the audio generation pipeline.","marker":"[12]"},{"why":"XTTS is one of the zero-shot text-to-speech models in the audio generation pool.","marker":"[6]"},{"why":"LatentSync is one of the lip-sync models in the visual generation pool.","marker":"[24]"}],"fun_headline_variants":["2M audio-visual deepfakes expose localization blind spot","36 real-world distortions sink deepfake temporal localization","AV-Deepfake1M++: 2M clips, 9 generators, 36 perturbations","New benchmark: 2M deepfake clips with real-world artifacts","Deepfake detection benchmark stresses time-localization limits"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The automatic generation pipeline assumes every assembled clip is correctly labeled; if the language model's semantic edit, the synthesized speech, or the lip-sync alignment fails, the clip's real/fake label and segment boundaries are wrong and every benchmark score inherits that error.","fun_headline_variants_meta":{"raw":{"variants":["2M audio-visual deepfakes expose localization blind spot","36 real-world distortions sink deepfake temporal localization","AV-Deepfake1M++: 2M clips, 9 generators, 36 perturbations","New benchmark: 2M deepfake clips with real-world artifacts","Deepfake detection benchmark stresses time-localization limits"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000195,"raw_usage":{"total_tokens":1336,"prompt_tokens":906,"completion_tokens":430,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":522,"completion_tokens_details":{"reasoning_tokens":341}},"tokens_in":522,"tokens_out":430,"duration_ms":4966,"temperature":1.0,"reasoning_tokens":341,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:40:28.616095+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of, say, 1,000 clips from the fake split of AV-Deepfake1M++, and have human annotators or an independently trained audio-visual synchronization model verify that each clip actually contains the intended semantic change and that the audio and visual manipulations are aligned in time. If a non-trivial fraction of clips are mislabeled or misaligned, the benchmark's labels and the leaderboard numbers derived from them would need to be recomputed.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"VoxCeleb2 is one of the three real-video source datasets."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Xception is the classification baseline whose 55.09 AUC contrasts with the top team's 97.83."}],"review_version":2}