{"id":"5d1e82f1-eb7c-4cf1-a801-a9bde46a533a","arxiv_id":"2412.00175","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":2,"one_line_summary":"A leading-silence artifact in FakeAVCeleb and AV-Deepfake1M lets a trivial classifier separate real from fake, and an unsupervised alignment method trained only on real data avoids relying on this shortcut.","lead":"Two widely used audio-visual deepfake datasets have a hidden flaw: fake videos begin with a short silence that alone reveals the fake class in over 98% of test cases. The paper shows supervised models exploit this artifact and presents an unsupervised method trained only on real data that is robust to it.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Leading-silence artifact interpretation is load-bearing but only indirectly supported; if the silence is intrinsic to the TTS/lip-sync tools, trimming discards a real cue.","rationale":"The reader's verdict identifies the same weakest assumption. The paper's bias discovery is well-supported empirically: a simple silence feature reaches >98% AUC, trimming degrades MDS dramatically, and the bias is absent in three other datasets. However, the causal mechanism remains speculative, and the fine-grained FVRA result (62.4% AUC) shows the silence is not exclusively tied to fake audio. The proposed concrete test would separate 'tool artifact' from 'dataset artifact' directly. Until that is resolved, the paper's strong wording ('spurious feature', 'unwanted artifact') is too definitive, and the conditional verdict is appropriate. No further adjustment is needed.","tokens_in":14283,"tokens_out":13230,"duration_ms":119913,"concrete_test":"Generate fake videos with the same pipelines used in FakeAVCeleb and AV-Deepfake1M (e.g., SV2TTS/TalkLip, VITS/YourTTS) but on a source dataset that is not VoxCeleb2, such as LRS3 or a self-recorded corpus. Measure the leading silence distribution of the generated fakes. If the silence consistently appears regardless of source dataset, it is a property of the generation tools, not the dataset construction; if it appears only when following the exact FakeAVCeleb/AV1M packaging pipeline, the artifact interpretation is supported and the trimming evaluation is valid.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that prior models exploit a spurious leading-silence artifact depends on the assumption that this silence is a dataset-construction artifact rather than a genuine property of the fake-generation pipelines. The paper's own Section 3.2 admits: 'we speculate that this happens when the audio may be slightly shorter than the video counterpart.' The supporting evidence is indirect: the bias is absent in LAV-DF, AVLips, DFDC, and weaker on FakeAVCeleb's FVRA subset (62.4% AUC vs 100% for RVFA). However, these observations do not establish the cause. If the TTS/lip-sync systems used (SV2TTS, VITS, YourTTS, TalkLip) intrinsically produce a short leading silence when combined with video, then the silence is a valid discriminative cue for deepfakes produced by those tools. In that case, trimming it from the evaluation removes a useful signal, and the conclusion that prior results 'partly reflect this artifact' overstates the spuriousness. The proposed unsupervised method would remain robust, but the paper's evaluation paradigm and its claim to have 'circumvented' a shortcut would be misleading.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a leading-silence bias in two widely used audio-visual deepfake datasets, FakeAVCeleb and AV-Deepfake1M: in the fully fake (FVFA) subsets, fake videos begin with roughly 25-30 ms of silence, and a simple threshold-based silence-duration classifier reaches above 98% AUC. The authors then show that trimming this leading silence degrades some supervised models, most notably MDS on AV-Deepfake1M, and propose AVH-Align, an unsupervised method trained only on real videos by aligning AV-HuBERT audio and video features. AVH-Align is reported to be robust to trimming and to achieve 85.24% AUC on the official AV-Deepfake1M test set, surpassing the visual-only baselines reported there. The paper concludes that prior supervised results partly reflect this spurious cue and that unsupervised real-data training is a useful evaluation and modeling paradigm.","tokens_in":14504,"tokens_out":7739,"duration_ms":70034,"significance":"If the bias is as general as claimed, the paper makes a valuable benchmarking contribution: it identifies a concrete, easily checked artifact, shows that it inflates several published results, and proposes a simple robustness test (trimming the leading silence). The silence analysis is strengthened by threshold-robustness experiments and by an independent maximum-amplitude measure, and the trimming experiments convincingly show that some supervised models rely on the cue. The official test-set evaluation of AVH-Align is a strong point, since it avoids self-reported validation-set scores. The proposed method is simple and reproducible, with code released. The main limitations are that the headline claim is overbroad relative to the actual experimental scope, and that the causal interpretation of the silence as a dataset-construction artifact is only speculative.","major_comments":[{"comment":"The headline claim that FakeAVCeleb and AV-Deepfake1M suffer from a leading-silence artifact, with a simple classifier reaching over 98% AUC, is computed only on the RVRA versus FVFA subsets, as stated in Section 3.2 and Table 1. On FakeAVCeleb's FVRA split (fake video, real audio), the leading-silence AUC is only 62.4%, and Section 3.2 reports that LAV-DF, AVLips, and DFDC do not exhibit the bias. The abstract's unqualified sentence \"Fake videos start with a very brief moment of silence\" is therefore overbroad. This is load-bearing because the claim that prior audio-visual results \"partly reflect this artifact\" is only established for fakes with synthesized audio. Please qualify the claims to the RVFA/FVFA fully-fake or synthesized-audio subsets, and state explicitly which fake splits are used in the full-dataset experiments in Tables 2 and 3.","section":"Abstract and Section 3.2"},{"comment":"The paper's only causal explanation for the leading silence is speculative: \"we speculate that this happens when the audio may be slightly shorter than the video counterpart.\" The interpretation of the silence as a dataset-construction artifact is load-bearing for the conclusion that trimming removes a spurious cue and that prior results are overestimated. If the TTS or lip-sync tools used to create the fakes (SV2TTS, VITS, YourTTS, TalkLip) intrinsically produce a short leading silence, then the silence is a genuine property of fakes produced by those tools, and the trimmed evaluation would discard a valid discriminative signal. The indirect evidence (absence in LAV-DF, AVLips, DFDC, and weaker bias on FVRA) does not establish the causal origin. Please either provide direct evidence about the generator outputs before dataset assembly, or rephrase the conclusions conditionally, making clear that \"spurious\" means \"not shared across deepfake generation pipelines\" rather than \"not caused by the fake-generation tools.\"","section":"Section 3.2, \"Why do fake samples have a leading silence?\""}],"minor_comments":[{"comment":"Equation (3) writes p(v_i|a_j) but the surrounding text and Eq. (2) use p(v_i|a_i); the subscript in Eq. (3) should presumably be a_i.","section":"Section 4.1, Eq. (3)"},{"comment":"The text says the AUC of AVH-Align/sup \"degrades by 16.9%\" when going from 100.0 to 83.1 AUC; since AUC is already a percentage, this should be expressed as 16.9 percentage points to avoid ambiguity with a relative drop.","section":"Section 4.2"},{"comment":"There is a typo in the discussion of Rajan et al.: \"reconstructring\" should be \"reconstructing\".","section":"Section 5"},{"comment":"The text says AVH-Align is \"not impacted\" by leading silence, but the AV-Deepfake1M AUC drops from 85.9 to 83.5 after trimming; please describe this as a small but nonzero effect rather than no effect.","section":"Section 4.3, Table 2"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a computer-vision benchmarking venue and the central finding is likely reproducible. I recommend major revision primarily to tighten the scope of the silence-bias claim to the subsets where it is actually measured, and to address the causal-origin concern, which is central to the 'spurious artifact' narrative. The official test evaluation is a notable strength and should be retained."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. First, the paper actually finds something: FakeAVCeleb and AV-Deepfake1M have a leading-silence artifact, and a trivial amplitude threshold separates real from fake at over 98% AUC. Second, the trimming experiments are the right check, and they show real models lean on it—MDS collapses from 99.2 to 54.9 AUC on AV-Deepfake1M. That alone is worth a serious referee.\n\nWhat's new: prior silence-bias work was on ASVspoof (audio-only), and the direction was reversed. The paper's controlled comparison—FVRA (fake video, real audio) shows 62.4% AUC while RVFA shows 100%—makes a strong case that the silence lives in the synthesized audio, not in the manipulation per se. The unsupervised AVH-Align method is a reasonable combination of AV-HuBERT features plus a learned alignment score trained on real data. It hits 85.24% AUC on the official held-out test, best among methods not using the bias. Code is released.\n\nSoft spots, in proportion. The abstract overstates: 'fake videos start with silence' is false for the FVRA split, which the paper itself reports; that claim should be qualified. The cause of the silence is speculative, and the paper says so. The stress-test worry—that the silence might be intrinsic to TTS and therefore a genuine cue—is plausible in the abstract, but the evidence points against it: the bias is absent in LAV-DF, AVLips, and DFDC, and it's weak when the audio is real. That is indirect but decent. Still, the paper does not prove the silence is artificial; it's an interpretation. I'd call that a moderate caveat, not a flaw. Also, there are no error bars anywhere—single runs—which matters some for the method comparisons. The VoxCeleb2 pretraining leakage is partially addressed by the AVLips evaluation; not airtight but honest.\n\nWho this is for: anyone benchmarking on FakeAVCeleb or AV-Deepfake1M needs to know about this bias, and anyone building robust audio-visual detectors should read the unsupervised alignment idea. It deserves a proper peer review; my recommendation is to engage with it, accepting that revisions should focus on rewriting the overbroad claims and tightening the causality discussion.","headline":"A real, previously missed shortcut: fake clips in two standard audio-visual deepfake datasets start with a brief silence, and the paper shows prior benchmark results partly ride on it.","tokens_in":15042,"tokens_out":3053,"would_cite":true,"duration_ms":27558,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fake videos in two major audio-video deepfake benchmarks begin with a telltale silence that lets a trivial classifier score over 98% AUC—and removing that silence collapses several published detectors.","keywords":["deepfake detection","audio-visual","leading silence","dataset bias","spurious features","unsupervised learning","self-supervised alignment","AV-HuBERT"],"falsifier":"Collect or create deepfakes using the same generators (e.g., TalkLip, VITS) but with audio and video streams length-matched before packaging and check whether fake samples still begin with 25–30 ms of silence; if they do, the silence is intrinsic to the generation process, not the dataset construction, and the artifact interpretation fails.","tokens_in":14094,"feed_emoji":"🎬","tokens_out":6393,"duration_ms":52601,"temperature":0.7,"pith_summary":"This paper claims that two widely used audio-video deepfake benchmarks—FakeAVCeleb and AV-Deepfake1M—carry an unintended shortcut: fake videos begin with about 25–30 ms of silence, while real videos start with noise. A trivial classifier that measures only the duration of this leading silence separates real from fake samples with over 98% AUC on both datasets. The authors show that several published audio and audio-visual detectors have been exploiting this feature, and that removing the silence degrades them sharply, for example dropping the MDS method from 99.2% to 54.9% AUC on AV-Deepfake1M. To avoid such dataset-specific biases, they propose training only on real videos: AVH-Align learns an audio-visual alignment score from self-supervised features and uses misalignment to flag fakes, without ever seeing a fake during training. If this holds, previous benchmark results partly reflect an artifact, and the field needs evaluation protocols and learning paradigms that are robust to such shortcuts.","feed_headline":"A 30 ms silence gap lets a trivial detector hit 98% AUC","feed_subtitle":"Trimming the silence collapses top models, so the authors train on real videos only to dodge the shortcut.","key_machinery":"The central mechanism is the leading-silence duration: the time until the audio magnitude first exceeds a small threshold ($5\\cdot10^{-4}$), a feature that alone yields AUC above 98% on both datasets. The proposed countermeasure, AVH-Align, uses a pretrained AV-HuBERT model to extract frame-level audio and visual features, then trains a four-layer MLP alignment network $\\Phi$ on real videos only, using an InfoNCE-style contrastive loss that maximizes the match between each audio frame and its corresponding visual frame while downweighting neighbouring frames. At inference, the negative of the per-frame alignment score, pooled with log-sum-exp, gives the fakeness score. The paper also uses a trimming protocol—discarding the leading silence rounded up to the video's frame period—to evaluate whether a method has latched onto the artifact.","core_discovery":"The paper's central discovery is that the leading silence is a spurious feature that pervades two major audio-visual deepfake datasets and inflates the measured performance of existing detectors. On the paper's own terms, the finding is that fake videos in FakeAVCeleb and AV-Deepfake1M start with a brief moment of quiet audio—around 25–30 ms—whereas real videos begin with noise, so a classifier built solely on this feature reaches over 98% AUC. The authors further show that state-of-the-art audio and audio-visual models rely on this silence: after trimming it, RawNet2 loses 11.8 points of AUC on AV-Deepfake1M and MDS collapses from 99.2% to 54.9% AUC. They propose that the remedy is to abandon supervised training on fake data altogether and instead learn on real videos only, and they demonstrate AVH-Align, which aligns self-supervised AV-HuBERT audio-visual features with a frame-level contrastive network and scores fakeness by the degree of audio-visual misalignment. AVH-Align is unaffected by the silence shortcut and achieves the highest reported AUC (85.24%) among methods that do not exploit the bias on the official AV-Deepfake1M test set.","pith_inferences":["A natural next step is to apply the leading-silence probe to other audio-visual datasets, since the same packaging pipeline (generated audio appended to real video) could introduce the artifact elsewhere.","The alignment-score signal that AVH-Align produces is frame-level, and the paper's 77.7% frame-level AUC suggests the approach could be extended from detection to localization of manipulated segments.","The trim-and-remeasure protocol could become a general evaluation discipline: for any suspected shortcut feature, compare performance with and without it to quantify reliance.","If the silence arises because generated audio is shorter than the video, dataset curators could fix it by length-matching rather than trimming, which would also preserve synchronization."],"forward_implications":["Reported scores on FakeAVCeleb and AV-Deepfake1M should be treated as untrustworthy unless accompanied by results on silence-trimmed data.","Existing deep-learning detectors that trained on these datasets may generalize much worse than their benchmarks suggest; a silence probe is a cheap robustness check.","Training on real data only, as AVH-Align does, removes the incentive to exploit dataset asymmetries and remains competitive with supervised approaches.","Audio-only methods hold up well after trimming, indicating the audio stream carries strong, underused signal for deepfake detection.","The official AV-Deepfake1M test set shares the silence bias, so leaderboard entries without a bias check should be interpreted cautiously."],"supporting_citations":[{"why":"Provides AV-Deepfake1M, the larger benchmark where the silence bias and official test results are reported.","marker":"[6]"},{"why":"Provides FakeAVCeleb, the other affected benchmark with full-video manipulations.","marker":"[23]"},{"why":"Supplies the self-supervised AV-HuBERT audio-visual features that AVH-Align aligns.","marker":"[49]"},{"why":"Prior unsupervised audio-visual anomaly detection baseline that is robust to the silence and the conceptual starting point for real-data training.","marker":"[17]"},{"why":"Audio-only RawNet2 baseline whose performance drop after trimming quantifies the bias's impact.","marker":"[53]"},{"why":"Audio-visual MDS baseline that collapses from 99.2% to 54.9% AUC after trimming.","marker":"[10]"},{"why":"Prior demonstration that silence duration leaks labels in ASVspoof, the precedent for silence as an artifact.","marker":"[35]"},{"why":"InfoNCE contrastive loss used as the training objective for aligning audio and visual frames.","marker":"[54]"},{"why":"Supervised and unsupervised baseline compared against AVH-Align on the deepfake detection task.","marker":"[30]"}],"fun_headline_variants":["Silence gap alone gives 98% AUC on deepfake datasets","Fake videos start with silence: trivial detector gets 98% AUC","Removing silence collapses state-of-the-art deepfake detectors","Self-supervised alignment avoids deepfake dataset silence bias","Leading silence is the hidden shortcut in deepfake detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The leading silence is a dataset-construction artifact rather than a real property of deepfakes; if genuine manipulated videos commonly start with silence, trimming it would discard a true signal and the evaluation would be misleading.","fun_headline_variants_meta":{"raw":{"variants":["Silence gap alone gives 98% AUC on deepfake datasets","Fake videos start with silence: trivial detector gets 98% AUC","Removing silence collapses state-of-the-art deepfake detectors","Self-supervised alignment avoids deepfake dataset silence bias","Leading silence is the hidden shortcut in deepfake detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000306,"raw_usage":{"total_tokens":1764,"prompt_tokens":967,"completion_tokens":797,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":583,"completion_tokens_details":{"reasoning_tokens":713}},"tokens_in":583,"tokens_out":797,"duration_ms":7327,"temperature":1.0,"reasoning_tokens":713,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:39:11.111787+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect or create deepfakes using the same generators (e.g., TalkLip, VITS) but with audio and video streams length-matched before packaging and check whether fake samples still begin with 25–30 ms of silence; if they do, the silence is intrinsic to the generation process, not the dataset construction, and the artifact interpretation fails.","supporting_citations":[{"cited_title":"A V-Deepfake1M: A large-scale LLM-driven audio-visual deepfake dataset, 2024","cited_arxiv_id":null,"evidence_quote":"Provides AV-Deepfake1M, the larger benchmark where the silence bias and official test results are reported."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides FakeAVCeleb, the other affected benchmark with full-video manipulations."},{"cited_title":"Learning audio-visual speech representation by masked multimodal cluster prediction","cited_arxiv_id":null,"evidence_quote":"Supplies the self-supervised AV-HuBERT audio-visual features that AVH-Align aligns."},{"cited_title":"Self- supervised video forensics by audio-visual anomaly detec- tion","cited_arxiv_id":null,"evidence_quote":"Prior unsupervised audio-visual anomaly detection baseline that is robust to the silence and the conceptual starting point for real-data training."},{"cited_title":"End-to-end anti-spoofing with RawNet2","cited_arxiv_id":null,"evidence_quote":"Audio-only RawNet2 baseline whose performance drop after trimming quantifies the bias's impact."},{"cited_title":"Not made for each other-audio- visual dissonance-based deepfake detection and localization","cited_arxiv_id":null,"evidence_quote":"Audio-visual MDS baseline that collapses from 99.2% to 54.9% AUC after trimming."},{"cited_title":"SpeechForensics: Audio-visual speech representation learn- ing for face forgery detection","cited_arxiv_id":null,"evidence_quote":"Supervised and unsupervised baseline compared against AVH-Align on the deepfake detection task."}],"review_version":1}