{"id":"8c236439-f3a7-4109-afa4-becddaa03dfd","arxiv_id":"2607.20579","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"Using LipNet, DeepSpeech2, BlazeFace, and ResNet18 features with Random Forest, the authors report 94% accuracy on FakeAVCeleb audio features, but evaluation flaws make that result unsupported.","lead":"This paper combines off-the-shelf lip-reading, speech-recognition, and face-recognition features to detect deepfake news videos, reporting 94% accuracy on FakeAVCeleb with audio features alone. The claim is presented as beating a prior ensemble baseline, but the evaluation has serious gaps that undermine it.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 94% accuracy claim rests on an unspecified split granularity: sliding-window segments from the same source video may straddle train/test, inflating accuracy. The text never rules out this leakage.","rationale":"The reader's weakest assumption is exactly the load-bearing point. The headline is an evaluation result, and the evaluation is only valid if segments from one video do not straddle train/test. The text never specifies split granularity, and the augmentation procedure makes the risk severe: 500 real videos become 50,000 real instances by sliding one-second windows, so each source video contributes many near-duplicate, highly correlated examples. Under an instance-level split, the classifier can exploit source-specific artifacts rather than learning to detect fake versus real content. The omission is internal to the paper; no external benchmark disagreement is needed. Secondary issues—the post-hoc DFDC criticism, absent error bars, no released code, and selecting the audio-only configuration after seeing the audio-visual result—all reinforce the REJECT verdict but are not needed to identify the central vulnerability. I agree with the reader's identification and do not adjust the verdict. If the authors can confirm a video-level split, the concern dissolves; that confirmation is precisely the concrete test.","tokens_in":11184,"tokens_out":6005,"duration_ms":54497,"concrete_test":"Re-run the FakeAVCeleb evaluation with a video-level split: group every augmented segment derived from one source video into the same partition, apply augmentation only within the training partition, then measure test accuracy on held-out source videos. Compare the Random Forest result to 0.9436. If the accuracy drops materially or the authors' split code shows instance-level random splitting, the 94% claim is inflated by leakage. As a secondary check, report per-source-video majority-vote accuracy on the original, un-augmented videos.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is the 94.36% augmented-audio Random Forest result (§V-C, Table III). That number is only meaningful if augmented instances are statistically independent across the 70/15/15 split. §IV-B says each video is preprocessed into 'five or more real instances' and augmentation is done 'by sliding a window over one-second segments of the video'; §IV-C says only 'Seventy percent of the data is used for training, fifteen percent for validation and fifteen percent for testing.' The split granularity is never stated. If the split is applied at the instance level, overlapping one-second segments from the same source video (same speaker, same recording, nearly identical LipNet/DeepSpeech2 features) appear in both training and test, so a Random Forest can memorize source-specific cues and report inflated accuracy. This is not a disagreement with outside consensus; it is an unresolved internal protocol gap. The paper's own counts make the risk concrete: 500 real videos become 50,000 real instances, i.e., roughly 100 highly overlapping segments per source video. No code, split indices, or video-level grouping is provided to rule out leakage.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a deepfake-news detection framework that extracts features from lip movements (LipNet), speech content (DeepSpeech2), and facial appearance (BlazeFace + ResNet18), concatenates them, and classifies the resulting vector with Random Forest, MLP, LSTM, and several other learners. Experiments are reported on FakeAVCeleb, with the claim that augmented audio features give 94% accuracy (Random Forest, Table III), outperforming a state-of-the-art multimodal ensemble baseline [25]. A second dataset, DFDC, is mentioned but only a 52% accuracy is reported in passing. The manuscript stresses a 'semantic-based feature analysis' and efficiency-oriented selection of informative sub-networks, but no such analysis is actually presented.","tokens_in":11436,"tokens_out":4589,"duration_ms":48142,"significance":"If the 94% accuracy figure were valid and reproducible, the proposed pipeline would be a simple, efficient, audio-centric detector for deepfake news videos, and the comparison against the ensemble baseline in Table III would be of interest to the community. The paper's strength is the systematic comparison of many classifiers on the official FakeAVCeleb categories and its clear tabular reporting. However, the central claim currently rests on two unresolved protocol issues: the split granularity for the sliding-window augmentation is not specified, creating a likely data-leakage path, and the audio-only configuration is selected after observing the audio-visual result despite the paper's promise of a principled feature-selection analysis. Without code, split indices, or a corrected experiment, the headline number cannot be taken as evidence for the framework's claimed performance.","major_comments":[{"comment":"The split granularity is not specified. The paper says each video is preprocessed into 'five or more real instances' and augmentation is done 'by sliding a window over one-second segments' (§IV-B), and then 'Seventy percent of the data is used for training, fifteen percent for validation and fifteen percent for testing' (§IV-C). Since 500 real videos become 50,000 real instances (about 100 segments per source video), an instance-level split would place nearly identical, overlapping segments from the same video in both training and test sets, letting Random Forest memorize source-video-specific cues. This would directly inflate the 94.36% accuracy in Table III. The paper must state whether the split was performed at the video level, and if so, provide the split indices or a video-grouped evaluation. Without this, the central claim is not interpretable.","section":"§IV-B and §IV-C"},{"comment":"The audio-only configuration is selected post hoc, contradicting the paper's stated methodology. Section III promises that 'the covariance between them is examined' and that 'only the sub-networks that show the greatest effectiveness are kept.' No covariance analysis, ablation table, or feature-selection results appear in §V. Instead, §V-D reports that the audio-visual Random Forest accuracy is 0.9334 and then states this 'motivates the design choice of emphasizing audio and lip-sync coherence.' This is circular: the model is evaluated on the test set, and the best-performing configuration is then described as the intended design. The authors should provide the promised semantic feature analysis, or clearly state that the audio-only configuration was chosen after observing validation performance, and re-evaluate accordingly.","section":"§V-D and §III"},{"comment":"The manuscript claims robust generalization across diverse datasets, but its only DFDC result is a 52% accuracy, reported without details (§V-A). The introduction claims the goal is 'robust generalization across diverse datasets,' and the abstract/conclusion claim 'robustness and practical potential.' Near-chance performance on DFDC directly contradicts these generalization claims. If DFDC is retained in the narrative, the authors should report the full metric set (precision, recall, F1) for the selected configuration and either temper the generalization claims or explain why DFDC is out of scope. As written, the gap between the claimed robustness and the actual results is too large to ignore.","section":"§V-A and §I"},{"comment":"The assertion that 'the updated dataset contains synthetic faces that are visually indistinguishable from real ones' is used to justify the decision to drop visual features. This is an empirical claim with no supporting human evaluation or quantitative analysis; the single example in Fig. 10 is anecdotal. The lack of improvement from adding ResNet18 features could equally be due to poor face alignment, feature extraction settings, or the leakage issue above. If this explanation is retained, it needs evidence; otherwise the claim should be removed or qualified.","section":"§V-D and Fig. 10"}],"minor_comments":[{"comment":"The augmentation description is inconsistent with the stated class balance: FakeAVCeleb has ~500 real and ~19,500 fake videos, but after preprocessing the paper reports 50,000 real and 100,000 fake instances. The augmentation multiplier (100× for real, ~5.1× for fake) is not explained, and the resulting 1:2 ratio is still imbalanced. Please clarify the exact augmentation procedure and whether the same procedure was applied to both classes.","section":"§IV-B"},{"comment":"Figure 3 lists eight classifiers (Random Forest, MLP, AdaBoost, Decision Trees, SVM, K-Nearest Neighbor, Naïve Bayes, LSTM), but §III-C describes only Random Forest, MLP, and LSTM. The remaining classifiers are never described in the methodology, even though they appear in the results tables. Either add brief descriptions or remove them from the figure.","section":"Fig. 3 and §III-C"},{"comment":"The 'Fine-Tuning' subsection is vague: it says weights are reused from 'existing detection models' and only the final classification layer is specialized, but no details are given about which layers were frozen, which were fine-tuned, or the optimization schedule. Please provide concrete fine-tuning settings or remove this subsection.","section":"§III-D"},{"comment":"Some references are non-archival or inappropriate: DeepSpeech2 is cited to a blog post, BlazeFace to a personal GitHub repository, and ResNet18 is cited to an Alzheimer's-disease classification paper rather than the original ResNet paper. Please replace with the primary sources.","section":"References [26]-[29]"},{"comment":"The comparison with baseline [25] in Table III reports a single accuracy (0.8943), but no information is given about whether this baseline was re-implemented on the same split and with the same augmentation. Without this, the 'surpassing the state-of-the-art' claim in Table V is not fully supported. Please describe the baseline evaluation protocol.","section":"Tables III and V"}],"recommendation":"reject","confidential_remarks":"The split-granularity issue in §IV-B/§IV-C is decisive: if the split is at the instance level, the 94% figure is an artifact of content leakage, and the paper does not rule this out. The post-hoc selection of the audio-only configuration compounds the problem. A resubmission that provides video-level split indices, the promised semantic analysis, and corrected experiments could be reconsidered, but the current manuscript does not meet the evidentiary bar for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nBefore you cite this one, know this: the central 94% accuracy claim on FakeAVCeleb is undermined by an unspecified split granularity. The authors take each video, slide a one-second window over it to create \"five or more\" instances, and then split the data 70/15/15. They never say whether the split is at the video level or the instance level. If it's at the instance level — and the wording suggests it is — overlapping segments from the same source video (same speaker, same recording, nearly identical features) will appear in both training and test sets. The numbers make the risk concrete: 500 real videos become 50,000 real instances, about 100 highly overlapping windows per source. A Random Forest can memorize source-specific cues and inflate accuracy. The paper provides no code, no split indices, and no video-level grouping to rule this out.\n\nWhat is actually new: the specific combination of LipNet, DeepSpeech2, and BlazeFace/ResNet18 features with a Random Forest is a legitimate extension of earlier ensemble and lip-sync detectors. The paper compares eight classifiers across three feature configurations and reports precision, recall, and F1, not just accuracy. The observation that adding visual features slightly hurts performance (audio-only 94.4% vs audio-visual 93.3%) is plausible and worth knowing for anyone working on FakeAVCeleb.\n\nThe soft spots beyond the split are real but secondary. The claimed \"semantic-based feature analysis\" that retains only informative sub-networks is never demonstrated with ablations. The audio-only configuration is selected after observing the audio-visual result, so the headline number is partly post-hoc best-case. The DFDC failure (52%) is explained away with a dataset critique rather than analyzed. No error bars, no seeds, no code.\n\nWho is this for? Someone building a quick baseline or wanting a compact summary of classifier behavior on concatenated features might find the recipe useful. But the evaluation is not trustworthy enough to support the central claim.\n\nI would not send this to a serious referee as is. The missing split specification is basic and fixable. If the authors state the split at video level and release indices, and the accuracy holds, it becomes a modest citable result. Until then, desk-reject with an invitation to resubmit.","headline":"A plausible feature-concatenation pipeline for deepfake detection, but the headline 94% accuracy is unverifiable because the paper never states whether the 70/15/15 split is at the video level or the instance level.","tokens_in":11961,"tokens_out":3091,"would_cite":false,"duration_ms":30073,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A detector that concatenates LipNet, DeepSpeech2, and ResNet18 features and classifies with Random Forest reaches 94 percent accuracy on FakeAVCeleb.","keywords":["deepfake detection","multimodal audio-visual","lip-sync analysis","speech-to-text features","Random Forest","FakeAVCeleb","sliding-window augmentation","deepfake news"],"falsifier":"Rerun the Random Forest training and testing after splitting FakeAVCeleb at the video level, so all one-second segments from a source video stay in one partition. If accuracy falls well below 94%, the reported result was inflated by segment leakage rather than genuine generalization.","tokens_in":11062,"feed_emoji":"🎭","tokens_out":5853,"duration_ms":57628,"temperature":0.7,"pith_summary":"This paper tries to establish that fake news videos can be caught by joining features from lip motion, speech content, and facial appearance and feeding the combined vector to standard classifiers. On the FakeAVCeleb dataset, the proposed Random Forest pipeline reaches 94% accuracy when trained on augmented audio and lip-sync features, beating a published multimodal ensemble baseline that scores 89%. The paper argues that the decisive evidence is in audio and lip-sync coherence: adding face appearance features slightly lowers accuracy to 93%, because current generators create visually convincing faces. The same pipeline drops to 52% on the DFDC dataset, which the authors attribute to that dataset lacking audio-specific labels. If the result holds, a relatively simple and efficient audio-first detector could screen manipulated news content near real time.","feed_headline":"Audio cues detect deepfake news at 94% accuracy","feed_subtitle":"Random Forest on lip and speech features beats an 89% multimodal baseline; adding face cues does not help.","key_machinery":"The mechanism is feature concatenation across three extractors. LipNet encodes cropped mouth regions into lip-motion features; DeepSpeech2 converts one-second Mel spectrograms into speech-content features; and BlazeFace detects and crops faces that ResNet18 encodes into appearance features. The concatenated vector is classified by Random Forest, which the experiments show handles the high-dimensional, non-linear feature space better than MLP, SVM, or LSTM. Sliding-window augmentation over one-second segments is the component credited with the largest accuracy improvement, expanding the minority class and exposing classifiers to varied acoustic conditions.","core_discovery":"The central claim is that concatenating transfer-learned features from LipNet and DeepSpeech2 and classifying them with a Random Forest yields 94% accuracy on FakeAVCeleb, outperforming a state-of-the-art multimodal ensemble baseline at 89%. The authors attribute the largest gain to sliding-window augmentation over one-second segments, which raises audio-only accuracy from 86% to 94% and F1 from 0.8712 to 0.9424. They also claim that visual features have reached diminishing returns: adding ResNet18 face features does not improve accuracy, settling at 93.34%, because modern fake faces are visually indistinguishable. The paper presents this as evidence that deepfake news detection should empha","pith_inferences":["If the same 94% accuracy survives a strict video-level split, audio-first detection could serve as a cheap first-pass filter in news verification, though cross-dataset accuracy would likely drop because FakeAVCeleb is a single benchmark.","The claim that faces are unhelpful may be specific to FakeAVCeleb's generation process; a direct test is to rerun the pipeline on newer deepfake datasets and watch whether the weight of visual features shifts.","Segment-level predictions could be used to localize manipulated portions of a video, turning the classifier into a temporal forensics tool rather than a whole-video labeler.","A stronger evaluation for real-world deployment would be per-video majority voting over one-second predictions under a video-level split."],"forward_implications":["If the 94% figure is correct, audio and lip-sync features alone are sufficient to outperform a multimodal ensemble baseline on FakeAVCeleb, making facial appearance optional for this benchmark.","The 86 to 94 percentage point gain shows that one-second sliding-window augmentation is an effective counter to class imbalance in synthetic media datasets.","The 93% result with visual features implies that as face generators improve, detectors should downweight facial appearance and rely on lip-sync and audio coherence.","The 52% DFDC result implies that datasets without audio-specific labels are unsuitable for training or evaluating audio-visual deepfake detectors.","A Random Forest on a concatenated feature vector is computationally light enough to support near-real-time screening of news video."],"fun_headline_variants":["Lip and audio cues detect deepfake news at 94%","Deepfake detection hits 94% using lip and speech features","Lip-audio model tops deepfake detection at 94% accuracy"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The 94% accuracy assumes that no one-second segment from the same source video appears in both the training and test sets; the paper does not state whether the split is at the video level or the segment level.","fun_headline_variants_meta":{"raw":{"variants":["Lip and audio cues detect deepfake news at 94%","Deepfake detection hits 94% using lip and speech features","Lip-audio model tops deepfake detection at 94% accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00153,"raw_usage":{"total_tokens":5982,"prompt_tokens":786,"completion_tokens":5196,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":530,"completion_tokens_details":{"reasoning_tokens":5139}},"tokens_in":530,"tokens_out":5196,"duration_ms":40679,"temperature":1.0,"reasoning_tokens":5139,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T10:57:29.471711+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the Random Forest training and testing after splitting FakeAVCeleb at the video level, so all one-second segments from a source video stay in one partition. If accuracy falls well below 94%, the reported result was inflated by segment leakage rather than genuine generalization.","supporting_citations":[],"review_version":1}