{"id":"48c79e4f-8cdb-4409-aaaa-e2943a5c92da","arxiv_id":"2412.13656","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"MSTF, a 143k-video talking face dataset built from 22 forgery techniques, and a global-local audio-visual coherence detector, outperform prior deepfake detectors on this benchmark.","lead":"This paper introduces MSTF, a large dataset of over 143,000 talking-face videos built with 22 forgery techniques, and a detection framework called GLCF that analyzes global and local coherence across video and audio. It reports accuracy gains over prior deepfake detectors on this new benchmark.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MSTF test accuracy is reported as a single number, but the test set is dominated by videos from the same generators used in training; the small held-out IP_LAP/LSR2 slices are not reported separately, so the claimed generalizability to real-world talking face forgeries is unsubstantiated.","rationale":"The reader's weakest assumption is that MSTF is representative of real-world talking face forgeries; my stress-test identifies the same load-bearing concern. The paper's own description of the test set ('a small portion' of LSR2 and IP_LAP, rest from the training generators) means the single reported accuracy is dominated by same-generator samples, and no per-subset results are given. This directly affects both main contributions: the dataset's value as a benchmark and the method's claimed superiority. The concern is concrete and testable: decomposing the test accuracy by generator would settle whether the 88.49% reflects genuine generalization or memorization of generator-specific artifacts. I do not see an internal inconsistency in the method's equations that would invalidate the framework; the ablation study (Table 4) shows each module contributes to within-dataset performance. However, the lack of dataset/code release and the absence of significance tests further support keeping the verdict conditional. Therefore, I recommend no change to the reader's CONDITIONAL verdict.","tokens_in":11857,"tokens_out":4768,"duration_ms":41635,"concrete_test":"Obtain the MSTF test set (or request per-video predictions from the authors) and compute accuracy separately for: (1) test videos generated by the same methods used in training; (2) the held-out IP_LAP-generated slice; and (3) the held-out LSR2 real-video slice, reporting the number of videos in each slice. If GLCF's accuracy on slices (2) and (3) is not significantly above chance or is below the best baseline on those slices, then the aggregate 88.49% result is an artifact of same-generator test samples and the generalization claim fails. A complementary check is to retrain GLCF and the top baseline with only the same-generator slices held out and test on IP_LAP/LSR2, comparing accuracy and AUC with confidence intervals.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claims are (i) MSTF is the first large-scale multi-scenario talking face dataset and (ii) GLCF outperforms state-of-the-art deepfake detectors. Both rest on the premise that MSTF is representative of real-world talking face forgeries. This premise is not established by the reported evaluation. In 'Dataset Distribution', the test set is described as containing a 'small portion' of LSR2 real video and a 'small portion' of IP_LAP-generated video, with the remainder drawn from the same generation methods used in training. The paper never reports the per-subset accuracy for these held-out slices. Consequently, the headline MSTF accuracy of 88.49% (Table 2) could be driven almost entirely by same-generator test videos, while performance on the held-out slices—the only evidence of generalization to new generators and to real video—could be near chance. The cross-dataset experiment (Table 3, 61.94% when training on FakeAVCeleb and testing on MSTF) likewise does not separate same-generator from held-out slices. Without this breakdown, the claimed 'superiority' over baselines is not shown to transfer to the practical scenarios the dataset is intended to simulate. The 'first' claim is also weakened by prior multimodal talking-face datasets such as FakeAVCeleb and LAV-DF, and no independently collected real-world TFG examples are included, so external validity remains untested.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MSTF, a large-scale audio-visual talking face forgery dataset containing 143,754 videos generated with 22 forgery techniques across 11 generation scenarios, and proposes GLCF, a detection framework with three main components: RSFDM for global inter-frame coherence, DCTAM for fine-grained discrepancy capture and multi-granularity temporal aggregation, and V-AFM for local audio-visual coherence analysis, augmented by frequency-domain features. Experiments compare GLCF with eight baselines on MSTF, FF++, FakeAVCeleb, and DFDC, reporting accuracy improvements (e.g., 88.49% vs. 86.12% MiNet on MSTF) and cross-dataset results (61.94% on FakeAVCeleb-to-MSTF), along with an ablation study showing positive contributions from each module.","tokens_in":12129,"tokens_out":3102,"duration_ms":29707,"significance":"If the claims are properly supported, MSTF would be a valuable community resource: it is larger than earlier multimodal talking-face forgery datasets in terms of forgery techniques (22) and explicitly simulates multiple generation and semantic scenarios, addressing a real gap in TFG detection research. GLCF is also a reasonable architecture: the three modules target complementary cues (global spatiotemporal smoothness, pixel-level temporal discrepancies, and audiovisual coherence), and the ablation and cross-dataset experiments suggest the components are not redundant. The paper also deserves credit for constructing a held-out evaluation slice with unseen generators (IP_LAP) and real videos (LSR2), which is the right idea for testing generalization. However, the current empirical reporting does not establish the central generalization claim: the held-out slices are not analyzed separately, no statistical significance or variance is reported, and several baselines were re-implemented without official code. These are fixable with additional experiments and reporting rather than fundamental flaws.","major_comments":[{"comment":"The headline MSTF accuracy of 88.49% (Table 2) is not informative about generalization because the test set is dominated by videos from the same generators used in training, with only a 'small portion' of LSR2 real videos and IP_LAP-generated videos held out. The paper must report accuracy separately for the held-out LSR2 and IP_LAP slices, and ideally a per-generator breakdown of the test set. Without this, the claim that GLCF 'shows superiority' for detecting unknown talking face forgeries is unsubstantiated, since the reported margin over MiNet (2.37%) could be driven by same-generator test videos while performance on unseen generators or real videos could be near chance.","section":"Dataset Distribution; Table 2"},{"comment":"All accuracy values in Tables 2, 3, and 4 are single numbers with no variance, confidence intervals, or statistical significance tests. Given that three baselines (RECCE, Joint A-V, and AV-D2-DWF) were re-implemented 'based on the original documentation' without official code, the comparison is not yet statistically grounded. The authors should report results over multiple random seeds (at least three) with mean and standard deviation, and perform a paired significance test where possible. This is particularly important because the claimed advantage on DFDC is only 0.13% over F3Net.","section":"Experiments: Performance Comparisons; Tables 2-4"},{"comment":"The inter-dataset experiment trains on FakeAVCeleb and tests on MSTF, but the reported MSTF accuracy of 61.94% again aggregates over the same-generator and held-out slices. The only claim the table can currently support is that GLCF transfers reasonably to the overall MSTF test distribution; it does not show transfer to unseen generators. Additional experiments testing on the LSR2 and IP_LAP slices alone, and ideally testing in the reverse direction (train on MSTF, test on FakeAVCeleb or DFDC), would substantiate the claim that the method 'captures common forgery traces ... rather than relying on overfitting.'","section":"Cross-Dataset Comparisons; Table 3"},{"comment":"The claim that MSTF is 'the first large-scale multi-scenario talking face dataset' is too strong given that Table 1 itself lists FakeAVCeleb (20,000 videos, audio/video) and LAV-DF (136,304 videos, audio/video), both of which contain talking-face-style generation and are described as multimodal. The paper's dismissal of these as 'limited talking face generation methods and video quantity' is contradicted by LAV-DF's size. The novelty should be reframed precisely: e.g., MSTF includes the largest number of distinct forgery techniques (22) and explicitly covers 11 generation scenarios. As written, the 'first' claim invites a dispute that weakens the paper's contribution.","section":"Talking Face Dataset: Comparison with Existing Datasets; Table 1"}],"minor_comments":[{"comment":"Typo: 'Comparision' should be 'Comparison'. Table 1 headings contain spacing errors such as 'FakeA VCeleb' and 'LA V-DF', and Table 2's caption reads 'in terms of a accuracy'.","section":"Talking Face Dataset: Comparison with Existing Datasets"},{"comment":"The text says the dataset covers 'more than 20 semantic scenarios' and then states that combining with generation scenarios yields 'about 40 different scenarios'; the relationship between semantic scenarios and the final count should be clarified so the reader can reproduce the arithmetic.","section":"Dataset Distribution"},{"comment":"The test-set distribution figure should show the exact number or percentage of videos from LSR2 and IP_LAP; a small pie chart without numeric labels is not sufficient to evaluate the claim that these slices are 'small portions'.","section":"Figure 3"},{"comment":"In Eq. (2), the notation 'FQ × FK' is ambiguous; since the Attention Score Matrix A is described as a similarity matrix, '×' likely denotes matrix multiplication after reshaping, and this should be stated explicitly. Additionally, the definition of the variance activation function Φ (how the k-th smallest variance is selected and applied) would benefit from a concrete formula.","section":"Methodology: Discrepancy Capture-Time Frame Aggregation Module, Eq. (2)"},{"comment":"Eq. (3) contains 'Up1×2' and 'Down1×2' without spaces; these denote upsampling and downsampling operations, but the interpolation method (e.g., bilinear or nearest) is not specified. Also, the text says 'Fh is similar, except that the convolution kernel is transposed' but the exact transposition is not defined.","section":"Methodology: Discrepancy Capture-Time Frame Aggregation Module, Eq. (3)"},{"comment":"No statement is provided about the availability of MSTF or the code for GLCF. For a dataset paper, a URL or a commitment to release the dataset and evaluation protocol is expected; without it, the community cannot reproduce or build on the benchmark.","section":"Experiments: Experimental Settings"},{"comment":"The Limitations section (a single sentence about compressed videos) is too terse. The analysis should at least mention the lack of real-world TFG examples in the test set and the use of face crops (MTCNN) which may discard contextual cues relevant to real-world deployment.","section":"Limitations"}],"recommendation":"major_revision","confidential_remarks":"The paper's core direction is sound and the dataset has clear potential value, but the evaluation as presented overclaims generalization. The authors are likely able to fix the issues by adding per-subset accuracy (especially LSR2/IP_LAP), variance and significance reporting, and by softening the 'first' claim. I would be inclined to accept after such a revision. The lack of dataset release details is a concern for a dataset paper and should be addressed in the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this one. First, it is probably the largest talking-face forgery dataset assembled so far, with 22 audio/video manipulation methods and 11 generation scenarios, and the proposed detector GLCF wins on that benchmark by a decent margin (88.49% vs. 86.12% for the best baseline). Second, the evaluation has a blind spot: the MSTF test set is mostly videos from the same generators used in training, with only a 'small portion' of real LSR2 and IP_LAP-generated videos as held-out slices. The paper does not report accuracy on those slices separately, so we have no idea whether the gain over baselines holds for genuinely unseen generators or real video. That is the main thing I'd want fixed before trusting the generalizability claims.\n\nWhat's genuinely new: the dataset scale and breadth of forgery methods, and the GLCF architecture—RSFDM, DCTAM, and V-AFM are a sensible combination of inter-frame difference, attention-based variance activation, and audio-visual fusion. The ablation in Table 4 is clean: each module contributes, V-AFM is the biggest. Cross-dataset results (train on FakeAVCeleb, test on MSTF) put GLCF best at 61.94%, though that number is low in absolute terms and still no error bars.\n\nSoft spots, in proportion: (1) The 'first large-scale multi-scenario talking face dataset' claim is a bit generous—FakeAVCeleb and LAV-DF already have audio-visual talking-face content, though smaller and fewer manipulation types. (2) No dataset or code release is mentioned; for a dataset paper that's a real problem. (3) All results are single ACC numbers, no variance or significance testing, and some baselines were re-implemented without official code. (4) The limitation section only mentions compression by social media, sidestepping the bigger issue of whether MSTF resembles real-world abuse.\n\nThe central argument—that GLCF works better on MSTF than existing detectors—is supported by the tables. The generalization to real-world scenarios is not established, but that's an addressable weakness, not a fatal flaw.\n\nThis paper should get a serious referee. For a dataset contribution, I'd want the dataset released (or at least a clear availability plan), per-slice test results for the held-out LSR2/IP_LAP portions, and ideally error bars. If I were advising the editor, I'd send it to review with a request for these revisions.","headline":"Large TFG dataset and a competent detector, but the test-set composition hides whether gains generalize to unseen generators.","tokens_in":12685,"tokens_out":2969,"would_cite":false,"duration_ms":25584,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Talking-face forgeries can be detected by checking global and local coherence across frames and audio, on a new 143,754-video benchmark where the proposed method reaches 88.49% accuracy.","keywords":["talking face generation detection","deepfake detection","multimodal forgery detection","audiovisual coherence","temporal coherence","video forensics dataset","cross-attention fusion"],"falsifier":"Train GLCF on MSTF and test it on a collection of talking-face videos produced by commercial or newly released generators not included in MSTF, at realistic re-encoded bitrates; if accuracy collapses toward chance while in-distribution accuracy stays near 88%, the claim that the method detects talking-face forgeries in practice is refuted.","tokens_in":11643,"feed_emoji":"🎭","tokens_out":7572,"duration_ms":61931,"temperature":0.7,"pith_summary":"Talking-face generation can turn a single image and a script into a realistic video, and detecting such forgeries is harder than detecting classic deepfakes because the generated videos are cleaner and better synchronized. The paper's claim is that a dedicated large-scale dataset plus a detection framework built on coherence analysis can close much of that gap. It introduces MSTF, the first large-scale multi-scenario talking-face dataset with 22 forgery techniques, and GLCF, which checks global temporal smoothness, subtle inter-frame discrepancies, and local audiovisual consistency. If the claim holds, forgery detection no longer needs to rely on visible spatial artifacts and can move to temporal and cross-modal cues that survive high-quality generation.","feed_headline":"Coherence checks catch talking-face fakes at 88.5%","feed_subtitle":"A 143,754-video benchmark with 22 forgery methods, plus a multimodal detector that beats prior work by 2.4 points.","key_machinery":"The load-bearing mechanism is the global-local multimodal coherence decomposition of a talking-face video. It is operationalized by three modules: RSFDM, which computes inter-frame differences and uses an attention-like weighting to emphasize moving regions such as the mouth; DCTAM, which builds a pixel-level temporal similarity matrix, computes variance along the time axis to activate anomalous regions, and then aggregates spatial features from neighboring frames at multiple receptive-field granularities to align audio and visual time scales; and V-AFM, a multi-head cross-attention module that scores coherence between audio features and the aggregated visual features over local windows. A frequency-statistics branch supplies complementary spectral clues. The claimed effect is that incoherences too subtle or too slow for any single detector become visible when these scales are combined.","core_discovery":"The paper's central claim is that talking-face forgeries can be reliably detected by analyzing coherence at two scales: global temporal coherence across the whole frame sequence and local audiovisual coherence within short segments, rather than by looking only for spatial or frequency artifacts that work for older deepfakes. To support this, the authors introduce MSTF, a 143,754-video dataset covering 22 audio and video forgery techniques across 11 generation scenarios and roughly 40 semantic combinations, and GLCF, a detector with three cooperating modules: RSFDM attends to motion regions through inter-frame differences, DCTAM uses variance-activated attention over a temporal similarity matrix to expose subtle per-frame discrepancies and aggregates spatial context at multiple granularities, and V-AFM applies multi-head cross-attention between audio and visual features. On MSTF the full system reaches 88.49% accuracy, 2.37 points above the best previous detector, and when trained on FakeAVCeleb and tested on MSTF it reaches 61.94%, the highest cross-dataset score in the comparison.","pith_inferences":["Implicit in the results but not stated: the cross-dataset accuracy of 61.94% is modest in absolute terms, so unseen talking-face generators remain a hard open problem and MSTF's held-out IP_LAP and LSR2 slices only begin to measure it.","An inference the paper does not draw: because the MSTF test set draws mostly from the same five generators seen in training, the in-distribution accuracy of 88.49% should not be read as real-world performance; an external benchmark built from commercial talking-face tools would be the natural stress test.","The variance-activation mechanism in DCTAM selects the k-th smallest variance as an activation threshold; tuning k per dataset may matter more than the paper reports, and a sensitivity analysis would clarify the mechanism's robustness.","The paper names social-media compression as future work; since real forgeries are re-encoded, compression-robust variants of RSFDM and DCTAM are the immediate next step that the paper itself points toward."],"forward_implications":["Detection of talking-face forgeries needs dedicated benchmarks: methods tuned on traditional deepfake datasets such as FF++ and DFDC transfer poorly to MSTF.","Global temporal coherence, measured through frame-transition smoothness and subtle inter-frame discrepancies, carries signal that frame-level spatial forgery traces miss.","Audiovisual coherence is the largest single contributor: removing V-AFM drops accuracy by 8.27 points in the ablation study.","The same framework transfers to ordinary deepfake datasets, reaching 98.92% accuracy on FakeAVCeleb and 92.58% on DFDC, suggesting shared forgery cues.","Cross-dataset training on FakeAVCeleb and testing on MSTF still beats prior methods, indicating some generalization to unseen talking-face generators."],"supporting_citations":[{"why":"Supplies FakeAVCeleb, the multimodal deepfake dataset used as a source of forged audio and as the cross-dataset evaluation partner for MSTF.","marker":"Khalid et al. 2021"},{"why":"SadTalker generates the referenced talking-face videos in several MSTF scenarios, providing the training and test data for the global-coherence modules.","marker":"Zhang et al. 2023"},{"why":"VideoRetalking generates the lip-sync editing scenarios in MSTF, defining part of the audiovisual coherence signal that V-AFM is designed to catch.","marker":"Cheng et al. 2022"},{"why":"HyperLips is one of the no-reference generators used to build MSTF videos and contributes to the diversity of forgery techniques.","marker":"Chen et al. 2023"},{"why":"Diff2Lip is another no-reference generator used in MSTF, adding diffusion-based forgeries to the benchmark.","marker":"Mukhopadhyay et al. 2024"},{"why":"IP_LAP is held out from training and used to generate a small slice of harder testing cases, supporting the generalization claim.","marker":"Zhong et al. 2023"},{"why":"The Local Frequency Statistic Method supplies the frequency-domain branch of GLCF, which the paper says complements the coherence modules.","marker":"Qian et al. 2020"},{"why":"MiNet is the strongest prior baseline on MSTF at 86.12% accuracy, the comparison point that GLCF surpasses by 2.37 points.","marker":"Ba et al. 2024"},{"why":"DFDC is used as an external deepfake dataset for generalization evaluation, showing the method transfers beyond talking-face forgeries.","marker":"Dolhansky et al. 2020"},{"why":"FF++ is used as the conventional deepfake benchmark, and the paper uses it to argue that MSTF is a harder detection task.","marker":"Rossler et al. 2019"}],"fun_headline_variants":["Coherence at two scales catches talking-face fakes at 88.5%","Global and local coherence unmask talking-face forgeries","Largest talking-face fake dataset plus coherence detector","Talking-face deepfakes exposed by global-local coherence analysis","Coherence checks beat prior deepfake detectors on new 143K benchmark"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The MSTF test set is largely produced by the same five tools used in training, with only small slices from two unseen sources, so accuracy on MSTF is assumed to predict performance on forgeries made by tools that are not in the benchmark.","fun_headline_variants_meta":{"raw":{"variants":["Coherence at two scales catches talking-face fakes at 88.5%","Global and local coherence unmask talking-face forgeries","Largest talking-face fake dataset plus coherence detector","Talking-face deepfakes exposed by global-local coherence analysis","Coherence checks beat prior deepfake detectors on new 143K benchmark"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000616,"raw_usage":{"total_tokens":2893,"prompt_tokens":1011,"completion_tokens":1882,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":627,"completion_tokens_details":{"reasoning_tokens":1795}},"tokens_in":627,"tokens_out":1882,"duration_ms":14049,"temperature":1.0,"reasoning_tokens":1795,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:55:19.429759+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train GLCF on MSTF and test it on a collection of talking-face videos produced by commercial or newly released generators not included in MSTF, at realistic re-encoded bitrates; if accuracy collapses toward chance while in-distribution accuracy stays near 88%, the claim that the method detects talking-face forgeries in practice is refuted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"SadTalker generates the referenced talking-face videos in several MSTF scenarios, providing the training and test data for the global-coherence modules."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"VideoRetalking generates the lip-sync editing scenarios in MSTF, defining part of the audiovisual coherence signal that V-AFM is designed to catch."},{"cited_title":"HyperLips: Hyper Control Lips with High Resolution Decoder for Talking Face Generation","cited_arxiv_id":"2310.05720","evidence_quote":"HyperLips is one of the no-reference generators used to build MSTF videos and contributes to the diversity of forgery techniques."},{"cited_title":"T.; and Shrivastava, A","cited_arxiv_id":null,"evidence_quote":"Diff2Lip is another no-reference generator used in MSTF, adding diffusion-based forgeries to the benchmark."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"IP_LAP is held out from training and used to generate a small slice of harder testing cases, supporting the generalization claim."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The Local Frequency Statistic Method supplies the frequency-domain branch of GLCF, which the paper says complements the coherence modules."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"FF++ is used as the conventional deepfake benchmark, and the paper uses it to argue that MSTF is a harder detection task."}],"review_version":1}