{"id":"d60523b6-d43f-4017-907c-a937b674060e","arxiv_id":"2507.16596","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A weakly-supervised method localizes forged segments in deepfake videos using only video-level labels, achieving near-fully-supervised accuracy on some metrics.","lead":"Researchers built a system that finds the exact moments a video has been tampered with, using only a whole-video label (fake or real) instead of frame-by-frame annotations. It combines video and audio signals to spot where they diverge, and reports near-fully-supervised accuracy on some benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The deviation perceiving loss (Eq. 9-11) assumes forged videos have larger adjacent-feature deviation than genuine videos, but this prior is untested and can fail on scene cuts and smooth forgeries; it is the main temporal supervision.","rationale":"The reader's weakest assumption is the same one I find most load-bearing: L_dp's forged-vs-genuine deviation prior. The central claim—that MDP localizes forged timestamps from video-level labels—depends on this prior supplying temporally relevant signal, since the classification loss alone is label-level. I agree with the reader's CONDITIONAL verdict. The missing inference details and overclaiming are secondary; the deviation assumption is the place where the argument is least secure. My concrete test is empirical and can be run with the public code: measure how well d discriminates forged from genuine videos, stratified by scene cuts and forgery type. If the assumption fails in a stratum, the loss can actively hurt localization there, which would require either revising the loss or tempering the claim. If the test passes, the concern is resolved and the framework's plausibility is strengthened. This does not change the reader's verdict, so I recommend UNCHANGED.","tokens_in":14888,"tokens_out":4585,"duration_ms":53146,"concrete_test":"On the released code/features, compute d (Eq. 9-10) for a held-out set of LAV-DF and AV-Deepfake1M before training and measure AUC of d against the video-level label, stratified by (a) genuine videos with scene cuts vs. without, and (b) visual-only, audio-only, and audio-visual forgeries. If AUC is near chance or below 0.6 in any stratum, the sign of Eq. 11 is wrong for that stratum; re-run the Table 3 ablation with L_dp removed and report AP on the same strata to see whether the loss is actually helping or hurting localization.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.4's L_dp is the only training signal with temporal content: the classification loss in Eq. 1 is video-level. Eq. 9-11 assumes d = sigma(sum_t E[(x_t-x_{t+1})^2]) is larger for forged than genuine videos, so minimizing Eq. 11 enlarges d for forged videos and shrinks it for genuine ones. This assumption is load-bearing because the ablation (Table 3) attributes roughly +14.7 AP to L_dp, and no experiment measures d's label-discriminative power. It is not secured by the cited refs [18,42], which study video-level deepfake detection and image forensics, not adjacent-boundary deviation statistics of LAV-DF / AV-Deepfake1M. Genuine videos contain scene cuts and audio transitions, which produce large adjacent-feature deviations, while many forged manipulations (face reenactment, inpainting, speech conversion) are temporally smooth. In those cases Eq. 11 pushes the model to suppress genuine boundaries or amplify smooth forged boundaries, moving the FAS in the wrong direction. Because d is a global scalar, the loss cannot point to the correct start/end timestamps even when the global correlation holds; it can only bias the whole activation sequence. The paper's central localization claim therefore rests on an unverified distributional prior.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MDP, a weakly-supervised temporal forgery localization (WS-TFL) framework that trains with only video-level genuine/forged labels. It extracts visual and audio features with pretrained backbones, aligns them to T segments, applies a temporal-property-preserving cross-modal attention in a probabilistic embedding space, and produces a temporal forgery activation sequence (FAS) from a classifier (Sections 3.2–3.3). Training combines a video-level classification loss with a deviation perceiving loss (Eqs. 9–12) that enlarges a scalar measure of adjacent-segment feature deviation for forged videos and shrinks it for genuine videos. Experiments on LAV-DF and AV-Deepfake1M report AP and AR; ablations on LAV-DF attribute gains to both the cross-modal attention and the deviation loss, and a comparison of L1/L2/MSE deviation measures is included (Table 4). The abstract and introduction claim results comparable to fully-supervised approaches in several metrics.","tokens_in":15201,"tokens_out":8910,"duration_ms":85735,"significance":"If the empirical results hold, MDP would provide a simple, well-specified baseline for WS-TFL, a task with clear practical relevance; its gains over transferred WS-TAL baselines are large (LAV-DF AP@0.5: 84.57 vs. 18.79/8.79 for FuSTAL/CoLA), both proposed components are ablated, the deviation measure function is explored in Table 4, and the code is released. The significance is tempered, though, because 'comparable to fully-supervised' holds only at lenient IoU on one dataset or at AP@0.1/0.2 on the other, and because the deviation prior—the only temporally non-uniform training signal—is verified only indirectly through ablation rather than by direct measurement.","major_comments":[{"comment":"The deviation perceiving loss is the only training objective with temporal granularity—L_cls receives a uniform gradient over segments through Eq. (1)—yet L_dp supervises a single scalar d aggregated over all T adjacent pairs, so by construction it can only bias the entire activation sequence and cannot point to the correct boundary; the ablation attributes about +11.7 average AP to it. The load-bearing premise that forged videos have larger adjacent-segment deviation than genuine videos is not directly verified on LAV-DF or AV-Deepfake1M; the cited refs [18,42] are video-level detectors, and genuine scene cuts or temporally smooth forgeries are natural counterexamples. Please add a diagnostic of the prior (e.g., distribution of d and of per-boundary deviations on held-out sets, overlap of top-deviance boundaries with GT boundaries) and, if possible, an ablation in which L_dp is replaced by per-segment or boundary-level pseudo-supervision.","section":"Section 3.4, Eqs. (9)–(11) and Table 3"},{"comment":"The core cross-modal attention formula is dimensionally inconsistent as written: bR is defined as a 1×T row vector (column sum of R) and V is T×d, so bR^T·V is not a valid matrix product; presumably the intended operation is row-wise scaling of each audio segment by its relevance r_t, i.e., ATT_v[t] = r_t·V[t]. Please rewrite with explicit tensor shapes, and define cM_v and cM_a, which appear in Eq. (8) but are never introduced.","section":"Section 3.3, Eqs. (6)–(7) and Algorithm 1"},{"comment":"The inference procedure is underspecified: no threshold is given for converting p_t into binary forged/background segment decisions, no rule is given for merging consecutive forged segments, and the AR evaluation (proposals of 20/10/5/2) requires a proposal-generation mechanism that is not described. These omissions block reproduction of the reported AP/AR numbers; please specify the threshold (or the selection procedure) and the proposal generation/merging algorithm.","section":"Sections 3.2 and 3.5"},{"comment":"The claim of 'comparable results to fully-supervised approaches' is only supportable at lenient IoU thresholds on LAV-DF (AP@0.5, AP@0.75) and at AP@0.1/AP@0.2 on AV-Deepfake1M; on AV-Deepfake1M the average AR is 8.86 versus 87.88–88.62 for fully-supervised methods and AP@0.5 drops to 22.39 versus 99.25+ for fully-supervised methods. Please temper the abstract and contributions accordingly and discuss the AR collapse on long-duration videos, since this is the localization metric where the weak supervision is most strained.","section":"Abstract and Section 4.2, Table 2"}],"minor_comments":[{"comment":"L_cls is referenced but never defined; please give the explicit loss form (presumably cross-entropy between ŷ and y).","section":"Section 3.5, Eq. (12)"},{"comment":"The segment count T and the choice of φ=0.5 are not justified, and no sensitivity analysis over φ is reported; please add one or state why these values are sufficient.","section":"Section 4.1"},{"comment":"The text says the attention measures relevance 'between each video segment v_t and audio segment a_t' but then defines a full T×T relevance matrix; please reconcile the wording.","section":"Section 3.3"},{"comment":"The sum over t=1..T includes a boundary pair (x_T, x_{T+1}); please state how the last segment is handled.","section":"Eq. (9)"},{"comment":"Several passages contain garbled or duplicated tokens (e.g., 'VVisual features', repeated labels next to arrows, symbol fragments such as 'dp', 'cls'); please check the camera-ready rendering.","section":"Figure 2 and inline math"},{"comment":"It is stated that only real and audio-visual forgery samples are used; please report the number of videos retained per dataset and how the forging masks are obtained, since this affects comparability with the fully-supervised baselines.","section":"Section 4.1"},{"comment":"The absence of CoLA and FuSTAL rows is explained only in the body text; please add a table note to make this visible.","section":"Table 2"},{"comment":"No error bars or multiple-seed results are reported; please state the number of runs and any variance, especially for the ablation deltas in Table 3.","section":"Experimental protocol"}],"recommendation":"major_revision","confidential_remarks":"The assumption underlying L_dp in Section 3.4 is supported by refs [18] and [42], both of which share an author with this manuscript; the support is plausible, but independent verification (e.g., a direct deviation-statistics diagnostic) would substantially increase confidence. The paper is formatted as an ACM MM proceedings paper; for a journal version, multiple seeds, a threshold analysis, and ideally a third dataset would be expected. The overall contribution is interesting and the method is clearly specified apart from the points raised in the major comments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things up front. This is a real, if modest, contribution: the first weakly-supervised audio-visual temporal forgery localization framework, with a clean design and a public code repo. The column-sum cross-modal attention is a sensible way to keep temporal structure when fusing modalities, and the ablation on LAV-DF shows both components help. But the paper overreaches with 'comparable to fully-supervised.' On LAV-DF, MDP hits 0.58 AP at IoU 0.95 versus UMMAFormer's 53.89; on AV-Deepfake1M, average AR is 8.86 versus 87.88. The claim only holds at loose IoU and on the easier dataset.\n\nThe bigger soft spot is the deviation perceiving loss. It assumes forged videos have more deviation between adjacent segments than genuine ones. That prior is load-bearing — the loss contributes ~12-14 AP in the ablation — but it is never tested directly. The cited references are from the same group and target video-level detection, not boundary statistics on these datasets. Genuine videos with scene cuts, or smooth forgeries like inpainting or face reenactment, would push the loss in the wrong direction. And because d is a single scalar per video, the loss can only bias the whole activation sequence; it can't point to correct start and end timestamps. It may still be a useful regularizer — the ablation suggests it helps on LAV-DF — but the paper doesn't demonstrate that the assumption holds, and the stress-test concern is legitimate.\n\nThere are also smaller reproducibility gaps: no inference threshold is specified, no error bars on any number, and phi is fixed to 0.5 without sensitivity analysis. The AV-Deepfake1M comparison is only against fully-supervised methods because the weakly-supervised baselines couldn't localize at all, which weakens the relative claim.\n\nWho is this for? Researchers working on deepfake detection or weakly-supervised temporal localization, as a first baseline for WS-TFL. It deserves a serious referee, but the authors need to temper the abstract, add error bars and threshold details, and test the deviation assumption directly — e.g., plot the distribution of d for forged versus genuine videos on both datasets. If the assumption collapses, the method still works but the contribution is mostly the feature interaction, not the deviation loss.","headline":"Genuinely new WS-TFL framework with helpful ablations, but the deviation loss is an untested assumption and the fully-supervised comparison claim only holds at loose IoU.","tokens_in":15685,"tokens_out":3554,"would_cite":false,"duration_ms":33553,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A weakly-supervised framework localizes forged video segments with only video-level labels and approaches fully supervised accuracy.","keywords":["deepfake detection","temporal forgery localization","weakly-supervised learning","multimodal forgery detection","cross-modal attention","deviation perceiving loss","video forensics","audio-visual deepfake"],"falsifier":"Train MDP on a set where genuine videos contain frequent scene cuts and forged segments are produced by a smooth model that preserves temporal coherence across the splice boundary; if the deviation-perceiving loss still separates forged from genuine videos by pushing adjacent-segment deviation apart, the core assumption survives, and if it does not, the loss is actively misleading the model.","tokens_in":1557,"feed_emoji":"🎬","tokens_out":2043,"duration_ms":92172,"temperature":0.7,"pith_summary":"This paper aims to prove that temporal forgery localization—finding the start and end timestamps of manipulated segments inside a deepfake video—can be done with only video-level fake/real labels, without frame-level timestamp annotations. The proposed MDP framework reads visual and audio features, aligns them in time, fuses them with cross-modal attention that preserves temporal ordering, and adds a deviation-perceiving loss that widens the gap between adjacent segments in forged videos while shrinking it in genuine ones. On the LAV-DF benchmark, MDP reaches 84.57 AP@0.5 and 75.91 AP@0.75 using only weak labels, closing much of the gap to fully supervised methods, and on AV-Deepfake1M it localizes very short forged segments in long videos. A sympathetic reader would take the paper's contribution to be evidence that weak supervision, combined with multimodal deviation, is a viable route to scalable deepfake forensics.","feed_headline":"Weak supervision finds forged video timestamps","feed_subtitle":"The method needs only whole-video labels and closes most of the gap to frame-supervised detectors.","key_machinery":"Two mechanisms carry the argument. The first is the multimodal interaction module, built around a temporal-property-preserving cross-modal attention: visual and audio features are mapped into a probabilistic embedding space by an MLP with LayerNorm, and attention is computed between the modalities in a way that sums the relevance matrix over columns before weighting values, so the temporal index is never collapsed. The second is the extensible deviation-perceiving loss $\\mathcal{L}_{dp}$: the temporal deviation $d$ is the normalized sum over adjacent segments of a deviation function $f$ (mean squared error in the main experiments), and the loss is binary cross-entropy between $d$ and the video-level label, enlarging adjacent-segment deviation for forged videos and reducing it for genuine ones. Together they convert weak video-level labels into segment-level supervision for timestamp localization.","core_discovery":"The central claim is that inter-modality and inter-segment deviations carry enough forgery signal to support weakly-supervised temporal forgery localization. The framework extracts frame-level visual and audio features, tokenizes them, pools both modalities to a common temporal length, then computes cross-modal attention in a probabilistic embedding space: query from one modality, key and value from the other, with the relevance matrix summed over columns so temporal position survives the attention step. The enhanced visual and audio features are concatenated into a comprehensive video feature, from which a temporal forgery activation sequence is produced. Training uses only the video-level label via a classification loss plus the deviation-perceiving loss, which measures the mean squared error between adjacent segment features, normalizes it, and applies binary cross-entropy against the video label. The result is a model that predicts forged segment timestamps at inference and, in several evaluation metrics on LAV-DF and AV-Deepfake1M, performs close to fully supervised temporal forgery localization methods.","pith_inferences":["One consequence the authors leave implicit: the deviation idea is not tied to audio-visual video, and the same loss could be applied to partially manipulated speech, image strips, or any sequence where forgery introduces discontinuity.","A direct stress test the paper does not run is genuine footage with scene cuts or rapid camera motion—these would inflate adjacent-segment deviation and could make the loss push genuine videos in the wrong direction.","The paper reports results for real and audio-visual forgery samples only; audio-only and visual-only forgeries are untested, so the relative contribution of each modality to deviation remains unknown.","Because deviation is computed on concatenated multimodal features, decomposing the loss per modality could reveal which stream carries the forgery trace, a diagnostic extension the framework supports."],"forward_implications":["Temporal forgery localization becomes trainable on large video collections that only carry whole-video labels, removing the main annotation bottleneck of frame-level supervision.","The temporal-preserving cross-modal attention is a reusable fusion pattern for any multimodal video task where order in time must survive attention.","The deviation-perceiving loss is plug-compatible with different deviation measures; the paper shows L1, L2, and MSE behave differently, so better measures can extend the same framework.","The reported AP@0.95 on LAV-DF remains far below fully supervised methods, so the next concrete step is refining boundary precision rather than coarse segment finding.","On AV-Deepfake1M, MDP localized forged segments as short as roughly 0.3 seconds inside a 23-second video using only video-level labels, indicating the method scales to long videos."],"supporting_citations":[{"why":"Supplies the LAV-DF content-driven multimodal dataset and the AP/AR evaluation protocol used in the main experiments.","marker":"[3]"},{"why":"Supplies the AV-Deepfake1M large-scale multimodal dataset that tests generalization to long videos.","marker":"[2]"},{"why":"UMMAFormer is a fully-supervised multimodal temporal forgery localization baseline whose results MDP is compared against.","marker":"[38]"},{"why":"MFMS is a fully-supervised deepfake detection and localization baseline used for comparison.","marker":"[39]"},{"why":"ActionFormer is a fully-supervised temporal localization baseline used on both datasets.","marker":"[36]"},{"why":"CoLA is a weakly-supervised temporal action localization baseline adapted for comparison.","marker":"[35]"},{"why":"FuSTAL is a weakly-supervised temporal action localization baseline adapted for comparison.","marker":"[10]"},{"why":"Supplies the probabilistic embedding space idea for weakly-supervised temporal action localization that the cross-modal attention builds on.","marker":"[16]"},{"why":"Supplies prior evidence that deviation between adjacent segments is larger in forged videos, grounding the deviation-perceiving loss assumption.","marker":"[18]"},{"why":"Defines the temporal forgery localization task and establishes audio-visual dissonance as a forgery cue that the framework builds on.","marker":"[8]"}],"fun_headline_variants":["Weak labels pinpoint forged video segments via multimodal deviation","Deviation between senses exposes fake video timing","Multimodal deviation finds forged timestamps with weak supervision","Weakly supervised forgery localization using audio-visual mismatch","Cross-modal deviation pinpoints fake video intervals"],"cache_read_input_tokens":17920,"weakest_assumption_plain":"The load-bearing premise is that forged splicing makes adjacent video segments measurably less smooth than genuine footage, and that this smoothness gap shows up in the pooled visual and audio features and can be supervised by a single whole-video label.","fun_headline_variants_meta":{"raw":{"variants":["Weak labels pinpoint forged video segments via multimodal deviation","Deviation between senses exposes fake video timing","Multimodal deviation finds forged timestamps with weak supervision","Weakly supervised forgery localization using audio-visual mismatch","Cross-modal deviation pinpoints fake video intervals"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000215,"raw_usage":{"total_tokens":1424,"prompt_tokens":939,"completion_tokens":485,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":555,"completion_tokens_details":{"reasoning_tokens":412}},"tokens_in":555,"tokens_out":485,"duration_ms":6068,"temperature":1.0,"reasoning_tokens":412,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:05:48.212936+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train MDP on a set where genuine videos contain frequent scene cuts and forged segments are produced by a smooth model that preserves temporal coherence across the splice boundary; if the deviation-perceiving loss still separates forged from genuine videos by pushing adjacent-segment deviation apart, the core assumption survives, and if it does not, the loss is actively misleading the model.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"ActionFormer is a fully-supervised temporal localization baseline used on both datasets."},{"cited_title":"International Journal of Computer Vision (2024), 1–15","cited_arxiv_id":null,"evidence_quote":"CoLA is a weakly-supervised temporal action localization baseline adapted for comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the LAV-DF content-driven multimodal dataset and the AP/AR evaluation protocol used in the main experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the AV-Deepfake1M large-scale multimodal dataset that tests generalization to long videos."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"UMMAFormer is a fully-supervised multimodal temporal forgery localization baseline whose results MDP is compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"MFMS is a fully-supervised deepfake detection and localization baseline used for comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the probabilistic embedding space idea for weakly-supervised temporal action localization that the cross-modal attention builds on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies prior evidence that deviation between adjacent segments is larger in forged videos, grounding the deviation-perceiving loss assumption."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the temporal forgery localization task and establishes audio-visual dissonance as a forgery cue that the framework builds on."}],"review_version":1}