{"id":"fc188ae1-ed6a-4d7e-822b-da5b6167aacd","arxiv_id":"2506.03179","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"Vid-SME computes Sharma-Mittal entropy differences between natural and reversed video frame sequences to infer training membership in video understanding LLMs, but its effectiveness is confounded by member/non-member dataset distribution shifts.","lead":"This paper introduces Vid-SME, a membership inference attack that uses adaptive entropy and reversed-frame comparisons to detect whether a video was in a video-LLM's training data. The method shows high detection rates on some model/dataset pairs, but its cleanest evaluation shows only near-chance performance.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The strong results in Table 2 come from cross-dataset member/non-member pairs; the only matched-distribution setting (NExT-QA) gives AUC≈0.54, so the headline attack may be detecting dataset differences rather than training membership.","rationale":"The reader's weakest assumption is the correct load-bearing concern. Membership is defined relative to training status, so the only valid test of the central claim is one where members and non-members come from the same distribution and differ only in membership. The paper provides exactly one such test, NExT-QA train versus test, and there Vid-SME is near chance. The high numbers that motivate the paper's conclusions all come from cross-dataset pairs, which are exactly the settings Appendix D acknowledges are imperfect approximations. The construction itself—Sharma-Mittal entropy with adaptive q/r and reversed frames—is internally coherent, and the self-training details are disclosed, so the issue is not the method's derivation or implementation but the validity of the evaluation as evidence for membership inference. I therefore agree with the reader and see no reason to change the REJECT verdict.","tokens_in":20834,"tokens_out":5658,"duration_ms":54380,"concrete_test":"Replace the cross-dataset non-member sets with held-out videos from the same dataset as the members. For example, randomly partition CinePile into a training half (used to instruction-tune Video-XL-CinePile-7B) and a held-out half, then run Vid-SME with the held-out half as non-members, keeping all attack settings fixed, and recompute AUC and TPR@5% FPR for the rows in Table 2. If performance falls to the NExT-QA range (AUC ≈ 0.54, TPR@5% ≈ 0.10), the cross-dataset results were confounded; if AUC remains above 0.8, the membership signal is real. A faster, less expensive alternative is to reuse the existing Video-XL-CinePile-7B model and simply replace its MLVU non-member set with a held-out CinePile split.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that Vid-SME yields robust membership scores is not supported by the evaluation design. In the only matched-distribution setting, Video-XL-NExT-QA-7B with NExT-QA train as members and NExT-QA test as non-members, all Vid-SME variants achieve AUC between 0.519 and 0.548 and TPR@5% FPR between 0.030 and 0.106—barely above chance. The strong claims come from evaluations where members and non-members are drawn from different datasets: CinePile vs MLVU, Video-Instruct-100K vs Video-XL captioning, and Video-XL vs VDC. In these settings, Vid-SME variants reach AUC 0.84 and TPR@5% FPR 0.42. Because the pairs differ in dataset identity, task format, and content, the natural-versus-reversed entropy gap may simply separate datasets rather than training status. Appendix D concedes that available non-member sets only approximate the training distribution and 'may not fully reflect the severity of the privacy risk.' That concession is exactly the load-bearing gap: the headline numbers do not establish membership inference, only that members and non-members come from different distributions, and the adaptive q/r parameters and K selection cannot remedy a confound in the evaluation itself.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes Vid-SME, a membership inference attack (MIA) against video understanding LLMs (VULLMs). The method computes Sharma–Mittal entropy over the next-token distributions of video-token slices, adaptively sets the entropy parameters q and r from video motion and illumination statistics, and forms a membership score from the minimum-K% difference between entropy values for natural and temporally-reversed frame sequences. The authors evaluate Vid-SME on three self-trained VULLMs and two open-source VULLMs, comparing it with perplexity, Min-K%, Rényi-based, and other metric-based baselines, and report consistent improvements, especially in TPR@5% FPR. Code is released.","tokens_in":21159,"tokens_out":6152,"duration_ms":57735,"significance":"The paper introduces a novel angle to video MIA—temporal reversal combined with an adaptive entropy measure—and contributes three self-trained VULLMs as an evaluation benchmark. If the empirical claims were supported, this would be a useful first dedicated video MIA method. However, the central claim is not backed by the evaluation: the only matched-distribution setting yields near-chance performance, and the strong headline numbers come from cross-dataset member/non-member pairs. The contribution is therefore currently a proposal with an unvalidated empirical basis rather than a demonstrated attack.","major_comments":[{"comment":"The evaluation confounds membership with dataset identity. The only matched-distribution setting is Video-XL-NExT-QA-7B, where members are the NExT-QA training split and non-members are the NExT-QA test split; there all Vid-SME variants obtain AUC 0.519–0.548 and TPR@5% FPR 0.030–0.106, barely above chance. The high-scoring results in Table 2 (e.g., AUC 0.840 for Video-XL-CinePile-7B with Mean Vid-SME) are obtained using non-members from a different dataset (MLVU), and the LongVA-Caption and LLaVA-NeXT-Video settings similarly pair members with non-members from Video-XL, VDC, or Video-Instruct-100K. Because these pairs differ in dataset source, task format, and content distribution, the natural-vs-reversed entropy gap may simply separate datasets rather than training membership. Appendix D acknowledges that available non-member sets only approximate the training distribution and 'may not fully reflect the severity of the privacy risk'; this is the load-bearing gap and invalidates the paper's central claim that Vid-SME derives robust membership scores.","section":"§5, Tables 1 and 2"},{"comment":"The attack's free parameters are selected post hoc. Equation (2) sets beta1=1.0 and beta2=0.1 with no derivation or sensitivity analysis, and Equation (4) introduces K%, with the paper reporting six variants (Mean, Min_0%, Min_5%, Min_30%, Min_60%, Min_90%) and highlighting the best-performing one per setting. Without a principled procedure for choosing beta1, beta2, and K, the reported numbers do not describe a fixed attack, and the claimed consistent improvements may reflect favorable parameter selection rather than intrinsic robustness of the method.","section":"§4.1, Eq. (2); §4.2, Eq. (4)"},{"comment":"The claim that Vid-SME 'consistently achieves the best performance under all settings' is not supported by Table 2. For LongVA-Caption-7B, Mean Vid-SME has AUC 0.496 and TPR@5% FPR 0.039, essentially tied with several baselines; for LLaVA-NeXT-Video-7B, most Vid-SME variants have AUC below 0.60. The only large gains (e.g., AUC 0.840 and TPR@5% FPR 0.420 for Video-XL-CinePile-7B) occur in the cross-dataset settings described above, so the headline improvement is an artifact of the evaluation confound rather than a consistent property of the method.","section":"§5.1, Table 2"}],"minor_comments":[{"comment":"The word 'dicussed' should be 'discussed'.","section":"§2.2"},{"comment":"The word 'algin' should be 'align'.","section":"§4.1"},{"comment":"The caption of Table 4 reads 'Performance comparison on different instructions' but the table reports results under different dataset scales; the caption should be corrected.","section":"§5.2, Table 4"},{"comment":"The figure contains the typo 'calculaiton'; it should be 'calculation'.","section":"Figure 1"},{"comment":"The figures label non-member examples as 'non-memebr'; this typo should be fixed.","section":"Appendix D and Figures 11–14"}],"recommendation":"reject","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What should you know: this paper introduces the first membership inference attack (MIA) specifically for video understanding LLMs, and the core idea—comparing Sharma–Mittal entropy of natural versus temporally-reversed frame sequences—is genuinely new and plausible. They also self-train three VULLMs and release code, which is more than most privacy papers do. The ablations show both the reversal trick and the adaptive q/r parameters contribute, so the method is not vacuous.\n\nThe problem is the evaluation. The only setting where members and non-members come from the same dataset is NExT-QA train vs. test, and there all Vid-SME variants land between 0.52 and 0.55 AUC with TPR@5% FPR around 0.03–0.11. That is essentially chance. The headline numbers (AUC up to 0.84, TPR@5% FPR 0.42) come from cross-dataset pairs: CinePile vs. MLVU, Video-Instruct-100K vs. Video-XL, and so on. Those videos differ in task, content, style, and distribution, so the entropy gap can just be separating datasets rather than training status. The authors concede exactly this in Appendix D: the non-member sets only \"approximate\" the training distribution and \"may not fully reflect\" the privacy risk. That concession is the load-bearing gap. No amount of tuning beta1, beta2, or K% can fix a confound in the experimental design.\n\nOther soft spots are secondary. The baselines are weak—many sit near 0.5 AUC—so beating them is not strong evidence. The K parameter is swept and the best variant highlighted, and the adaptive parameters are heuristics without a derivation of optimality. These are real but minor compared to the evaluation issue.\n\nIf the authors redo the evaluation with proper matched-distribution protocols—for example, holding out a split from the same dataset for each self-trained model—the method might show real membership signal. As it stands, the paper cannot support its central claim, but the idea is worth engaging and the failure mode is instructive. I would not cite this as evidence of a working video MIA.\n\nRecommendation: send to peer review, but expect the reviews to demand a new evaluation. The problem is important and the approach is novel; the evidence just does not match the conclusions yet.","headline":"First video MIA with a novel natural-vs-reversed entropy idea, but the only matched-distribution evaluation gives AUC ≈ 0.54; the strong results are cross-dataset and likely detect dataset identity, not membership.","tokens_in":21601,"tokens_out":2854,"would_cite":false,"duration_ms":28810,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that videos used to train video-understanding language models can be detected from output logits by comparing the model's prediction entropy on natural versus temporally reversed frame sequences.","keywords":["membership inference attack","video understanding large language model","Sharma-Mittal entropy","temporal reversal","data privacy","multimodal LLM","low false positive rate"],"falsifier":"Build a VULLM on a single dataset and split it randomly into a training set and a held-out set, then run Vid-SME with members from the training split and non-members from the held-out split; if TPR@5% FPR drops to near the baseline level, the cross-dataset results were detecting dataset identity, not membership. A second check is to replace reversed order with a random temporal shuffle: if the entropy gap persists, the mechanism is not specifically the natural-versus-reversed asymmetry.","tokens_in":20639,"feed_emoji":"🎥","tokens_out":6825,"duration_ms":74128,"temperature":0.7,"pith_summary":"Vid-SME claims that a video's membership in the instruction-tuning data of a video-understanding large language model can be detected from the model's output logits alone. The attack computes Sharma-Mittal entropy over next-token probabilities for both the natural frame order and the temporally reversed frame order, adaptively choosing the entropy parameters from each video's motion and illumination statistics. For videos the model saw during training, natural-order frames produce more confident predictions and reversed frames produce a larger entropy jump, so the natural-minus-reversed entropy gap becomes the membership score. The authors report that this score consistently outperforms existing token-level MIAs on five VULLMs, including three models trained for the benchmark and two open-sourced models, with the largest gains at the low-false-positive rates that matter for real auditing.","feed_headline":"Reversing frames reveals which videos trained a video LLM","feed_subtitle":"New attack compares Sharma-Mittal entropy on normal and reversed clips, beating prior methods at low false-positive rates.","key_machinery":"The load-bearing object is Sharma-Mittal entropy, $$S_{q,r}(p)=\\frac{1}{1-r}\\left(\\left(\\sum_i p_i^q\\right)^{\\frac{1-r}{1-q}}-1\\right),$$ a two-parameter entropy that generalizes Shannon, Rényi, and Tsallis entropies. Vid-SME chooses $q$ and $r$ per video from normalized optical-flow variance and illumination variation, then computes $S_{q,r}$ on the video-token slices of the next-token distributions for the natural and reversed frame sequences. The final score is the average of the smallest $K\\%$ of per-position differences $\\Delta S = S_{\\text{nat}} - S_{\\text{rev}}$, which the paper identifies as the membership signal.","core_discovery":"The discovery is the temporal asymmetry in memorization: a video-understanding LLM that has seen a video during instruction tuning predicts next tokens with higher confidence when frames are in their original order, and suffers a sharper confidence drop when those same frames are reversed; a model that has never seen the video shows a smaller natural-versus-reversed entropy difference. Vid-SME turns this gap into a membership detector by computing $S_{q,r}$ with per-video adaptive $q$ and $r$, taking the element-wise difference between natural and reversed entropy slices, and aggregating the smallest $K\\%$ of differences as the score. The paper argues that existing text/image MIA metrics fail on video because they ignore both inter-frame variation and the effect of the number of sampled frames, whereas Vid-SME is designed to capture both. Across five target models, the method reports higher AUC, accuracy, and TPR@5% FPR than all baselines in nearly every setting.","pith_inferences":["An implication the paper leaves implicit: the same natural-versus-reversed entropy comparison could serve as a diagnostic for temporal memorization in other sequential modalities, such as audio or 3D event streams, though the paper only tests video.","A caveat worth testing: the cross-dataset non-member sets used in several experiments (for example CinePile members against MLVU non-members) could allow the score to separate dataset identity rather than membership; a same-source held-out evaluation would settle whether the signal is true membership leakage.","If the signal is real, a natural defensive response would be to randomize frame order during instruction tuning or to add an entropy-gap regularizer, but the paper does not propose defenses."],"forward_implications":["An auditor with query access to a VULLM can decide whether a specific video was in the instruction-tuning set without knowing the model weights.","The natural-versus-reversed entropy gap is a more reliable signal at low false-positive rates than perplexity or Rényi-based text/image MIA baselines, where baseline TPR@5% FPR is often near zero.","The attack stays effective as the number of sampled frames changes, as the target dataset grows, and under motion-blur and brightness corruption, because the adaptive $q/r$ parameters track video statistics.","Because the instruction context has only a limited influence on the score, the signal comes primarily from the video frames themselves, not from the wording of the question."],"supporting_citations":[{"why":"defines the Sharma-Mittal entropy formula that Vid-SME's score is built on.","marker":"[42]"},{"why":"introduces the MaxRényi/ModRényi token-level MIA baselines and the grey-box attack setting for MLLMs that Vid-SME compares against.","marker":"[23]"},{"why":"supplies the Min-K% token-probability metric used as a baseline and as the aggregation pattern that Vid-SME adapts to entropy differences.","marker":"[43]"},{"why":"establishes the membership inference task, the threshold-based detector formulation, and the TPR@ low-FPR evaluation metric.","marker":"[44]"},{"why":"provides the Video-XL architecture and training recipe used to build two of the self-trained target models.","marker":"[46]"},{"why":"provides the LongVA architecture and training recipe used for the LongVA-Caption-7B target model.","marker":"[65]"},{"why":"releases the open-sourced LLaVA-NeXT-Video-7B and 34B models used as target models.","marker":"[67]"}],"fun_headline_variants":["Frame reversal uncovers video LLM training membership","Video LLM training data inferred by reversing frames","Membership attack on video LLMs via temporal reversal","Reversing video order reveals training data to attackers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the comparison videos labeled as non-members differ from the member videos only in whether they were in the training set; in several experiments the two pools come from different datasets, so the reported separation could partly reflect dataset or task differences rather than membership, and the authors acknowledge this in Appendix D.","fun_headline_variants_meta":{"raw":{"variants":["Frame reversal uncovers video LLM training membership","Video LLM training data inferred by reversing frames","Membership attack on video LLMs via temporal reversal","Reversing video order reveals training data to attackers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000228,"raw_usage":{"total_tokens":1498,"prompt_tokens":991,"completion_tokens":507,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":607,"completion_tokens_details":{"reasoning_tokens":446}},"tokens_in":607,"tokens_out":507,"duration_ms":5981,"temperature":1.0,"reasoning_tokens":446,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:45:20.747419+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a VULLM on a single dataset and split it randomly into a training set and a held-out set, then run Vid-SME with members from the training split and non-members from the held-out split; if TPR@5% FPR drops to near the baseline level, the cross-dataset results were detecting dataset identity, not membership. A second check is to replace reversed order with a random temporal shuffle: if the entropy gap persists, the mechanism is not specifically the natural-versus-reversed asymmetry.","supporting_citations":[{"cited_title":"New non-additive measures of entropy for discrete probability distributions","cited_arxiv_id":null,"evidence_quote":"defines the Sharma-Mittal entropy formula that Vid-SME's score is built on."}],"review_version":1}