Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Vid-SME: Membership Inference Attacks against Large Video Understanding Models

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that videos used to train video-understanding language models can be detected from output logits by comparing the model's prediction entropy on natural versus temporally reversed frame sequences.

desk verdict First video MIA with a novel natural-vs-reversed entropy idea, but the only matched-distribution evaluation gives AUC ≈ 0.54; the strong results are cross-dataset and likely detect dataset identity, not membership. read the letter →

arxiv 2506.03179 v1 pith:THGEN4B6 submitted 2025-05-29 cs.CV cs.AI

Qi Li , Runpeng Yu , Xinchao Wang This is my paper
classification cs.CVcs.AI
keywords membershipinferenceattackvideounderstandinglargelanguagemodelSharma-MittalentropytemporalreversaldataprivacymultimodalLLMlowfalsepositiverate
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Vid-SME claims that a video's membership in the instruction-tuning data of a video-understanding large language model can be detected from the model's output logits alone. The attack computes Sharma-Mittal entropy over next-token probabilities for both the natural frame order and the temporally reversed frame order, adaptively choosing the entropy parameters from each video's motion and illumination statistics. For videos the model saw during training, natural-order frames produce more confident predictions and reversed frames produce a larger entropy jump, so the natural-minus-reversed entropy gap becomes the membership score. The authors report that this score consistently outperforms existing token-level MIAs on five VULLMs, including three models trained for the benchmark and two open-sourced models, with the largest gains at the low-false-positive rates that matter for real auditing.

What carries the argument

The load-bearing object is Sharma-Mittal entropy, $$S_{q,r}(p)=\frac{1}{1-r}\left(\left(\sum_i p_i^q\right)^{\frac{1-r}{1-q}}-1\right),$$ a two-parameter entropy that generalizes Shannon, Rényi, and Tsallis entropies. Vid-SME chooses $q$ and $r$ per video from normalized optical-flow variance and illumination variation, then computes $S_{q,r}$ on the video-token slices of the next-token distributions for the natural and reversed frame sequences. The final score is the average of the smallest $K\%$ of per-position differences $\Delta S = S_{\text{nat}} - S_{\text{rev}}$, which the paper identifies as the membership signal.

What would settle it

Build a VULLM on a single dataset and split it randomly into a training set and a held-out set, then run Vid-SME with members from the training split and non-members from the held-out split; if TPR@5% FPR drops to near the baseline level, the cross-dataset results were detecting dataset identity, not membership. A second check is to replace reversed order with a random temporal shuffle: if the entropy gap persists, the mechanism is not specifically the natural-versus-reversed asymmetry.

Watch

Extended reading notes

Core claim

The discovery is the temporal asymmetry in memorization: a video-understanding LLM that has seen a video during instruction tuning predicts next tokens with higher confidence when frames are in their original order, and suffers a sharper confidence drop when those same frames are reversed; a model that has never seen the video shows a smaller natural-versus-reversed entropy difference. Vid-SME turns this gap into a membership detector by computing $S_{q,r}$ with per-video adaptive $q$ and $r$, taking the element-wise difference between natural and reversed entropy slices, and aggregating the smallest $K\%$ of differences as the score. The paper argues that existing text/image MIA metrics fail on video because they ignore both inter-frame variation and the effect of the number of sampled frames, whereas Vid-SME is designed to capture both. Across five target models, the method reports higher AUC, accuracy, and TPR@5% FPR than all baselines in nearly every setting.

Load-bearing premise

The load-bearing premise is that the comparison videos labeled as non-members differ from the member videos only in whether they were in the training set; in several experiments the two pools come from different datasets, so the reported separation could partly reflect dataset or task differences rather than membership, and the authors acknowledge this in Appendix D.

Editorial extensions

If this is right

  • An auditor with query access to a VULLM can decide whether a specific video was in the instruction-tuning set without knowing the model weights.
  • The natural-versus-reversed entropy gap is a more reliable signal at low false-positive rates than perplexity or Rényi-based text/image MIA baselines, where baseline TPR@5% FPR is often near zero.
  • The attack stays effective as the number of sampled frames changes, as the target dataset grows, and under motion-blur and brightness corruption, because the adaptive $q/r$ parameters track video statistics.
  • Because the instruction context has only a limited influence on the score, the signal comes primarily from the video frames themselves, not from the wording of the question.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the paper leaves implicit: the same natural-versus-reversed entropy comparison could serve as a diagnostic for temporal memorization in other sequential modalities, such as audio or 3D event streams, though the paper only tests video.
  • A caveat worth testing: the cross-dataset non-member sets used in several experiments (for example CinePile members against MLVU non-members) could allow the score to separate dataset identity rather than membership; a same-source held-out evaluation would settle whether the signal is true membership leakage.
  • If the signal is real, a natural defensive response would be to randomize frame order during instruction tuning or to add an entropy-gap regularizer, but the paper does not propose defenses.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper proposes Vid-SME, a membership inference attack (MIA) against video understanding LLMs (VULLMs). The method computes Sharma–Mittal entropy over the next-token distributions of video-token slices, adaptively sets the entropy parameters q and r from video motion and illumination statistics, and forms a membership score from the minimum-K% difference between entropy values for natural and temporally-reversed frame sequences. The authors evaluate Vid-SME on three self-trained VULLMs and two open-source VULLMs, comparing it with perplexity, Min-K%, Rényi-based, and other metric-based baselines, and report consistent improvements, especially in TPR@5% FPR. Code is released.

Significance. The paper introduces a novel angle to video MIA—temporal reversal combined with an adaptive entropy measure—and contributes three self-trained VULLMs as an evaluation benchmark. If the empirical claims were supported, this would be a useful first dedicated video MIA method. However, the central claim is not backed by the evaluation: the only matched-distribution setting yields near-chance performance, and the strong headline numbers come from cross-dataset member/non-member pairs. The contribution is therefore currently a proposal with an unvalidated empirical basis rather than a demonstrated attack.

major comments (3)
  1. [§5, Tables 1 and 2] The evaluation confounds membership with dataset identity. The only matched-distribution setting is Video-XL-NExT-QA-7B, where members are the NExT-QA training split and non-members are the NExT-QA test split; there all Vid-SME variants obtain AUC 0.519–0.548 and TPR@5% FPR 0.030–0.106, barely above chance. The high-scoring results in Table 2 (e.g., AUC 0.840 for Video-XL-CinePile-7B with Mean Vid-SME) are obtained using non-members from a different dataset (MLVU), and the LongVA-Caption and LLaVA-NeXT-Video settings similarly pair members with non-members from Video-XL, VDC, or Video-Instruct-100K. Because these pairs differ in dataset source, task format, and content distribution, the natural-vs-reversed entropy gap may simply separate datasets rather than training membership. Appendix D acknowledges that available non-member sets only approximate the training distribution and 'may not fully reflect the severity of the privacy risk'; this is the load-bearing gap and invalidates the paper's central claim that Vid-SME derives robust membership scores.
  2. [§4.1, Eq. (2); §4.2, Eq. (4)] The attack's free parameters are selected post hoc. Equation (2) sets beta1=1.0 and beta2=0.1 with no derivation or sensitivity analysis, and Equation (4) introduces K%, with the paper reporting six variants (Mean, Min_0%, Min_5%, Min_30%, Min_60%, Min_90%) and highlighting the best-performing one per setting. Without a principled procedure for choosing beta1, beta2, and K, the reported numbers do not describe a fixed attack, and the claimed consistent improvements may reflect favorable parameter selection rather than intrinsic robustness of the method.
  3. [§5.1, Table 2] The claim that Vid-SME 'consistently achieves the best performance under all settings' is not supported by Table 2. For LongVA-Caption-7B, Mean Vid-SME has AUC 0.496 and TPR@5% FPR 0.039, essentially tied with several baselines; for LLaVA-NeXT-Video-7B, most Vid-SME variants have AUC below 0.60. The only large gains (e.g., AUC 0.840 and TPR@5% FPR 0.420 for Video-XL-CinePile-7B) occur in the cross-dataset settings described above, so the headline improvement is an artifact of the evaluation confound rather than a consistent property of the method.
minor comments (5)
  1. [§2.2] The word 'dicussed' should be 'discussed'.
  2. [§4.1] The word 'algin' should be 'align'.
  3. [§5.2, Table 4] The caption of Table 4 reads 'Performance comparison on different instructions' but the table reports results under different dataset scales; the caption should be corrected.
  4. [Figure 1] The figure contains the typo 'calculaiton'; it should be 'calculation'.
  5. [Appendix D and Figures 11–14] The figures label non-member examples as 'non-memebr'; this typo should be fixed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Vid-SME's entropy-difference score is an empirically evaluated heuristic; the cross-dataset evaluation confound is a validity concern, not a definitional reduction.

full rationale

Vid-SME's membership score is defined as a natural-versus-reversed Sharma-Mittal entropy difference (Eqs. 3-4), and the claim that this difference separates members from non-members is an empirical hypothesis tested against held-out labels, not a quantity fitted to those labels. The adaptive q/r values in Eq. 2 are heuristic functions of optical-flow variance and illumination with fixed scaling coefficients (beta1=1.0, beta2=0.1), not parameters fitted to the membership outcome, and K is swept across 0, 5, 30, 60, 90, 100 and reported as separate variants rather than a single fitted prediction. The self-citations in the reference list (e.g., Qi Li et al. data lineage and unlearning works) are peripheral related-work entries and are not load-bearing for the central claim. The main threat is the evaluation design: for Video-XL-CinePile-7B the member set is CinePile and the non-member set is MLVU, and similar cross-dataset pairs are used for LongVA-Caption and LLaVA-NeXT-Video, so the reported AUC/TPR numbers may reflect dataset discrimination rather than training membership; this is acknowledged in Appendix D as an approximation of the training distribution. That is a correctness and validity concern about confounded evaluation, but it is not circularity under the requested reduction standard, because the attack scores are not constructed from the member/non-member labels and no equation in the paper defines membership in terms of the score. No circular step can be exhibited.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method rests on an unverified mapping from video statistics to entropy parameters, on a behavioural assumption about natural/reversed entropy differences, and on an evaluation design that assumes cross-dataset non-members are equivalent to same-distribution non-members. These are not derived from first principles.

free parameters (3)
  • beta1 = 1.0
    Scaling coefficient in Eq. (2) mapping normalized motion complexity to q; set by hand, no search or derivation.
  • beta2 = 0.1
    Scaling coefficient in Eq. (2) mapping normalized illumination variation to r; set by hand.
  • K percent = multiple values (0,5,30,60,90,100) reported; best highlighted
    The Vid-SME-K% score aggregates the smallest K% of entropy differences; K is swept and the best result per setting is emphasized, making it a tuned parameter.
assumptions (4)
  • ad hoc to paper Non-member videos from a different dataset are valid non-members that differ only in membership status.
    Used to build member/non-member sets for CinePile, LongVA-Caption, and open LLaVA models; not justified and contradicted by dataset differences.
  • domain assumption Members show lower natural-order entropy and larger natural-versus-reversed entropy gap than non-members.
    Core hypothesis of the attack, motivated by Figure 2b but not independently verified.
  • domain assumption Video model memorization during instruction tuning is detectable through output logits.
    Standard grey-box MIA assumption; referenced to prior work.
  • ad hoc to paper Motion and illumination statistics can be mapped to Sharma-Mittal parameters q and r via the fixed formula in Eq. (2).
    The mapping is a heuristic; no evidence it is optimal or even beneficial beyond the ablation showing removal hurts.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Vid-SME: Membership Inference Attacks against Large Video Understanding Models." pith.science (2026). https://pith.science/paper/THGEN4B6

@misc{pith2026250603179,
  author       = {Pith},
  title        = {Pith review of: Vid-SME: Membership Inference Attacks against Large Video Understanding Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/THGEN4B6}},
  note         = {Machine review of arXiv:2506.03179}
}
read the original abstract

Multimodal large language models (MLLMs) demonstrate remarkable capabilities in handling complex multimodal tasks and are increasingly adopted in video understanding applications. However, their rapid advancement raises serious data privacy concerns, particularly given the potential inclusion of sensitive video content, such as personal recordings and surveillance footage, in their training datasets. Determining improperly used videos during training remains a critical and unresolved challenge. Despite considerable progress on membership inference attacks (MIAs) for text and image data in MLLMs, existing methods fail to generalize effectively to the video domain. These methods suffer from poor scalability as more frames are sampled and generally achieve negligible true positive rates at low false positive rates (TPR@Low FPR), mainly due to their failure to capture the inherent temporal variations of video frames and to account for model behavior differences as the number of frames varies. To address these challenges, we introduce Vid-SME, the first membership inference method tailored for video data used in video understanding LLMs (VULLMs). Vid-SME leverages the confidence of model output and integrates adaptive parameterization to compute Sharma-Mittal entropy (SME) for video inputs. By leveraging the SME difference between natural and temporally-reversed video frames, Vid-SME derives robust membership scores to determine whether a given video is part of the model's training set. Experiments on various self-trained and open-sourced VULLMs demonstrate the strong effectiveness of Vid-SME.

Figures

Figures reproduced from arXiv: 2506.03179 by the authors.

Figure 2
Figure 2. Example of the q/r value distribution and entropy distribution on Video-XL-CinePile-7B. To do so, for the i-th video, after the frame sequence sampling, we quantify its motion complexity ϕi as the mean variance of optical flow [14] between consecutive frames, while its illumination variation λi is measured as the standard deviation of average brightness across frames. Specifically, each frame is first converted to g… view at source ↗
Figure 3
Figure 3. The ROC and TPR@5% FPR curves. caption dataset Video-Instruct-100K [32] that serves as part of their training data as the mem￾ber set. Each video in this dataset has multiple questions, from which we select the one with the longest text length, resulting in 869 samples. The non-member set consists of 869 samples randomly selected from the captioning data from Video-XL training set [46]. Baselines. We adopt several m… view at source ↗
Figure 4
Figure 4. Analysis on: (a) Train-Test Gap v.s. # frames, (b) Attack performance v.s. # frames, (c) [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figures from the paper (10 more)
Figure 5
Figure 5. Figure 5: A comparison between with and without full context. Relationship between model memorization, frame conditions and attack performance. To analyze the relationship among model memorization, frame conditions, and attack performance, we further investigate how the train-te…
Figure 6
Figure 6. Figure 6: A comparison between with and without full instructions. The results when # frames = 8 and the target model is LLaVA-NeXT-Video-34B are reported in [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Training Loss and Gradient Norm over Steps for the three self-trained models. [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: Example of the video-text instruction context under different types and levels of corruptions. [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: The three different instruction contexts used in the ablation study. [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: The short query text used in the ablation study. [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 11
Figure 11. Figure 11: An example of member and non-member data for Video-XL-NExT-QA-7B. [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 12
Figure 12. Figure 12: An example of member and non-member data for Video-XL-CinePile-7B. [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]
Figure 13
Figure 13. Figure 13: An example of member and non-member data for Longva-Caption-7B. [PITH_FULL_IMAGE:figures/full_fig_p017_13.png]
Figure 14
Figure 14. Figure 14: An example of member and non-member data for LLaVA-NeXT-Video-7B/34B. [PITH_FULL_IMAGE:figures/full_fig_p017_14.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Video-XL-2: Towards Very Long-Video Understanding Through Task-Aware KV Sparsification

    cs.CV 2025-06 conditional novelty 5.0 of 10

    Video-XL-2 cuts long-video inference cost with chunked pre-filling and query-gated dense-or-sparse KV reloading, reporting half the FLOPs and a third less decoding memory at roughly equal benchmark scores.

Reference graph

Works this paper leans on

68 extracted references · 24 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Generalised information and entropy measures in physics

    Christian Beck. Generalised information and entropy measures in physics. Contemporary Physics, 50(4):495–510, 2009

    work page 2009

  3. [3]

    Is space-time attention all you need for video understanding? In ICML, volume 2, page 4, 2021

    Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In ICML, volume 2, page 4, 2021

    work page 2021

  4. [4]

    Membership inference attacks from first principles

    Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference attacks from first principles. In 2022 IEEE symposium on security and privacy (SP), pages 1897–1914. IEEE, 2022

    2022

  5. [5]

    The secret sharer: Evaluating and testing unintended memorization in neural networks

    Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX security symposium (USENIX security 19), pages 267–284, 2019

    work page 2019

  6. [6]

    Extracting training data from large language models

    Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-V oss, Kather- ine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In 30th USENIX security symposium (USENIX Security 21), pages 2633–2650, 2021

    2021

  7. [7]

    Auroracap: Efficient, performant video detailed captioning and a new benchmark

    Wenhao Chai, Enxin Song, Yilun Du, Chenlin Meng, Vashisht Madhavan, Omer Bar-Tal, Jenq- Neng Hwang, Saining Xie, and Christopher D Manning. Auroracap: Efficient, performant video detailed captioning and a new benchmark. arXiv preprint arXiv:2410.03051, 2024

    arXiv

  8. [8]

    Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts

    Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3558–3568, 2021

    2021

Show all 68 references
  1. [9]

    A hierarchical variational neural uncertainty model for stochastic video prediction

    Moitreya Chatterjee, Narendra Ahuja, and Anoop Cherian. A hierarchical variational neural uncertainty model for stochastic video prediction. InProceedings of the IEEE/CVF international conference on computer vision, pages 9751–9761, 2021

    2021

  2. [10]

    Gan-leaks: A taxonomy of member- ship inference attacks against generative models

    Dingfan Chen, Ning Yu, Yang Zhang, and Mario Fritz. Gan-leaks: A taxonomy of member- ship inference attacks against generative models. In Proceedings of the 2020 ACM SIGSAC conference on computer and communications security, pages 343–362, 2020

    2020

  3. [11]

    Sharegpt4v: Improving large multi-modal models with better captions

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. In European Conference on Computer Vision, pages 370–387. Springer, 2024

    2024

  4. [12]

    Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality

    Wei-Lin Chiang, Zhuohan Li, Ziqing Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023...

    2023

  5. [13]

    A summary on entropy statistics

    Maria Dolores Esteban and Domingo Morales. A summary on entropy statistics. Kybernetika, 31(4):337–346, 1995

    1995

  6. [14]

    Polynomial expansion for orientation and motion estimation

    Gunnar Farnebäck. Polynomial expansion for orientation and motion estimation. Linkopings Universitet (Sweden), 2002

    2002

  7. [15]

    Dropout as a bayesian approximation: Representing model uncertainty in deep learning

    Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. Ininternational conference on machine learning, pages 1050–1059. PMLR, 2016

    2016

  8. [16]

    Vision-language models for medical report generation and visual question answering: A review

    Iryna Hartsock and Ghulam Rasool. Vision-language models for medical report generation and visual question answering: A review. Frontiers in Artificial Intelligence, 7:1430984, 2024

    2024

  9. [17]

    Logan: Membership inference attacks against generative models

    Jamie Hayes, Luca Melis, George Danezis, and Emiliano De Cristofaro. Logan: Membership inference attacks against generative models. arXiv preprint arXiv:1705.07663, 2017. 10

    2017 arXiv

  10. [18]

    Bliva: A simple multimodal llm for better handling of text-rich visual questions

    Wenbo Hu, Yifan Xu, Yi Li, Weiyue Li, Zeyuan Chen, and Zhuowen Tu. Bliva: A simple multimodal llm for better handling of text-rich visual questions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 2256–2264, 2024

    2024

  11. [19]

    Membership inference attacks against vision-language models

    Yuke Hu, Zheng Li, Zhihao Liu, Yang Zhang, Zhan Qin, Kui Ren, and Chun Chen. Membership inference attacks against vision-language models. arXiv preprint arXiv:2501.18624, 2025

    2025 arXiv

  12. [20]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730–19742. PMLR, 2023

    2023

  13. [21]

    Videochat: Chat-centric video understanding

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355, 2023

    2023 arXiv

  14. [22]

    Data lineage inference: Uncovering privacy vulnerabilities of dataset pruning

    Qi Li, Cheng-Long Wang, Yinzhi Cao, and Di Wang. Data lineage inference: Uncovering privacy vulnerabilities of dataset pruning. arXiv preprint arXiv:2411.15796, 2024

    2024 arXiv

  15. [23]

    Membership inference attacks against large vision-language models

    Zhan Li, Yongtao Wu, Yihang Chen, Francesco Tonin, Elias Abad Rocamora, and V olkan Cevher. Membership inference attacks against large vision-language models. Advances in Neural Information Processing Systems, 37:98645–98674, 2024

    2024

  16. [24]

    Next-qa: Next phase of question answering to explaining tem- poral actions

    Xiao Lin and Chenliang Xu. Next-qa: Next phase of question answering to explaining tem- poral actions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9777–9786, 2021

    2021

  17. [25]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024

    2024

  18. [26]

    Llava-next: Improved reasoning, ocr, and world knowledge, January 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024

    2024

  19. [27]

    St-llm: Large language models are effective temporal learners

    Ruyang Liu, Chen Li, Haoran Tang, Yixiao Ge, Ying Shan, and Ge Li. St-llm: Large language models are effective temporal learners. In European Conference on Computer Vision, pages 1–18. Springer, 2024

    2024

  20. [28]

    Tempcompass: Do video llms really understand videos? arXiv preprint arXiv:2403.00476, 2024

    Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. Tempcompass: Do video llms really understand videos? arXiv preprint arXiv:2403.00476, 2024

    2024 arXiv

  21. [29]

    Videoeval-pro: Robust and realistic long video understanding evaluation

    Wentao Ma, Weiming Ren, Yiming Jia, Zhuofeng Li, Ping Nie, Ge Zhang, and Wenhu Chen. Videoeval-pro: Robust and realistic long video understanding evaluation. arXiv preprint arXiv:2505.14640, 2025

    2025 arXiv

  22. [31]

    Membership inference on word embedding and beyond

    Saeed Mahloujifar, Huseyin A Inan, Melissa Chase, Esha Ghosh, and Marcello Hasegawa. Membership inference on word embedding and beyond. arXiv preprint arXiv:2106.11384, 2021

    2021 arXiv

  23. [32]

    Video-chatgpt: Towards detailed video understanding via large vision and language models

    Salman Khan Muhammad Maaz, Hanoona Rasheed and Fahad Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. ArXiv 2306.05424, 2023

    2023 arXiv

  24. [33]

    Expanding language-image pretrained models for general video recognition

    Bolin Ni, Houwen Peng, Minghao Chen, Songyang Zhang, Gaofeng Meng, Jianlong Fu, Shiming Xiang, and Haibin Ling. Expanding language-image pretrained models for general video recognition. In European conference on computer vision, pages 1–18. Springer, 2022

    2022

  25. [34]

    Slowfocus: Enhancing fine-grained temporal understanding in video llm

    Ming Nie, Dan Ding, Chunwei Wang, Yuanfan Guo, Jianhua Han, Hang Xu, and Li Zhang. Slowfocus: Enhancing fine-grained temporal understanding in video llm. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. 11

    2024

  26. [35]

    Learning latent subevents in activity videos using temporal attention filters

    A Piergiovanni, Chenyou Fan, and Michael Ryoo. Learning latent subevents in activity videos using temporal attention filters. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017

    2017

  27. [36]

    Fine-tuned clip models are efficient video learners

    Hanoona Rasheed, Muhammad Uzair Khattak, Muhammad Maaz, Salman Khan, and Fa- had Shahbaz Khan. Fine-tuned clip models are efficient video learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6545–6554, 2023

    2023

  28. [37]

    Cinepile: A long video question answering dataset and benchmark

    Ruchit Rawal, Khalid Saifullah, Ronen Basri, David Jacobs, Gowthami Somepalli, and Tom Goldstein. Cinepile: A long video question answering dataset and benchmark. arXiv preprint arXiv:2405.08813, 2024

    2024 arXiv

  29. [38]

    On measures of entropy and information

    Alfréd Rényi. On measures of entropy and information. In Proceedings of the fourth Berkeley symposium on mathematical statistics and probability, volume 1: contributions to the theory of statistics, volume 4, pages 547–562. University of California Press, 1961

    1961

  30. [39]

    Ml-leaks: Model and data independent membership inference attacks and defenses on machine learning models

    Ahmed Salem, Yang Zhang, Mathias Humbert, Pascal Berrang, Mario Fritz, and Michael Backes. Ml-leaks: Model and data independent membership inference attacks and defenses on machine learning models. arXiv preprint arXiv:1806.01246, 2018

    2018 arXiv

  31. [40]

    Laion-400m: Open dataset of clip-filtered 400 million image-text pairs

    Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114, 2021

    2021 arXiv

  32. [41]

    A mathematical theory of communication

    Claude E Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3):379–423, 1948

    1948

  33. [42]

    New non-additive measures of entropy for discrete probability distributions

    Bhudev D Sharma and Dharam P Mittal. New non-additive measures of entropy for discrete probability distributions. J. Math. Sci, 10(75):28–40, 1975

    1975

  34. [43]

    Detecting pretraining data from large language models

    Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. arXiv preprint arXiv:2310.16789, 2023

    2023 arXiv

  35. [44]

    Membership inference attacks against machine learning models

    Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pages 3–18. IEEE, 2017

    2017

  36. [45]

    Visual text processing: A comprehensive review and unified evaluation

    Yan Shu, Weichao Zeng, Fangmin Zhao, Zeyu Chen, Zhenhang Li, Xiaomeng Yang, Yu Zhou, Paolo Rota, Xiang Bai, Lianwen Jin, et al. Visual text processing: A comprehensive review and unified evaluation. arXiv preprint arXiv:2504.21682, 2025

    2025 arXiv

  37. [46]

    Video-xl: Extra-long vision language model for hour-scale video understanding

    Yan Shu, Peitian Zhang, Zheng Liu, Minghao Qin, Junjie Zhou, Tiejun Huang, and Bo Zhao. Video-xl: Extra-long vision language model for hour-scale video understanding. arXiv preprint arXiv:2409.14485, 2024

    2024 arXiv

  38. [47]

    Two-stream convolutional networks for action recognition in videos

    Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. Advances in neural information processing systems, 27, 2014

    2014

  39. [48]

    Information leakage in embedding models

    Congzheng Song and Ananth Raghunathan. Information leakage in embedding models. In Proceedings of the 2020 ACM SIGSAC conference on computer and communications security, pages 377–390, 2020

    2020

  40. [49]

    Machine learning models that remember too much

    Congzheng Song, Thomas Ristenpart, and Vitaly Shmatikov. Machine learning models that remember too much. In Proceedings of the 2017 ACM SIGSAC Conference on computer and communications security, pages 587–601, 2017

    2017

  41. [50]

    Systematic evaluation of privacy risks of machine learning models

    Liwei Song and Prateek Mittal. Systematic evaluation of privacy risks of machine learning models. In 30th USENIX Security Symposium (USENIX Security 21), pages 2615–2632, 2021

    2021

  42. [51]

    Membership inference attacks against adversari- ally robust deep learning models

    Liwei Song, Reza Shokri, and Prateek Mittal. Membership inference attacks against adversari- ally robust deep learning models. In 2019 IEEE Security and Privacy Workshops (SPW), pages 50–56. IEEE, 2019. 12

    2019

  43. [52]

    On generalized information measures and their applications

    Inder Jeet Taneja. On generalized information measures and their applications. In Advances in Electronics and Electron Physics, volume 76, pages 327–413. Elsevier, 1989

    1989

  44. [53]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  45. [54]

    Possible generalization of boltzmann-gibbs statistics

    Constantino Tsallis. Possible generalization of boltzmann-gibbs statistics. Journal of statistical physics, 52:479–487, 1988

    1988

  46. [55]

    Towards lifecycle unlearn- ing commitment management: Measuring sample-level approximate unlearning completeness

    Cheng-Long Wang, Qi Li, Zihang Xiang, Yinzhi Cao, and Di Wang. Towards lifecycle unlearn- ing commitment management: Measuring sample-level approximate unlearning completeness. arXiv preprint arXiv:2403.12830, 2024

    2024 arXiv

  47. [56]

    Cogvlm: Visual expert for pretrained language models

    Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Song XiXuan, et al. Cogvlm: Visual expert for pretrained language models. Advances in Neural Information Processing Systems, 37:121475–121499, 2024

    2024

  48. [57]

    Videoagent: Long-form video understanding with large language model as agent

    Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung-Levy. Videoagent: Long-form video understanding with large language model as agent. In European Conference on Computer Vision, pages 58–76. Springer, 2024

    2024

  49. [58]

    Membership inference attacks on large-scale models: A survey

    Hengyu Wu and Yang Cao. Membership inference attacks on large-scale models: A survey. arXiv preprint arXiv:2503.19338, 2025

    2025 arXiv

  50. [59]

    Optimizing video prediction via video frame interpola- tion

    Yue Wu, Qiang Wen, and Qifeng Chen. Optimizing video prediction via video frame interpola- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17814–17823, 2022

    2022

  51. [60]

    Privacy risk in machine learning: Analyzing the connection to overfitting

    Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: Analyzing the connection to overfitting. In 2018 IEEE 31st computer security foundations symposium (CSF), pages 268–282. IEEE, 2018

    2018

  52. [61]

    Memory-enhanced retrieval augmentation for long video understanding

    Huaying Yuan, Zheng Liu, Minhao Qin, Hongjin Qian, Y Shu, Zhicheng Dou, and Ji-Rong Wen. Memory-enhanced retrieval augmentation for long video understanding. arXiv preprint arXiv:2503.09149, 2025

    2025 arXiv

  53. [62]

    Low-cost high-power membership inference attacks

    Sajjad Zarifzadeh, Philippe Liu, and Reza Shokri. Low-cost high-power membership inference attacks. arXiv preprint arXiv:2312.03262, 2023

    2023 arXiv

  54. [63]

    Understanding deep learning (still) requires rethinking generalization

    Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107– 115, 2021

    2021

  55. [64]

    Min-k%++: Improved baseline for detecting pre-training data from large language models

    Jingyang Zhang, Jingwei Sun, Eric Yeats, Yang Ouyang, Martin Kuo, Jianyi Zhang, Hao Frank Yang, and Hai Li. Min-k%++: Improved baseline for detecting pre-training data from large language models. arXiv preprint arXiv:2404.02936, 2024

    2024 arXiv

  56. [65]

    Long context transfer from language to vision

    Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv preprint arXiv:2406.16852, 2024

    2024 arXiv

  57. [66]

    Instruction tuning for large language models: A survey

    Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, et al. Instruction tuning for large language models: A survey. arXiv preprint arXiv:2308.10792, 2023

    2023

  58. [67]

    Llava-next: A strong zero-shot video understanding model, April 2024

    Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava-next: A strong zero-shot video understanding model, April 2024

    2024

  59. [68]

    Mlvu: A comprehensive benchmark for multi-task long video understanding

    Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding. arXiv preprint arXiv:2406.04264, 2024

    2024 arXiv

  60. [69]

    A survey on deep learning technique for video segmentation

    Tianfei Zhou, Fatih Porikli, David J Crandall, Luc Van Gool, and Wenguan Wang. A survey on deep learning technique for video segmentation. IEEE transactions on pattern analysis and machine intelligence, 45(6):7099–7122, 2022. 13 Appendix A Model Configurations and Training Sta...

    2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.