Pith. sign in

REVIEW 4 major objections 6 minor 53 references

VUDG: A Dataset for Video Understanding Domain Generalization

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper introduces VUDG, a video-understanding benchmark that holds a fixed semantic space across 11 visual domains, and reports that most current models, including state-of-the-art LVLMs, degrade substantially when the domain shifts.

desk verdict VUDG is a genuinely useful dataset and the first real attempt at controlled domain-shift evaluation in VideoQA, but the semantic-similarity premise is only weakly supported and the 'degradation' headline needs a same-domain baseline. read the letter →

arxiv 2505.24346 v1 pith:NH4QOEO7 submitted 2025-05-30 cs.CV

Ziyi Wang , Zhi Gao , Boxuan Yu , Zirui Dai , Yuxiang Song , Qingyuan Lu , Jin Chen , Xinxiao Wu This is my paper
classification cs.CV
keywords videounderstandingdomaingeneralizationquestionansweringlargevideo-languagemodelsbenchmarkdatasetshiftdistributionmulti-expertannotation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces VUDG, a dataset built to test whether video-understanding models generalize across visual domains while the underlying semantics stay the same. It collects roughly 7,900 videos from 11 domains grouped into three shift types (visual style, viewpoint, and environmental condition) and annotates them with 36,388 question-answer pairs spanning action, attribute, object, and temporal reasoning. The central claim is that VUDG is the first benchmark that isolates domain shift from semantic change, so a model's accuracy drop across domains can be read as a genuine generalization failure. On this benchmark, state-of-the-art LVLMs and traditional VideoQA methods show large and uneven degradation under domain shift, with zero-shot multiple-choice accuracy ranging from roughly 13% to 72%. The authors propose a multi-expert annotation pipeline, where one model generates questions, another generates wrong options, a third filters, and humans correct, to keep the QA pairs reliable.

What carries the argument

The load-bearing object is the VUDG dataset itself, a corpus of video clips constructed so that domain and semantics are decoupled. Construction has two mechanisms: first, a fixed activity list of 37 daily human actions is used by Qwen2.5-VL to select videos from all 11 domains, so every domain covers the same semantic space; second, a progressive multi-expert annotation pipeline, where Gemini-2.5-Flash generates open-ended QA pairs, DeepSeek-v3 generates wrong options, Gemini-2.5-Pro screens, and humans correct, produces multiple-choice and open-ended questions in four categories: action recognition, attribute identification, object identification, and temporal understanding. Evaluation uses leave-one-domain-out and leave-but-one-domain-out protocols, plus zero-shot testing, with accuracy for multiple-choice and a 0-10 LLM-judged score for open-ended answers. The paper's main evidence that the decoupling was achieved is a t-SNE visualization in which CLIP frame features cluster by domain while question and answer embeddings mix across domains.

What would settle it

Compute per-domain frequencies of each of the 37 activities and human-rated difficulty per question; if any domain's activity mix deviates sharply from the others or the same activity is systematically harder in one domain, the semantic-similarity premise is violated and the accuracy gaps cannot be attributed to domain shift.

Watch

Extended reading notes

Core claim

The paper's central discovery, stated on its own terms, is that a domain-generalization benchmark for video understanding can be built by holding a fixed semantic space of 37 daily activities constant while varying surface appearance: cartoon, game, movie/TV, virtual, egocentric, surveillance, shaky, foggy, night, rainy, and snowy. Because every domain is supposed to depict the same activity types, performance differences between domains measure robustness to distribution shift rather than task difficulty. The paper reports that most models degrade under this shift: in multiple-domain generalization the best LLM-based method averages 66.9% while non-LLM VideoQA methods stay near chance (around 18%), and in zero-shot multiple-choice evaluation only the largest Qwen2.5-VL variant reaches 72.1% average accuracy. The authors interpret the gap as evidence that current video models are not robust to domain shifts, and that VUDG can support multiple-domain, single-domain, and zero-shot generalization evaluation.

Load-bearing premise

The load-bearing assumption is that all 11 domains end up with the same mix of the 37 daily activities and equally difficult questions, so accuracy differences can be blamed on domain shift rather than content differences.

Editorial extensions

If this is right

  • If the central claim is correct, VUDG gives the video-understanding community a standard way to compare domain-generalization methods: a method that improves average accuracy on held-out domains without training on them demonstrates genuine generalization.
  • The consistent finding that temporal-understanding questions are harder than object-identification questions across domains suggests current LVLMs are more robust at appearance-based reasoning than at dynamic temporal reasoning under distribution shift.
  • Models fine-tuned on the full VUDG training set outperform models trained under domain-generalization protocols, so the training set can serve as source data for studying how to close the generalization gap.
  • The zero-shot results establish a controlled baseline that future video-understanding models can measure themselves against, with semantics held fixed and only visual domains changing.
  • The benchmark supports both multiple-domain and single-domain generalization evaluation, so it can separate the challenge of learning from many source domains from the harder challenge of learning from one.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the paper's core design is the fixed 37-activity list, the same recipe could be applied to deliberately rendered synthetic domains, where semantic equivalence is guaranteed by the renderer rather than by a vision-language model's filtering.
  • The authors note that only visual domain shifts are covered; extending the controlled-activity design to textual paraphrase shifts or added audio would test whether robustness failures are specific to the visual modality.
  • A stricter validation than the reported t-SNE would be to restrict scoring to only the activities that actually appear in all 11 domains; if the domain gaps persist on that subset, the attribution of accuracy drops to domain shift is much stronger.
  • The paper's overview states 7,899 videos while its statistics section sums to 7,869 training and testing clips, a discrepancy that should be resolved before the dataset is used as a formal benchmark.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces VUDG, a video-understanding benchmark for domain generalization, with 11 domains grouped into three shift types (visual style, viewpoint, environmental condition) and 36,388 QA pairs over 7,899 training and 1,532 testing clips. The authors describe a multi-expert annotation pipeline (Gemini-2.5-Flash for questions, DeepSeek-v3 for options, Gemini-2.5-Pro for review, plus human correction) and evaluate several VideoQA baselines and nine LVLMs under multiple-domain, single-domain, and zero-shot generalization protocols. The central claim is that VUDG preserves semantic similarity across domains while introducing visual domain shifts, so that observed accuracy gaps reflect domain-shift effects rather than semantic differences.

Significance. If validated, VUDG would be a useful resource: it is the first benchmark aimed specifically at video-understanding domain generalization, covers a broad set of realistic domains, separates training and testing data across sources to reduce leakage, and reports a wide range of model evaluations. The multi-model annotation pipeline with a human review step is also a constructive attempt to mitigate single-model bias. However, the paper's central claim depends on two pieces of evidence that are currently missing or underdeveloped: a quantitative demonstration of cross-domain semantic similarity, and a same-domain baseline that would let the reader attribute accuracy gaps to domain shift rather than to semantic or difficulty confounds. The paper therefore has a potentially valuable contribution, but the benchmark's validity is not yet established at the level claimed.

major comments (4)
  1. [§3.1 and §4.4] The claim that VUDG isolates visual domain shift from semantic content is load-bearing, but the supporting evidence is qualitative only. Section 3.1 filters videos by a 37-activity list using Qwen2.5-VL-7B, but the paper never reports the resulting per-domain activity distribution or per-domain question-type distribution. The only quantitative-looking evidence, the t-SNE in Section 4.4, is a nonlinear two-dimensional projection that can obscure large distributional differences, and no distances or divergences are computed. Please report per-domain statistics for the 37 activities and QA categories, and compute an embedding-based divergence (e.g., MMD or centered kernel alignment on text features) between domains, together with an analysis of per-domain perceptual difficulty (e.g., question length, answer type, or human-rated difficulty). Without these, the accuracy gaps in Tables 2 and 4 cannot be attributed uniquely to visual domain shift.
  2. [§4.2, Table 3] The text in Section 4.2 states that 'a clear gap remains compared to models fine-tuned across all domains (Table 3)', but Table 3 reports single-domain generalization (one source domain, the remaining domains as targets), not an all-domain fine-tuned baseline. No in-domain upper bound is presented anywhere. The paper must add a same-domain or all-domain fine-tuned baseline (e.g., training on all source domains and testing on held-out test videos from the same domains, or within-domain train/test splits) to quantify the performance drop specifically attributable to domain shift. Without this baseline, the conclusion that most models 'suffer performance degradation under domain shifts' is not directly supported by the experiments.
  3. [§3.2 and §4.1] The benchmark's QA quality and the open-ended evaluation both rely on the same family of LLM-based judges (Gemini for generation and review, DeepSeek for options and open-ended scoring), and no human-agreement statistics are provided. The paper says humans review samples flagged by Gemini-2.5-Pro, but it does not report how many samples were corrected or removed, nor any inter-annotator agreement on a held-out subset. Please include a human evaluation study on a random sample of QA pairs and open-ended scores (e.g., percentage of questions judged valid, accuracy of generated answers, correlation between DeepSeek scores and human scores). This is needed to rule out the possibility that rankings in Tables 5 and 6 partly reflect judge-model preferences rather than video-understanding ability.
  4. [§5] The conclusion states that 'models fine-tuned on the complete training set of VUDG achieve substantial gains', but I could not find any table or experiment in the paper that fine-tunes on the complete training set (all 11 domains) and reports the resulting performance. The only fine-tuned results are in Tables 2 and 3, which use leave-one-domain-out and leave-one-source-domain-out protocols. Either add the missing experiment or remove the claim, since as written the conclusion is unsupported by the reported results.
minor comments (6)
  1. [Abstract and title] The phrase 'VideoUnderstandingDomain Generalization' is missing spaces in the title and abstract; it should read 'Video Understanding Domain Generalization'.
  2. [§4.3] In the first paragraph of Section 4.3, the text says 'As shown in Table 2' when referring to zero-shot multiple-choice results; the correct reference is Table 4.
  3. [§3.1 and Appendix B] Section 3.1 states that Qwen2.5-VL-7B is used for activity filtering, while Appendix B refers to Qwen2.5-VL-3B in the filtering prompt; please reconcile the model version used.
  4. [Throughout] The model naming is inconsistent: the paper alternates between 'Qwen2.5VL-3B' and 'Qwen2.5-VL-3B', and 'mPLUG-Owl3-7B-' appears with a trailing hyphen in Section 4.1. Please standardize all model names.
  5. [Table 1] The table contains '%' symbols in the 'Sem.' column that appear to be placeholders rather than meaningful values; please replace them with explicit check/cross marks or '—' for not applicable.
  6. [§4.4] The caption of Figure 6 mentions 'frame features, question embeddings, and ground-truth answer embeddings', but the text only describes CLIP/BGE encoders in general terms; please specify the exact encoder and pooling used for frame features and state whether the embeddings are from the same videos across domains.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: VUDG's semantic-similarity claim is enforced by an activity filter and validated with t-SNE; weak validation is a confound concern, not a circular reduction.

full rationale

I find no circular step in the paper's derivation chain. The central claims are that VUDG contains 11 domains with cross-domain semantic similarity and that models degrade under domain shift. The semantic-similarity claim is a dataset-construction property, not a derived prediction: Section 3.1 says 'To ensure semantic similarity across domains, we manually define a list of daily human activity scenes ... and utilize Qwen2.5-VL-7B to select videos belonging to this predefined activity list.' This is an input-driven filtering procedure whose adequacy is then visualized in Section 4.4 with t-SNE plots of question and answer embeddings. That evidence is qualitative, and the paper does not report per-domain activity distributions or quantitative divergences, so the semantic-similarity claim is under-validated; however, this is a validity/confound concern, not an equation-level reduction of the result to its inputs. Similarly, the QA/evaluation pipeline does not reduce by construction: open-ended answers are generated by Gemini-2.5-Flash, multiple-choice options by DeepSeek-v3, reviewed by Gemini-2.5-Pro and humans, and open-ended answers are scored by DeepSeek-v3 using the scoring formula in Eq. (3); no fitted parameter is renamed as a prediction, and multiple-choice accuracy in Table 4 is computed directly without an LLM judge. The only author-overlapping citation is [13], used alongside [41] to name the standard leave-but-one-domain-out protocol whose formulas are given in Eqs. (1)-(2); that citation is not load-bearing. One textual inconsistency should be noted but is not circular: Section 4.2 says 'a clear gap remains compared to models fine-tuned across all domains (Table 3)', yet Table 3 reports single-domain generalization and no all-domain fine-tuned baseline is shown; this missing comparison weakens the surrounding claim but does not make the derivation circular. Overall, the paper's benchmark construction and experimental evaluation are self-contained in the relevant sense, so the circularity score is 0.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

The central claim that VUDG measures domain generalization rests on several unverified assumptions: that the activity list creates semantic balance, that the video filter and annotation pipeline are accurate, that the LLM judge is reliable, and that the train/test splits truly avoid pretraining leakage. None of these are backed by quantitative validation in the paper.

assumptions (6)
  • domain assumption The 37-item human daily activity list defines a common semantic space shared by all 11 domains.
    The paper assumes that if videos from different domains depict these activities, then semantic content is comparable across domains. This is load-bearing for the claim that accuracy differences reflect domain shift, not semantic difficulty. The paper does not report per-domain activity distributions to confirm balance.
  • domain assumption Qwen2.5-VL-7B can reliably filter videos by domain and activity from the source datasets.
    The video collection in Section 3.1 relies on Qwen2.5-VL-7B to select videos belonging to the predefined activity list. If this filtering is noisy, the semantic similarity and domain purity of VUDG are compromised. No validation of the filter accuracy is provided.
  • domain assumption The multi-expert annotation pipeline (Gemini-2.5-Flash, DeepSeek-v3, Gemini-2.5-Pro, human review) produces high-quality and unbiased QA pairs.
    The dataset quality rests entirely on this pipeline. The paper does not report human acceptance rates, inter-annotator agreement, or statistics on how many QA pairs were flagged or corrected. LLM-generated questions can contain hallucinations or biases.
  • domain assumption DeepSeek-v3 is a valid judge for open-ended video QA answers.
    Open-ended answers in Section 3.4 are scored by DeepSeek-v3 as an automated judge. The paper provides no human evaluation on a subset to validate judge reliability or agreement. This affects the open-ended results in Table 5.
  • domain assumption The training and testing splits from different source datasets prevent data leakage from LVLM pretraining.
    Section 3.1 claims a check that the test sources do not overlap with the training sources, but provides no details of how the check was performed. Since LVLMs are pretrained on large web corpora, some test videos could still have been seen during pretraining, which would inflate zero-shot scores.
  • domain assumption The t-SNE visualization of question and answer embeddings demonstrates semantic similarity across domains.
    Section 4.4 uses t-SNE of CLIP and BGE embeddings as evidence that question/answer distributions are mixed across domains. Visual inspection of an embedding plot is weak evidence for semantic balance, especially without quantitative distance metrics.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VUDG: A Dataset for Video Understanding Domain Generalization." pith.science (2026). https://pith.science/paper/NH4QOEO7

@misc{pith2026250524346,
  author       = {Pith},
  title        = {Pith review of: VUDG: A Dataset for Video Understanding Domain Generalization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NH4QOEO7}},
  note         = {Machine review of arXiv:2505.24346}
}
read the original abstract

Video understanding has made remarkable progress in recent years, largely driven by advances in deep models and the availability of large-scale annotated datasets. However, existing works typically ignore the inherent domain shifts encountered in real-world video applications, leaving domain generalization (DG) in video understanding underexplored. Hence, we propose Video Understanding Domain Generalization (VUDG), a novel dataset designed specifically for evaluating the DG performance in video understanding. VUDG contains videos from 11 distinct domains that cover three types of domain shifts, and maintains semantic similarity across different domains to ensure fair and meaningful evaluation. We propose a multi-expert progressive annotation framework to annotate each video with both multiple-choice and open-ended question-answer pairs. Extensive experiments on 9 representative large video-language models (LVLMs) and several traditional video question answering methods show that most models (including state-of-the-art LVLMs) suffer performance degradation under domain shifts. These results highlight the challenges posed by VUDG and the difference in the robustness of current models to data distribution shifts. We believe VUDG provides a valuable resource for prompting future research in domain generalization video understanding.

Figures

Figures reproduced from arXiv: 2505.24346 by the authors.

Figure 1
Figure 1. Overview of the proposed VUDG dataset. by not only the domain shift but also the semantic space differences. Hence, the generalization of the model may not be evaluated well. To address this issue, we propose Video Understanding Domain Generalization (VUDG), a dataset specifically designed to evaluate the performance of domain generalization in video understanding. VUDG contains 11 domains with video data collected … view at source ↗
Figure 2
Figure 2. The pipeline diagram of the proposed multi-expert annotation framework. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Overview of various domains and question types in our VUDG dataset. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Statistics showing the domain and question category distributions of QA pairs in [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Statistics illustrating the distribution of video durations across [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: The t-SNE visualization of frame features, question embeddings, and ground-truth answer [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Prompts for download. The list of human daily activity scenes containing 37 items that the VUDG dataset focuses on are detailed in [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: Prompts for generating open-ended answers for Q1, Q2 and Q3. [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: Prompts for generating open-ended answers for Q4 and Q5. [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: Prompts for generating options for Q1, Q2 and Q3. [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]
Figure 11
Figure 11. Figure 11: Prompts for generating options for Q4 and Q5. [PITH_FULL_IMAGE:figures/full_fig_p020_11.png]
Figure 12
Figure 12. Figure 12: Prompts for Checking Q1, Q2 and Q3. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_12.png]
Figure 13
Figure 13. Figure 13: Prompts for Checking Q4 and Q5. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_13.png]
Figure 14
Figure 14. Figure 14: Prompts for Open-ended Evaluation of Q1 to Q3(Action Recognition, Attribute Identifica [PITH_FULL_IMAGE:figures/full_fig_p023_14.png]
Figure 15
Figure 15. Figure 15: Prompts for Open-ended Evaluation of Q4 and Q5(Temporal Understanding) [PITH_FULL_IMAGE:figures/full_fig_p024_15.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

53 extracted references · 27 canonical work pages

  1. [1]

    Mm-vit: Multi-modal video transformer for compressed video action recognition

    Jiawei Chen and Chiu Man Ho. Mm-vit: Multi-modal video transformer for compressed video action recognition. InProceedings of the IEEE/CVF winter conference on applications of computer vision, pages 1910–1921, 2022

    work page 1910

  2. [2]

    Mar: Masked autoencoders for efficient action recognition.IEEE Transactions on Multimedia, 26:218–233, 2023

    Zhiwu Qing, Shiwei Zhang, Ziyuan Huang, Xiang Wang, Yuehuan Wang, Yiliang Lv, Changxin Gao, and Nong Sang. Mar: Masked autoencoders for efficient action recognition.IEEE Transactions on Multimedia, 26:218–233, 2023

    work page 2023

  3. [3]

    Mnv3-mfae: A lightweight network for video action recognition.Electronics, 14(5):981, 2025

    Jie Liu, Wenyue Liu, and Ke Han. Mnv3-mfae: A lightweight network for video action recognition.Electronics, 14(5):981, 2025

    work page 2025

  4. [4]

    Swinbert: End-to-end transformers with sparse attention for video captioning

    Kevin Lin, Linjie Li, Chung-Ching Lin, Faisal Ahmed, Zhe Gan, Zicheng Liu, Yumao Lu, and Lijuan Wang. Swinbert: End-to-end transformers with sparse attention for video captioning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 17949–17958, 2022

    work page 2022

  5. [5]

    End-to-end generative pretraining for multimodal video captioning

    Paul Hongsuck Seo, Arsha Nagrani, Anurag Arnab, and Cordelia Schmid. End-to-end generative pretraining for multimodal video captioning. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 17959–17968, 2022

    work page 2022

  6. [6]

    Text with knowledge graph augmented transformer for video captioning

    Xin Gu, Guang Chen, Yufei Wang, Libo Zhang, Tiejian Luo, and Longyin Wen. Text with knowledge graph augmented transformer for video captioning. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18941–18951, 2023

    work page 2023

  7. [7]

    Automatic video captioning using tree hierarchical deep convolutional neural network and asrnn-bi-directional lstm.Computing, 106(11):3691–3709, 2024

    N Kavitha, K Ruba Soundar, R Karthick, and J Kohila. Automatic video captioning using tree hierarchical deep convolutional neural network and asrnn-bi-directional lstm.Computing, 106(11):3691–3709, 2024

    work page 2024

  8. [8]

    Invariant grounding for video question answering

    Yicong Li, Xiang Wang, Junbin Xiao, Wei Ji, and Tat-Seng Chua. Invariant grounding for video question answering. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2928–2937, 2022

    work page 2022

Show all 53 references
  1. [9]

    From representation to reasoning: Towards both evidence and commonsense reasoning for video question-answering

    Jiangtong Li, Li Niu, and Liqing Zhang. From representation to reasoning: Towards both evidence and commonsense reasoning for video question-answering. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 21273–21282, 2022

    2022

  2. [10]

    Morevqa: Exploring modular reasoning models for video question answering

    Juhong Min, Shyamal Buch, Arsha Nagrani, Minsu Cho, and Cordelia Schmid. Morevqa: Exploring modular reasoning models for video question answering. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13235–13245, 2024

    2024

  3. [11]

    Diversifying spatial-temporal perception for video domain generalization.Advances in Neural Information Processing Systems, 36:56012–56026, 2023

    Kun-Yu Lin, Jia-Run Du, Yipeng Gao, Jiaming Zhou, and Wei-Shi Zheng. Diversifying spatial-temporal perception for video domain generalization.Advances in Neural Information Processing Systems, 36:56012–56026, 2023

    2023

  4. [12]

    A multi-modal egocentric activity recognition approach towards video domain generalization.Sensors, 24(8):2491, 2024

    Antonios Papadakis and Evaggelos Spyrou. A multi-modal egocentric activity recognition approach towards video domain generalization.Sensors, 24(8):2491, 2024

    2024

  5. [13]

    Meta-causal learning for single domain generalization

    Jin Chen, Zhi Gao, Xinxiao Wu, and Jiebo Luo. Meta-causal learning for single domain generalization. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7683–7692, 2023

    2023

  6. [14]

    Tgif-qa: Toward spatio-temporal reasoning in visual question answering

    Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. Tgif-qa: Toward spatio-temporal reasoning in visual question answering. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 2758–2766, 2017. 10

    2017

  7. [15]

    Mvbench: A comprehensive multi-modal video understanding benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195–22206, 2024

    2024

  8. [16]

    Videovista: A versatile benchmark for video understanding and reasoning.arXiv preprint arXiv:2406.11303, 2024

    Yunxin Li, Xinyu Chen, Baotian Hu, Longyue Wang, Haoyuan Shi, and Min Zhang. Videovista: A versatile benchmark for video understanding and reasoning.arXiv preprint arXiv:2406.11303, 2024

    2024 arXiv

  9. [17]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis.arXiv preprint arXiv:2405.21075, 2024

    Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis.arXiv preprint arXiv:2405.21075, 2024

    2024 arXiv

  10. [18]

    Vatex: A large-scale, high-quality multilingual dataset for video-and-language research

    Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. Vatex: A large-scale, high-quality multilingual dataset for video-and-language research. InProceedings of the IEEE/CVF international conference on computer vision, pages 4581–4591, 2019

    2019

  11. [19]

    Multimodal datasets and benchmarks for reasoning about dynamic spatio-temporality in everyday environments.arXiv preprint arXiv:2408.11347, 2024

    Takanori Ugai, Kensho Hara, Shusaku Egami, and Ken Fukuda. Multimodal datasets and benchmarks for reasoning about dynamic spatio-temporality in everyday environments.arXiv preprint arXiv:2408.11347, 2024

    2024 arXiv

  12. [20]

    Internvid: A large-scale video-text dataset for multimodal understanding and generation

    Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, Ping Luo, Ziwei Liu, Yali Wang, Limin Wang, and Yu Qiao. Internvid: A large-scale video-text dataset for multimodal understanding and generation. InThe Twelfth Intern...

    2024

  13. [21]

    Sharegpt4video: Improving video understanding and generation with better captions.Advances in Neural Information Processing Systems, 37:19472– 19495, 2024

    Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Zhenyu Tang, Li Yuan, et al. Sharegpt4video: Improving video understanding and generation with better captions.Advances in Neural Information Processing Systems, 37:19472– 19495, 2024

    2024

  14. [22]

    Qwen2.5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2.5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

    2025 arXiv

  15. [23]

    Activitynet: A large-scale video benchmark for human activity understanding

    Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. InProceedings of the ieee conference on computer vision and pattern recognition, pages 961–970, 2015

    2015

  16. [24]

    The kinetics human action video dataset.arXiv preprint arXiv:1705.06950, 2017

    Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset.arXiv preprint arXiv:1705.06950, 2017

    2017 arXiv

  17. [25]

    Hollywood in homes: Crowdsourcing data collection for activity understanding

    Gunnar A Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. Hollywood in homes: Crowdsourcing data collection for activity understanding. InComputer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Pro...

    2016

  18. [26]

    Tvqa: Localized, compositional video question answering.arXiv preprint arXiv:1809.01696, 2018

    Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L Berg. Tvqa: Localized, compositional video question answering.arXiv preprint arXiv:1809.01696, 2018

    2018 arXiv

  19. [27]

    Video question answering via gradually refined attention over appearance and motion

    Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In Proceedings of the 25th ACM international conference on Multimedia, pages 1645–1653, 2017

    2017

  20. [28]

    Msr-vtt: A large video description dataset for bridging video and language

    Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 5288–5296, 2016

    2016

  21. [29]

    Domain generalization for video anomaly detection considering diverse anomaly types.Signal, Image and Video Processing, 18(4):3691–3704, 2024

    Zhiqiang Wang, Xiaojing Gu, Huaicheng Yan, and Xingsheng Gu. Domain generalization for video anomaly detection considering diverse anomaly types.Signal, Image and Video Processing, 18(4):3691–3704, 2024. 11

    2024

  22. [30]

    Video-audio domain generalization via confounder disentanglement

    Shengyu Zhang, Xusheng Feng, Wenyan Fan, Wenjing Fang, Fuli Feng, Wei Ji, Shuo Li, Li Wang, Shanshan Zhao, Zhou Zhao, et al. Video-audio domain generalization via confounder disentanglement. InProceedings of the AAAI conference on artificial intelligence, volume 37, pages 1532...

    2023

  23. [31]

    Videodg: General- izing temporal relations in videos to novel domains.IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11):7989–8004, 2021

    Zhiyu Yao, Yunbo Wang, Jianmin Wang, Philip S Yu, and Mingsheng Long. Videodg: General- izing temporal relations in videos to novel domains.IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11):7989–8004, 2021

    2021

  24. [32]

    Ani-gifs: A benchmark dataset for domain generalization of action recognition from gifs.Frontiers in Computer Science, 4:876846, 2022

    Shoumik Sovan Majumdar, Shubhangi Jain, Isidora Chara Tourni, Arsenii Mustafin, Diala Lteif, Stan Sclaroff, Kate Saenko, and Sarah Adel Bargal. Ani-gifs: A benchmark dataset for domain generalization of action recognition from gifs.Frontiers in Computer Science, 4:876846, 2022

    2022

  25. [33]

    What can a cook in italy teach a mechanic in india? action recognition generalisation over scenarios and locations

    Chiara Plizzari, Toby Perrett, Barbara Caputo, and Dima Damen. What can a cook in italy teach a mechanic in india? action recognition generalisation over scenarios and locations. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13656– 13666, 2023

    2023

  26. [34]

    Ego4d: Around the world in 3,000 hours of egocentric video

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. InProceedings of the IEEE/CVF conference on computer vision a...

    2022

  27. [35]

    Multimodal motion conditioned diffusion model for skeleton- based video anomaly detection

    Alessandro Flaborea, Luca Collorone, Guido Maria D’Amely Di Melendugno, Stefano D’Arrigo, Bardh Prenkaj, and Fabio Galasso. Multimodal motion conditioned diffusion model for skeleton- based video anomaly detection. InProceedings of the IEEE/CVF international conference on comp...

    2023

  28. [36]

    Activitynet-qa: A dataset for understanding complex web videos via question answering

    Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. InProceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 9127–9134, 2019

    2019

  29. [37]

    Egoschema: A diagnostic benchmark for very long-form video language understanding.Advances in Neural Information Processing Systems, 36:46212–46244, 2023

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding.Advances in Neural Information Processing Systems, 36:46212–46244, 2023

    2023

  30. [38]

    Video-bench: A comprehensive benchmark and toolkit for evaluating video-based large language models.arXiv preprint arXiv:2311.16103, 2023

    Munan Ning, Bin Zhu, Yujia Xie, Bin Lin, Jiaxi Cui, Lu Yuan, Dongdong Chen, and Li Yuan. Video-bench: A comprehensive benchmark and toolkit for evaluating video-based large language models.arXiv preprint arXiv:2311.16103, 2023

    2023 arXiv

  31. [39]

    Tempcompass: Do video llms really understand videos?arXiv preprint arXiv:2403.00476, 2024

    Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. Tempcompass: Do video llms really understand videos?arXiv preprint arXiv:2403.00476, 2024

    2024 arXiv

  32. [40]

    Video-chatgpt: Towards detailed video understanding via large vision and language models, 2024

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models, 2024

    2024

  33. [41]

    Poem: polarization of embeddings for domain-invariant representations

    Sang-Yeong Jo and Sung Whan Yoon. Poem: polarization of embeddings for domain-invariant representations. InProceedings of the AAAI Conference on Artificial Intelligence (AAAI), volume 37, pages 8150–8158, 2023

    2023

  34. [42]

    Video-text as game players: Hierarchical banzhaf interaction for cross-modal representation learning

    Peng Jin, Jinfa Huang, Pengfei Xiong, Shangxuan Tian, Chang Liu, Xiangyang Ji, Li Yuan, and Jie Chen. Video-text as game players: Hierarchical banzhaf interaction for cross-modal representation learning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern R...

    2023

  35. [43]

    Clifton, and Jie Chen

    Peng Jin, JinFa Huang, Fenglin Liu, Xian Wu, Shen Ge, Guoli Song, David A. Clifton, and Jie Chen. Expectation-maximization contrastive learning for compact video-and-language representations. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors,Advances...

    2022

  36. [44]

    Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms.arXiv preprint arXiv:2406.07476, 2024

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, and Lidong Bing. Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms.arXiv preprint arXiv:2406.07476, 2024

    2024 arXiv

  37. [45]

    Minigpt4-video: Advancing multimodal llms for video understanding with interleaved visual-textual tokens.arXiv preprint arXiv:2404.03413, 2024

    Kirolos Ataallah, Xiaoqian Shen, Eslam Abdelrahman, Essam Sleiman, Deyao Zhu, Jian Ding, and Mohamed Elhoseiny. Minigpt4-video: Advancing multimodal llms for video understanding with interleaved visual-textual tokens.arXiv preprint arXiv:2404.03413, 2024

    2024 arXiv

  38. [46]

    Videochat: Chat-centric video understanding.arXiv preprint arXiv:2305.06355, 2023

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wan, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding.arXiv preprint arXiv:2305.06355, 2023

    2023 arXiv

  39. [47]

    Video-llava: Learning united visual representation by alignment before projection.arXiv preprint arXiv:2311.10122, 2023

    Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection.arXiv preprint arXiv:2311.10122, 2023

    2023 arXiv

  40. [48]

    mPLUG-owl3: Towards long image-sequence understanding in multi-modal large language models

    Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mPLUG-owl3: Towards long image-sequence understanding in multi-modal large language models. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  41. [49]

    Video-ccam: Enhancing video-language understanding with causal cross-attention masks for short and long videos, 2024

    Jiajun Fei, Dian Li, Zhidong Deng, Zekun Wang, Gang Liu, and Hui Wang. Video-ccam: Enhancing video-language understanding with causal cross-attention masks for short and long videos, 2024

    2024

  42. [50]

    Videollama 3: Frontier multimodal foundation models for image and video understanding, 2025

    Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, Peng Jin, Wenqi Zhang, Fan Wang, Lidong Bing, and Deli Zhao. Videollama 3: Frontier multimodal foundation models for image and video understanding, 2025

    2025

  43. [51]

    Lora: Low-rank adaptation of large language models.ICLR, 1(2):3, 2022

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models.ICLR, 1(2):3, 2022

    2022

  44. [52]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at...

    2021

  45. [53]

    C- pack: Packed resources for general chinese embeddings

    Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian-Yun Nie. C- pack: Packed resources for general chinese embeddings. InProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), 2024. 13...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.