Pith. sign in

REVIEW 3 major objections 6 minor 4 cited by

LongViTU: Instruction Tuning for Long-Form Video Understanding

T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that LongViTU, a 121k-pair automatically generated video dataset with an average certificate length of 276.8 seconds, is high-quality, and that supervised fine-tuning on it lifts long-video understanding performance by…

desk verdict LongViTU is a genuinely useful dataset with a clever construction pipeline, but the headline 4.6-minute certificate length is a design instruction, not a verified property. read the letter →

arxiv 2501.05037 v2 pith:BC32GMNF submitted 2025-01-09 cs.CV cs.LG

Rujie Wu , Xiaojian Ma , Hai Ci , Yue Fan , Yuxuan Wang , Haozhe Zhao , Qing Li , Yizhou Wang This is my paper · ORCID
classification cs.CVcs.LG
keywords long-formvideounderstandinginstructiontuningquestionansweringdatasetgenerationegocentriccertificatelengthhierarchicalrepresentationsupervisedfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces LongViTU, a machine-generated dataset of roughly 121,000 video question-answer pairs drawn from about 900 hours of egocentric video, and argues that it is a high-quality resource for training models to understand long videos. The dataset is engineered so that each question genuinely needs a long temporal context: on average, a model must watch 276.8 seconds of video to answer, far longer than most existing benchmarks. The paper reports that fine-tuning two open video-language models on LongViTU improves their average performance by 2.5% and 3.7% across four long-video benchmarks, and that human reviewers rate the generated questions as close in quality to a manually annotated benchmark. The claim, in short, is that long-form video understanding can be advanced at scale by automatically synthesizing instruction data from video summaries rather than hand-annotating it.

What carries the argument

The central object is the hierarchical video tree, which organizes about 900 hours of egocentric video into frame-level dense captions, event-level summaries, and segment-level summaries, each anchored with explicit start and end timestamps. QA generation runs on a five-segment sliding window with a strict dependency rule: the answer must come from the first three segments and the question from the last two, guaranteeing long certificate lengths. A self-revision stage uses pure-text evaluation to discard QA pairs that can be answered without watching the video.

What would settle it

A concrete check: take the LongViTU test set, give a strong model only the last two segments—the 'ask content'—or only a text transcript of the full video, and measure GPT-4 scores; if scores remain high, the claimed certificate length collapses. Alternatively, re-run the human rubric evaluation on a much larger random sample than the 100 questions used and count the share of questions judged answerable without the earlier segments.

Watch

Extended reading notes

Core claim

The central discovery is that a hierarchical, tree-structured representation of a long video—dense frame captions condensed into event descriptions, then merged into segment summaries—lets an LLM generate QA pairs whose certificate length averages 276.8 seconds. The paper's key mechanism forces long dependency: the question is posed using only the last two of five segments, while the answer must be extracted from the first three, so a model must look back several minutes of video. Combined with a reasoning taxonomy that pushes questions into categories like causality, planning, and risk, and a self-revision pass that filters text-only answerable pairs, the pipeline produces data on which supervised fine-tuning yields gains on both in-distribution and out-of-distribution long-video benchmarks. The human study places the best fine-tuned model at a GPT-4 score of 55.9 against a human score of 81.0, evidence that the questions remain hard.

Load-bearing premise

The whole pipeline assumes that the text summaries fed to the LLM preserve enough visual and temporal detail that the generated questions genuinely depend on watching the long video, rather than being answerable from language priors or from the later segments alone; the paper concedes around 9% of pairs retain textual bias.

Editorial extensions

If this is right

  • Fine-tuning on LongViTU improves EgoSchema accuracy by 4.7% for LongVU and 9.6% for LLaVA-Video, with larger relative gains on longer video subsets.
  • LongViTU questions remain far from solved: the best fine-tuned open model scores 55.9 versus a human 81.0, and the proprietary Gemini-1.5-Pro scores only 52.3 zero-shot.
  • Every QA pair carries explicit timestamps for the events it refers to, enabling future work on temporal grounding and localization in long videos.
  • The self-revision filter and structured reasoning taxonomy reduce, though do not eliminate, textual bias; the paper reports about 9% of QA pairs still retain such bias.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If long-form instruction data can be synthesized this way, the bottleneck shifts from annotation cost to the quality of base video captions, so stronger caption models should directly raise the ceiling of generated QA quality.
  • Certificate length could be used as a training signal: preferentially sampling QA pairs with longer certificate lengths may yield even larger gains for long-video capabilities than the uniform dataset does.
  • The tree-and-window recipe may transfer to other long-horizon domains, such as long audio streams or embodied trajectories, wherever hierarchical summaries of the input can be built.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces LongViTU, a large-scale automatically generated video QA dataset built from Ego4D videos. The construction pipeline extracts frame-, event-, and segment-level descriptions into a hierarchical video tree, then prompts GPT-4 to generate QA pairs with a sliding window that places answers in the first three segments ('Memory Content') and questions in the last two ('Ask Content'), followed by an LLM self-revision step. The paper claims an average certificate length of 276.8 seconds, explicit timestamp annotations, a condensed-reasoning taxonomy, and high quality validated by human studies. It further reports that supervised fine-tuning of LongVU and LLaVA-Video on LongViTU improves average performance by 2.5% and 3.7% respectively on EgoSchema, VideoMME-Long, MLVU, and LVBench, and it introduces a 600-question human-reviewed test benchmark on which current models score far below human annotators.

Significance. If the certificate-length and quality claims are reliable, LongViTU is a potentially valuable community resource: it is among the first large-scale automatically generated long-video QA datasets with explicit timestamps and a structured reasoning taxonomy, and the reported OOD gains on MLVU and LVBench support the practical utility of the data. The paper's strengths include a clearly described pipeline, reproducible fine-tuning settings, comparisons with both open and proprietary models, and human studies on a subset of the data. However, the central distinguishing property—the 276.8-second certificate length—is prompt-enforced rather than directly verified, and the SFT gains are small in several reported conditions, with selective reporting for MVBench and no significance testing. These issues are load-bearing for the paper's main claims and require revision.

major comments (3)
  1. [Section 2.1.2, Appendix C.2] The central distinguishing property, the 276.8-second average certificate length, is instructed by the prompt but never measured. The prompt asks the LLM to place answers in Memory Content and questions in Ask Content, but no check establishes that the question is unanswerable from Ask Content alone or from language priors. The self-revision step (Appendix C.3) is a pure-text evaluation, and Appendix A concedes that roughly 9% of QA pairs retain textual bias. Because human review covers only the 600-question test set and the rubric study uses only 100 questions, the 121k-pair training distribution is not certified. Please add a direct measurement of certificate length: for a random sample of QA pairs, have human raters and a strong blind LLM answer with Ask Content only versus full Memory+Ask content, and report the fraction of questions whose answers require the earlier segments. Without this, the dataset's advertised long-context property remains an assumption rather than an established characteristic.
  2. [Table 3, Section 3.3] The claim of 'substantial performance improvements across nearly all' benchmarks is weakened by selective reporting and small effects. The MVBench caption states that the table 'only shows the subc-category that have shown improvement', which is a clear selection bias; the full MVBench results must be reported. In addition, several reported subsets decline, including VideoMME Short (-0.1 for LLaVA-Video, -9.5 for Video-LLaVA), LVBench Summarization (-6.3 for LongVU), MLVU Anomaly Reco. (-1.3 for LongVU), and OpenEQA (-7.1/-12.6 for Video-LLaVA). No error bars, confidence intervals, or significance tests are provided, so gains such as +0.4% on VideoMME Long for LongVU and +0.6% on LVBench average for LongVU are not distinguishable from noise. Please report complete result tables, including all MVBench categories, and add statistical significance measures or per-seed variability.
  3. [Section 2.3, Figure 4] The human quality assessment is too small to support the strong wording that the results 'prove the quality' of LongViTU. Only 100 randomly selected questions were rubric-scored, yielding 46% 'Good', 45% 'Fair', and 9% 'Poor', and the comparative assessment against VideoMME also uses only 100 questions per dataset. These samples cannot certify the full 121k-pair corpus, especially because the 600 human-reviewed samples are limited to the test set. Please report confidence intervals for the observed proportions, scale up the human evaluation, or temper the conclusion to reflect that the quality evidence comes from a small sample.
minor comments (6)
  1. [Abstract] The abstract contains a doubled closing parenthesis 'etc.)).'; please fix the punctuation.
  2. [Table 2] The 'Overall Avg.' for Human is 81.0, but the mean of the three category averages (84.1, 74.3, and 75.5) is approximately 78.0; please clarify how the overall average is computed.
  3. [Section 3.3] The text says 'LLaVA-Video SFT improved by 1% on VideoMME Long', but Table 3 reports +0.3% for that condition; please align the text with the table.
  4. [Figure 3a] The caption refers to a 'bottom horizontal axis' while the figure appears to have both top and bottom horizontal axes with different scales; please make the axis mapping explicit and improve readability.
  5. [Table 3] The MVBench caption contains the typo 'subc-category'; please correct it to 'sub-category'.
  6. [Section 3.3] The word 'significant' is used repeatedly in a non-statistical sense; please reserve it for cases where significance tests are actually reported.

Circularity Check

1 steps flagged · score 6.0 of 10

LongViTU's 276.8s certificate-length claim is prompt-enforced by construction; SFT gains on external benchmarks remain independently measured.

  1. self definitional [Section 2.1.2 (Ensuring Long Certificate Length); Section 2.2 (QA duration statistics); Appendix A limitation; Section C.2 prompt]
    "we further design the prompt to instructs the LLM to formulate questions based solely on information in the last two segments (Ask Content), while requiring answers exclusively found in the first three segments (Memory Content, see full prompt in Section C.2). This design explicitly ensures that generated questions necessitate referencing substantial preceding video context to guarante certificate lengths."

    The headline property 'average certificate length of 276.8s' is presented as evidence that LongViTU requires long-term context, but it is not measured or externally verified; it is the temporal span produced by instructing the LLM to place answers in the first three of five window segments and the question in the last two. The reported QA duration/certificate length is thus the prompt instruction itself by construction. The self-revision 'pure-text evaluation' can filter only obvious textual leakage; it cannot establish that the question is unanswerable from the Ask Content alone or from language priors.

full rationale

The paper's central empirical result — SFT on LongViTU improving LongVU and LLaVA-Video by 2.5% and 3.7% on EgoSchema, VideoMME-Long, MLVU, and LVBench — is a genuine external measurement: those benchmarks are human-annotated and outside the GPT-4 generation/scoring loop, and the gains are not fitted parameters. The human quality study also provides independent (if small-sample) support. However, one headline claim is circular: the 276.8s average certificate length is not verified as 'context required to answer' but is constructed by the Ask Content/Memory Content split in the QA-generation prompt, so the paper's central 'long-context' selling point reduces to its own instruction. The use of GPT-4 both to generate answers and to score model predictions on the LongViTU test set is a stylistic-alignment risk, but it does not affect the external-benchmark comparison, so it is noted rather than scored as circularity. No load-bearing self-citation was found.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claims rest on the accuracy of Ego4D annotations, the fidelity of automatic dense captions, the reliability of GPT-4 as generator and evaluator, and the design assumption that the sliding window structure forces long-range reasoning.

free parameters (2)
  • Sliding window size = 5 segments
    Chosen by hand; directly controls average certificate length (276.8s) and the amount of context the LLM sees.
  • Ask/Memory split = 2 segments / 3 segments
    The prompt instructs the LLM to ask about the last two segments and answer from the first three; this ratio was designed, not derived, to enforce long certificate lengths.
assumptions (4)
  • domain assumption Ego4D human event annotations are temporally accurate.
    The tree structure and QA timestamps inherit Ego4D's annotated event boundaries; if these are wrong, the certificate-length and timestamp claims weaken.
  • domain assumption InternLM-XComposer2 dense captions capture the visual details needed for QA.
    Frame-level captions are the only visual input to the pipeline; missed objects or actions would produce unfaithful QA pairs.
  • domain assumption GPT-4 scoring is a valid measure of answer correctness.
    All LongViTU benchmark scores come from GPT-4-judge with a rubric; no human-score calibration is reported.
  • ad hoc to paper The reasoning taxonomy is a meaningful decomposition of video reasoning.
    The taxonomy is designed by the authors and used to prompt QA generation; its completeness and orthogonality are not empirically validated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LongViTU: Instruction Tuning for Long-Form Video Understanding." pith.science (2026). https://pith.science/paper/BC32GMNF

@misc{pith2026250105037,
  author       = {Pith},
  title        = {Pith review of: LongViTU: Instruction Tuning for Long-Form Video Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BC32GMNF}},
  note         = {Machine review of arXiv:2501.05037}
}
read the original abstract

This paper introduces LongViTU, a large-scale (~121k QA pairs, ~900h videos), automatically generated dataset for long-form video understanding. We propose a systematic approach that organizes videos into a hierarchical tree structure for QA generation and incorporates self-revision mechanisms to ensure high-quality QA pairs. Each QA pair in LongViTU features: 1) long-term context (average certificate length of 4.6 minutes); 2) rich knowledge and condensed reasoning (commonsense, causality, planning, etc.)). We also offer explicit timestamp annotations of relevant events for each QA pair. We have conducted extensive human studies on LongViTU, and the results prove the quality of our dataset. To better evaluate the challenges posed by LongViTU's emphasis on long-term context and condensed reasoning, we manually curate a subset of LongViTU into a benchmark. Evaluations using a state-of-the-art open-source model (LongVU), a proprietary model (Gemini-1.5-Pro), and human annotators yield GPT-4 scores of 49.9, 52.3, and 81.0, respectively, underscoring the substantial difficulty presented by LongViTU questions. Performing supervised fine-tuning (SFT) of LongVU and LLaVA-Video on LongViTU data results in average performance gains of 2.5% and 3.7%, respectively, across a suite of long video understanding benchmarks (EgoSchema, VideoMME-Long, MLVU, LVBench).

Figures

Figures reproduced from arXiv: 2501.05037 by the authors.

Figure 1
Figure 1. Illustration of LongViTU. The top row shows an example video sequence, with red boxes highlighting key clues for the posed question and yellow boxes marking objects in key frames related to the answer. The middle row emphasizes the primary advantages of our proposed LongViTU over previous datasets, along with QA examples; refer to Section 1 for further details. The bottom row displays predictions from canonical open… view at source ↗
Figure 2
Figure 2. The construction of LongViTU. We design an automatic pipeline to generate QA pairs from Ego4D [19] videos while addressing the challenge of long-form video comprehension. To mitigate the context length limitation of LLMs and grasp spatial-temporal information in long video context, we employ a hierarchical tree structure. This structure first condenses dense captions to refine event descriptions and then segments co… view at source ↗
Figure 3
Figure 3. Statistics of LongViTU. Subfigure (a) depicts the distributions of video and QA durations within LongViTU. The bottom horizontal axis (from left to right) represents QA duration in seconds, while the right vertical axis (from bottom to top) shows the percentage of the total dataset. QA durations predominantly vary from 6 to 600 seconds, with an average of 276.8 seconds. The top horizontal axis (from right to left) d… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Quality Assessment. Rubric-based assessment of LongViTU and comparative assessment against VideoMME [16]. 3. Experiments In our experiments, we aim to answer the following: 1. Do LongViTU questions indeed impose substantial chal￾lenges (long-video context and condensed…
Figure 6
Figure 6. Figure 6: Epoch Ablation. Performance improvements during fine-tuning diminish as training epochs approach the full scale of LongViTU. B.2. Qualitative Evaluation To facilitate a more comprehensive qualitative analysis, we provide visualizations of various question-answering sce…
Figure 5
Figure 5. Figure 5: Duration Ablation. Regardless of whether in the zero￾shot or fine-tuned setting on LongViTU, LongVU demonstrates a performance decline on longer subsets [300, 600) compared to shorter ones [60, 300) and [0, 60). Training Epochs [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 7
Figure 7. Figure 7: Qualitative Results. The yellow box indicates the key frame that contains the answer, while the red box highlights the relevant objects. For better illustration, only concise key information is presented in the predictions. 2 [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: LongViTU Examples. For better illustration, only concise key information is presented in the predictions. 8 [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Strefer: Empowering Video LLMs with Space-Time Referring and Reasoning via Synthetic Instruction Data

    cs.CV 2025-09 conditional novelty 6.0 of 10

    Adding Strefer's synthetic space-time reference questions to video instruction tuning improves mask-referred description/QA, timestamp QA, and temporal reasoning over a video-LLM baseline.

  2. ViaRL: Adaptive Temporal Grounding via Visual Iterated Amplification Reinforcement Learning

    cs.CV 2025-05 conditional novelty 6.0 of 10

    ViaRL uses rule-based reinforcement learning to train a frame selector for video QA, improving Qwen2.5-VL on VideoMME, LVBench, and MLVU by several points.

  3. Less Data, Faster Convergence: Goal-Driven Data Optimization for Multimodal Instruction Tuning

    cs.CV 2026-03 conditional novelty 5.5 of 10

    Goal-driven selection of 1× multimodal instruction subsets reaches a 512k Uni-10x baseline after ~27–35k samples and improves accuracy by up to +3.08 pp under a fixed Qwen3-VL recipe.

  4. Infinite Video Understanding

    cs.CV 2025-07 conditional novelty 3.0 of 10

    The paper argues that video understanding research should aim at processing streams of arbitrary, unbounded duration and outlines the challenges, directions, and metrics needed.

Reference graph

Works this paper leans on

72 extracted references · 24 canonical work pages · cited by 4 Pith papers

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. 7

    arXiv 2023

  2. [2]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716–23736,

  3. [3]

    Palm 2 technical report

    Rohan Anil, Andrew M Dai, Orhan Firat, Melvin John- son, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023

    arXiv 2023

  4. [4]

    Language models are few-shot learners

    Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020. 7

    arXiv 2005

  5. [5]

    Matryoshka multimodal models

    Mu Cai, Jianwei Yang, Jianfeng Gao, and Yong Jae Lee. Matryoshka multimodal models. arXiv preprint arXiv:2405.17430, 2024. 7

    arXiv 2024

  6. [6]

    Internlm2 technical report

    Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. Internlm2 technical report. arXiv preprint arXiv:2403.17297, 2024. 7

    arXiv 2024

  7. [7]

    Videollm-online: Online video large language model for streaming video

    Joya Chen, Zhaoyang Lv, Shiwei Wu, Kevin Qinghong Lin, Chenan Song, Difei Gao, Jia-Wei Liu, Ziteng Gao, Dongxing Mao, and Mike Zheng Shou. Videollm-online: Online video large language model for streaming video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18407–18418, 2024. 7

    work page 2024

  8. [8]

    Sharegpt4video: Improving video understand- ing and generation with better captions

    Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, et al. Sharegpt4video: Improving video understand- ing and generation with better captions. arXiv preprint arXiv:2406.04325, 2024. 7

    arXiv 2024

Show all 72 references
  1. [9]

    An image is worth 1/2 tokens after layer 2: Plug-and-play inference ac- celeration for large vision-language models

    Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Jun- yang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference ac- celeration for large vision-language models. arXiv preprint arXiv:2403.06764, 2024. 7

    2024 arXiv

  2. [10]

    Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.07476, 2024. 7

    2024 arXiv

  3. [11]

    Scannet: Richly-annotated 3d reconstructions of indoor scenes

    Angela Dai, Angel X Chang, Manolis Savva, Maciej Hal- ber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5828–5839, 2017. 2

    2017

  4. [12]

    Instructblip: Towards general-purpose vision- language models with instruction tuning, 2023

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision- language models with instruction tuning, 2023. 7

    2023

  5. [13]

    Internlm-xcomposer2: Mastering free-form text-image composition and compre- hension in vision-language large model

    Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, et al. Internlm-xcomposer2: Mastering free-form text-image composition and compre- hension in vision-language large model. arXiv preprint arXiv:2401.1...

    2024 arXiv

  6. [14]

    Egovqa-an egocentric video question answer- ing benchmark dataset

    Chenyou Fan. Egovqa-an egocentric video question answer- ing benchmark dataset. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops , pages 0–0, 2019. 2

    2019

  7. [15]

    Videoagent: A memory-augmented multimodal agent for video understanding

    Yue Fan, Xiaojian Ma, Rujie Wu, Yuntao Du, Jiaqi Li, Zhi Gao, and Qing Li. Videoagent: A memory-augmented multimodal agent for video understanding. arXiv preprint arXiv:2403.11481, 2024. 6

    2024 arXiv

  8. [16]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075, 2024. 4, 5, 6, 8

    2024 arXiv

  9. [17]

    Env-qa: A video question answering benchmark for comprehensive understanding of dynamic environments

    Difei Gao, Ruiping Wang, Ziyi Bai, and Xilin Chen. Env-qa: A video question answering benchmark for comprehensive understanding of dynamic environments. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1675–1685, 2021. 2

    2021

  10. [18]

    Chatglm: A family of large language models from glm-130b to glm-4 all tools

    Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, et al. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793, 2024. 7

    2024 arXiv

  11. [19]

    Ego4d: Around the world in 3,000 hours of egocentric video

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision ...

    2022

  12. [20]

    Ma-lmm: Memory-augmented large multimodal model for long-term video understanding

    Bo He, Hengduo Li, Young Kyun Jang, Menglin Jia, Xue- fei Cao, Ashish Shah, Abhinav Shrivastava, and Ser-Nam Lim. Ma-lmm: Memory-augmented large multimodal model for long-term video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni...

    2024

  13. [21]

    Lemma: A multi-view dataset for le arning m ulti-agent m ulti-task a ctivities

    Baoxiong Jia, Yixin Chen, Siyuan Huang, Yixin Zhu, and Song-chun Zhu. Lemma: A multi-view dataset for le arning m ulti-agent m ulti-task a ctivities. In European Conference on Computer Vision, pages 767–786. Springer, 2020. 2

    2020

  14. [22]

    Egotaskqa: Understanding human tasks in egocentric videos

    Baoxiong Jia, Ting Lei, Song-Chun Zhu, and Siyuan Huang. Egotaskqa: Understanding human tasks in egocentric videos. Advances in Neural Information Processing Systems, 35:3343– 3360, 2022. 2 9

    2022

  15. [23]

    Chat-univi: Unified visual representation em- powers large language models with image and video un- derstanding

    Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation em- powers large language models with image and video un- derstanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 137...

    2024

  16. [24]

    Ai2-thor: An interactive 3d environment for visual ai

    Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Matt Deitke, Kiana Ehsani, Daniel Gordon, Yuke Zhu, et al. Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint arXiv:1712.05474,

    arXiv

  17. [25]

    Mimic-it: Multi-modal in-context instruction tuning

    Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Mimic-it: Multi-modal in-context instruction tuning. arXiv preprint arXiv:2306.05425, 2023. 2

    2023 arXiv

  18. [26]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chun- yuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 7

    2024 arXiv

  19. [27]

    Blip- 2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip- 2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730–19742. PMLR,

  20. [28]

    Videochat: Chat-centric video understanding

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355, 2023. 2, 7

    2023 arXiv

  21. [29]

    Mvbench: A comprehensive multi-modal video understand- ing benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understand- ing benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195–...

    2024

  22. [30]

    Llama-vid: An image is worth 2 tokens in large language models

    Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. arXiv preprint arXiv:2311.17043, 2023. 6, 5, 7

    2023 arXiv

  23. [31]

    Video-llava: Learning united visual represen- tation by alignment before projection

    Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual represen- tation by alignment before projection. arXiv preprint arXiv:2311.10122, 2023. 6, 5, 7

    2023 arXiv

  24. [32]

    Llava-next: Improved reason- ing, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reason- ing, ocr, and world knowledge, 2024. 7

    2024

  25. [33]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 7

    2024

  26. [34]

    Video-chatgpt: Towards detailed video understanding via large vision and language models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424, 2023. 2, 7

    2023 arXiv

  27. [35]

    Openeqa: Embodied question answering in the era of foundation models

    Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Mc- vay, Oleksandr Maksymets, Sergio Arnaud, et al. Openeqa: Embodied question answering in the era of foundation models. In Proceedings of the IEEE/CVF Conference on...

    2024

  28. [36]

    Egoschema: A diagnostic benchmark for very long- form video language understanding

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long- form video language understanding. Advances in Neural Information Processing Systems, 36, 2024. 2, 6, 8

    2024

  29. [37]

    Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730–27744,

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Car- roll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:277...

  30. [38]

    Streaming long video understanding with large language models

    Rui Qian, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Shuangrui Ding, Dahua Lin, and Jiaqi Wang. Streaming long video understanding with large language models. arXiv preprint arXiv:2405.16009, 2024. 7

    2024 arXiv

  31. [39]

    Habitat-matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai

    Santhosh K Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alex Clegg, John Turner, Eric Un- dersander, Wojciech Galuba, Andrew Westbury, Angel X Chang, et al. Habitat-matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai. arXiv prepri...

    2021 arXiv

  32. [40]

    Cinepile: A long video question answering dataset and benchmark

    Ruchit Rawal, Khalid Saifullah, Ronen Basri, David Jacobs, Gowthami Somepalli, and Tom Goldstein. Cinepile: A long video question answering dataset and benchmark. arXiv preprint arXiv:2405.08813, 2024. 2

    2024 arXiv

  33. [41]

    Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context

    Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrit- twieser, et al. Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context. arXiv...

    2024 arXiv

  34. [42]

    Llava-prumerge: Adaptive token reduction for efficient large multimodal models

    Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. Llava-prumerge: Adaptive token reduction for efficient large multimodal models. arXiv preprint arXiv:2403.15388,

  35. [43]

    Longvu: Spa- tiotemporal adaptive compression for long video-language understanding

    Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Bal- akrishnan Varadarajan, Florian Bordes, et al. Longvu: Spa- tiotemporal adaptive compression for long video-language understanding. arXiv preprint arXiv:2410.17434, 20...

    arXiv

  36. [44]

    Moviechat: From dense token to sparse memory for long video understanding

    Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, et al. Moviechat: From dense token to sparse memory for long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patt...

    2024

  37. [45]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063,

  38. [46]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. 7

    2023 arXiv

  39. [47]

    Llama: 10 Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Mar- tinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Roz- ière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: 10 Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 7

    2023 arXiv

  40. [48]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Am- jad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. 7

    2023 arXiv

  41. [49]

    Lvbench: An extreme long video understanding benchmark

    Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Shiyu Huang, Bin Xu, Yuxiao Dong, Ming Ding, et al. Lvbench: An extreme long video understanding benchmark. arXiv preprint arXiv:2406.08035, 2024. 6, 8, 7

    2024 arXiv

  42. [50]

    Internvideo: General video foundation models via generative and discriminative learning

    Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, et al. Internvideo: General video foundation models via generative and discriminative learning. arXiv preprint arXiv:2212.03191,

    arXiv

  43. [51]

    Videollamb: Long-context video understanding with recurrent memory bridges

    Yuxuan Wang, Cihang Xie, Yang Liu, and Zilong Zheng. Videollamb: Long-context video understanding with recurrent memory bridges. arXiv preprint arXiv:2409.01071, 2024. 7

    2024 arXiv

  44. [52]

    Bongard- openworld: Few-shot reasoning for free-form visual concepts in the real world

    Rujie Wu, Xiaojian Ma, Qing Li, Wei Wang, Zhenliang Zhang, Song-Chun Zhu, and Yizhou Wang. Bongard- openworld: Few-shot reasoning for free-form visual concepts in the real world. arXiv preprint arXiv:2310.10207, 2023. 7

    2023 arXiv

  45. [53]

    Next-qa: Next phase of question-answering to explaining tem- poral actions

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question-answering to explaining tem- poral actions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9777–9786,

  46. [54]

    Ryoo, and David J

    Mingze Xu, Chenyou Fan, Yuchen Wang, Michael S. Ryoo, and David J. Crandall. Joint person segmentation and iden- tification in synchronized first- and third-person videos. In European Conference on Computer Vision (ECCV), 2018. 2

    2018

  47. [55]

    Longvila: Scaling long-context visual language models for long videos

    Fuzhao Xue, Yukang Chen, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhi- jian Liu, et al. Longvila: Scaling long-context visual language models for long videos. arXiv preprint arXiv:2408.10188,

    arXiv

  48. [56]

    Baichuan 2: Open large-scale language models

    Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305, 2023. 7

    2023 arXiv

  49. [57]

    Qwen2 technical report

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024. 7

    2024 arXiv

  50. [58]

    Panoptic video scene graph generation

    Jingkang Yang, Wenxuan Peng, Xiangtai Li, Zujin Guo, Liangyu Chen, Bo Li, Zheng Ma, Kaiyang Zhou, Wayne Zhang, Chen Change Loy, et al. Panoptic video scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18675– 1868...

    2023

  51. [59]

    Mm-ego: Towards building egocentric multimodal llms

    Hanrong Ye, Haotian Zhang, Erik Daxberger, Lin Chen, Zongyu Lin, Yanghao Li, Bowen Zhang, Haoxuan You, Dan Xu, Zhe Gan, et al. Mm-ego: Towards building egocentric multimodal llms. arXiv preprint arXiv:2410.07177, 2024. 2

    2024 arXiv

  52. [60]

    Activitynet-qa: A dataset for understanding complex web videos via question answering

    Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 9127–9134, 2019. 2

    2019

  53. [61]

    Video-llama: An instruction-tuned audio-visual language model for video un- derstanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video un- derstanding. arXiv preprint arXiv:2306.02858, 2023. 7

    2023 arXiv

  54. [62]

    Movqa: A benchmark of versatile question-answering for long-form movie understanding

    Hongjie Zhang, Yi Liu, Lu Dong, Yifei Huang, Zhen-Hua Ling, Yali Wang, Limin Wang, and Yu Qiao. Movqa: A benchmark of versatile question-answering for long-form movie understanding. arXiv preprint arXiv:2312.04817, 2023. 2

    2023 arXiv

  55. [63]

    Long context transfer from language to vision

    Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv preprint arXiv:2406.16852, 2024. 6, 7

    2024 arXiv

  56. [64]

    Video instruction tuning with synthetic data

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713, 2024. 2, 6, 5

    2024 arXiv

  57. [65]

    Worldqa: Multimodal world knowledge in videos through long-chain reasoning

    Yuanhan Zhang, Kaichen Zhang, Bo Li, Fanyi Pu, Christo- pher Arif Setiadharma, Jingkang Yang, and Ziwei Liu. Worldqa: Multimodal world knowledge in videos through long-chain reasoning. arXiv preprint arXiv:2405.03272, 2024. 2, 6

    2024 arXiv

  58. [66]

    Mlvu: A comprehensive benchmark for multi-task long video understanding

    Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding. arXiv preprint arXiv:2406.04264, 2024. 6, 8

    2024 arXiv

  59. [67]

    a plant on the windowsill

    Xingyi Zhou, Anurag Arnab, Shyamal Buch, Shen Yan, Austin Myers, Xuehan Xiong, Arsha Nagrani, and Cordelia Schmid. Streaming dense video captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18243–18252, 2024. 7 11 LongViTU: In...

    2024

  60. [68]

    Review event descriptions and group consecutive events that are closely related into a segment

  61. [69]

    index" represents order of event,

    For each group of events, write a brief summary. "index" represents order of event, "event" outlines this moment. Video Content: {video content} Return each segment in JSON format: "start": start index, "end": end index, "segment": brief descrip- tion of video segment. Assembl...

    2024

  62. [70]

    For each selected event or sequence of events, generate question-answer pairs that reflect their spatial and temporal characteristics

    Selection of Events: Select either a single specific event or a series of interrelated events from the video content (’Memory Content’). For each selected event or sequence of events, generate question-answer pairs that reflect their spatial and temporal characteristics. Use "...

  63. [71]

    These pairs should mimic a retrospective dialogue between me and an AI assistant, where I pose questions and the AI provides answers based on the video content

    Creation of Question-Answer Pairs: From the selected events, formulate questions that will be posed later in the video related to a single, specific event (’Ask Content’). These pairs should mimic a retrospective dialogue between me and an AI assistant, where I pose questions ...

  64. [72]

    memory": [list of memory events index],

    Categorization of Questions: Categorize each question under a specific type such as: Object, Attribute, Location, Action, Function, Affordance, Comparison, Relationship, Causality, Motivation, Planning, Risk, or any other category you suggest. Output Format: Return question-an...

    2024

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.