Pith. sign in

REVIEW 4 major objections 5 minor 63 references

IQViC: In-context, Question Adaptive Vision Compressor for Long-term Video Understanding LMMs

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper proposes a question-conditioned visual compressor, IQViC, that stores only task-relevant context and claims to beat prior long-video QA models with fewer memory tokens.

desk verdict The compressor idea is worth your time, but the headline long-video result sits on a self-filtered benchmark that likely does not isolate visual understanding, so treat the 4.8-point claim with caution. read the letter →

arxiv 2412.09907 v2 pith:JGMMLS6F submitted 2024-12-13 cs.CV

Sosuke Yamao , Natsuki Miyahara , Yuki Harazono , Shun Takeuchi This is my paper · ORCID
classification cs.CV
keywords long-termvideounderstandingvisualcompressorquestion-conditionedcompressioncontextmemoryquestionansweringtokenslargemultimodalmodelsInfiniBench-Vision
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tackles long-term video question answering, where full frame sequences overflow the language model's context window and memory banks blur details. It proposes IQViC, a visual compressor that takes each video frame together with the text question and outputs a small set of context tokens, so the bounded context memory stores only question-relevant information. On a newly filtered 599-question subset of InfiniBench called InfiniBench-Vision, containing 265 videos averaging 49 minutes, the authors report 49.1% accuracy and a 2.6 score using 640 memory tokens, against Flash-VStream's 44.3% and 2.4 at 681 tokens. They also report top scores on NExT-QA, MSVD-QA, and MSRVTT-QA, and an ablation showing the compressor beats average pooling by more than ten accuracy points at the same token budget. The intended payoff is that selective, question-conditioned compression rather than larger memory banks is the efficient route to long-video understanding.

What carries the argument

The IQViC visual compressor is a transformer-based encoder, adapted from the in-context autoencoder idea, that concatenates the text prompt embedding, the frame's visual embedding, and a learnable context embedding lookup; the output states of the context tokens after self-attention are the compressed context. The context memory stores these C=64-token embeddings in L=10 slots, and when the memory is full a temporal compressor removes redundancy by averaging the pair of adjacent embeddings with the highest cosine similarity. The load-bearing mechanism is that attention across question, visual, and context tokens lets the context tokens carry only the visual content the question needs, so no pooling or downsampling of the full frame is required.

What would settle it

Have human annotators judge each of the 599 InfiniBench-Vision questions as answerable or not from video frames alone, then rerun MovieChat, MA-LMM, Flash-VStream, and IQViC only on the questions marked answerable; if the accuracy gap over Flash-VStream shrinks below four points, the claimed advantage is an artifact of the filtering.

Watch

Extended reading notes

Core claim

The central claim is that conditioning visual compression on the question preserves the information needed to answer that question while discarding what is irrelevant, and that this is enough to outperform memory-augmented models that try to retain the whole video. The model encodes each frame with a frozen CLIP ViT-L/14 encoder, projects it, concatenates the text prompt and C=64 learnable context tokens, and reads the context-token outputs from a LoRA-tuned transformer compressor as the frame's compressed memory. A parameter-free temporal compressor merges the most similar adjacent memory entries when the L=10 slots fill. With this design, IQViC reaches 49.1% accuracy and a 2.6 score on InfiniBench-Vision using 640 total memory tokens, a 4.8-point accuracy gain over Flash-VStream at 681 tokens, and it achieves scores of 4.0 on NExT-QA, 4.0 on MSVD-QA, and 3.5 on MSRVTT-QA. The ablation on LLaVA-Bench in the wild shows 55.4% accuracy at C=64, more than ten points above average pooling at the same 11% compression ratio.

Load-bearing premise

The load-bearing assumption is that deleting questions flagged as containing non-identifiable character names by an automated name detector leaves a benchmark that is genuinely answerable from video frames alone, and no human validation is provided for that filtering.

Editorial extensions

If this is right

  • Long-video QA can run with a fixed 640-token context memory instead of storing thousands of frame tokens, making inference and memory usage predictable.
  • The same compressor generalizes to short videos: IQViC reports the best score on NExT-QA, MSVD-QA, and MSRVTT-QA among compared methods.
  • Context-token count is a tunable cost-accuracy knob, with C=1 already giving 49.9% on LLaVA-Bench in the wild and C=64 giving 55.4%.
  • Two-stage LoRA training, image-QA for the compressor and video-QA for the decoder, avoids end-to-end video backpropagation while preserving accuracy.
  • Online QA is possible because context-memory construction can be parallelized with response generation, as in Flash-VStream.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because InfiniBench-Vision was built by deleting questions whose character names a language model judged non-visual, the reported 4.8-point lead is a claim about the filtered subset; re-running the same comparison on the unfiltered InfiniBench questions would show how much of the advantage depends on that filtering.
  • The authors leave implicit that the compressor is re-run on the whole video for every new question; a dynamic memory update that only re-encodes unseen frames would reduce per-question cost and is a natural architectural extension.
  • Extending IQViC to audio and subtitles, which the paper names as future work, would target exactly the questions removed by the name filter and could broaden the method to full multimodal long-video understanding.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes IQViC, an in-context, question-adaptive visual compressor for long-term video understanding in large multimodal models (LMMs). The compressor is a transformer-based module that, conditioned on the text question, compresses visual embeddings of each frame into a small set of context tokens stored in a context memory with fixed capacity; temporal compression merges similar adjacent memory embeddings. Training is performed in two steps: first, the visual compressor and embedding lookup are trained on image QA datasets; second, the LLM decoder is fine-tuned on video QA data. The authors introduce InfiniBench-Vision, a filtered subset of InfiniBench, and report results on it and on short-term benchmarks (NExT-QA, MSVD-QA, MSRVTT-QA), claiming state-of-the-art long-term accuracy (49.1% vs. 44.3% for Flash-VStream) with fewer memory tokens (640 vs. 681). The central long-term claim depends on the validity of InfiniBench-Vision as a frame-only benchmark, which is not established.

Significance. The architectural idea is clear and potentially useful: conditioning the visual memory on the question is a natural way to reduce memory tokens while retaining task-relevant information. The paper gives a reproducible training recipe, detailed hyperparameters, and an ablation showing the compressor outperforms average pooling on LLaVA-Bench. The short-term results on standard benchmarks are informative. However, the headline contribution—superior long-term video understanding—is not currently supported because the evaluation rests on an unvalidated, self-constructed benchmark (InfiniBench-Vision) and a single-run comparison with no error bars. If the long-term result were established, the paper would be a solid contribution to efficient long-video LMMs; as it stands, the central claim is conditional on resolving the benchmark-validity and statistical-robustness concerns.

major comments (4)
  1. [Section 4.1, Supplementary §6] The central long-term evaluation uses InfiniBench-Vision, a dataset created by removing QA pairs whose questions contain character names detected by GPT-4o-mini. The curation process in Supplementary §6 does not verify that the remaining 599 questions are answerable from video frames alone. The manuscript's own Figure 5b shows that several surviving questions require dialogue or plot knowledge, e.g., 'What role does the connection to the previous murders in Florida play in solving the recent cases?' and 'How does the plastic surgeon's anticipated disappearance influence the urgency of the investigation?' These cannot be answered from visual frames alone. Consequently, Table 2's 49.1% accuracy and the claimed 4.8-point improvement over Flash-VStream may be an artifact of benchmark construction rather than evidence of superior frame-only long-term understanding. To support the claim, the authors need to provide human validation of visual answerability, compare performance on the original InfiniBench, and analyze how filtering changes question types and difficulty.
  2. [Section 4.2, Table 2] The long-term comparison is based on a single run on 599 QA pairs, with no error bars, confidence intervals, or significance tests. A 4.8-percentage-point gap could plausibly be within noise; a small number of GPT-judged answers could flip the ranking. The same lack of statistical grounding applies to the short-term benchmarks in Table 3. The authors should report multiple runs, bootstrapped confidence intervals, or at least a sensitivity analysis of the GPT-based evaluation.
  3. [Table 2 and Related Work §2.2] The long-term comparison includes only MovieChat, MA-LMM, and Flash-VStream. The paper cites other recent long-video LMMs (LongVA, MM-Vid, Artemis, LongVILA) in the related work but does not evaluate them. The claim of 'superiority over state-of-the-art methods' is therefore not established by the presented evidence. At minimum, the claim should be restricted to the compared methods, or additional baselines should be evaluated.
  4. [Section 3.6 vs. Section 7] The manuscript states in Section 3.6 that IQViC 'enables online QA through the parallelization of context memory construction and response generation,' but Section 7 (Limitation) admits that 'the current requirement to reprocess the entire video for each query is computationally expensive and impractical for online, interactive, long-video understanding applications.' Because the visual compressor is question-conditioned, every new question requires re-encoding and re-compressing the entire video, so the token-count advantage does not necessarily translate to lower total compute per question. The authors should quantify the total computational cost (e.g., FLOPs or latency) per query for IQViC versus baselines and reconcile the online-QA claim with the stated limitation.
minor comments (5)
  1. [Abstract and Section 4.1] The abstract refers to 'a new dataset based on InfiniBench' without noting that it is a filtered subset whose visual-answerability is not validated; please clarify this in both the abstract and the dataset description.
  2. [Section 4.2, Table 3] IQViC's NExT-QA accuracy (49.8) is slightly lower than MovieChat's (49.9); the text claims state-of-the-art performance 'according to the scores,' but the accuracy trade-off should be acknowledged explicitly.
  3. [Figure 4] The case-study question ('What message does the movie convey about the concept of fooling ourselves to get through life?') is itself arguably not answerable from visual frames alone, which weakens the demonstration; choosing an example whose frame-only answerability is clear would be more convincing.
  4. [Section 1, first paragraph] The phrase 'inferent context window' appears to be a typo; it should be 'inference context window.'
  5. [Section 4.1] The statement that 'due to licensing restrictions, InfiniBench is the only long-term video understanding benchmark dataset accessible to us' is a significant caveat that should appear in the main text rather than only in the supplementary material.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported gains are empirical results on shared benchmarks, and the only author-constructed element is a benchmark subset whose validity is separate from circularity.

full rationale

IQViC is a learned compressor trained on standard image QA datasets (COCO, GQA, TextVQA, OCR-VQA, Visual Genome) and video QA data (ActivityNet via the Video-ChatGPT-filtered-98K split), with no parameter fitted to the evaluation benchmarks. The long-term claim is measured on InfiniBench-Vision, a subset of InfiniBench that the authors filtered according to the MLVU name-based policy. This is not a circular step: the filtering is not derived from IQViC, no IQViC parameter is chosen using InfiniBench-Vision accuracies, and all compared methods are evaluated on the same resulting set. The paper contains no self-citations that bear on the architecture's validity; the compressor is explicitly credited to the external ICAE work. Equations (1)-(9) describe the forward pass and training objectives, and none of them re-introduce a target quantity as an input. The remaining concern is benchmark validity: the supplementary's own examples in Figure 5b suggest that some questions still require dialogue or prior plot knowledge, so InfiniBench-Vision may not cleanly measure frame-only understanding. That is a legitimate external-validity limitation, and the limitation section acknowledges the method ignores subtitles and audio. But a benchmark-construction concern is not circularity: the paper's result is an empirical measurement on a fixed dataset, not a derivation equivalent to its own assumptions.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim relies on hand-picked hyperparameters (C, L) and on three domain assumptions about the validity of the filtered benchmark, the adequacy of frozen pretrained components, and the transferability of two-step training. No new physical or conceptual entities are introduced.

free parameters (2)
  • Context token count C = 64
    Number of compressed tokens per frame; chosen by hand to balance accuracy and memory, ablated in Table 4 (C=1,32,64).
  • Context memory capacity L = 10
    Number of compressed frame embeddings stored in memory; chosen by hand and not ablated.
assumptions (3)
  • domain assumption The name-filtering rule in InfiniBench-Vision curation produces QA pairs answerable from video frames alone.
    Supplementary Section 6 assumes questions containing character names are unanswerable from frames and removes them; this is not validated.
  • domain assumption Pre-trained CLIP ViT-L/14 and Vicuna-1.5 provide suitable visual and language representations for compression.
    Sections 3.1 and 3.2 rely on frozen pretrained models without re-evaluating their adequacy for the compression task.
  • domain assumption Two-step training (image QA then short video QA) transfers to very-long videos.
    Section 3.5 trains only on images and short ActivityNet clips; no long-video training is performed, and end-to-end training on video was not feasible due to VRAM limits.

how reviews work

0 comments
Cite this review

Pith. "Pith review of IQViC: In-context, Question Adaptive Vision Compressor for Long-term Video Understanding LMMs." pith.science (2026). https://pith.science/paper/JGMMLS6F

@misc{pith2026241209907,
  author       = {Pith},
  title        = {Pith review of: IQViC: In-context, Question Adaptive Vision Compressor for Long-term Video Understanding LMMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JGMMLS6F}},
  note         = {Machine review of arXiv:2412.09907}
}
read the original abstract

With the increasing complexity of video data and the need for more efficient long-term temporal understanding, existing long-term video understanding methods often fail to accurately capture and analyze extended video sequences. These methods typically struggle to maintain performance over longer durations and to handle the intricate dependencies within the video content. To address these limitations, we propose a simple yet effective large multi-modal model framework for long-term video understanding that incorporates a novel visual compressor, the In-context, Question Adaptive Visual Compressor (IQViC). The key idea, inspired by humans' selective attention and in-context memory mechanisms, is to introduce a novel visual compressor and incorporate efficient memory management techniques to enhance long-term video question answering. Our framework utilizes IQViC, a transformer-based visual compressor, enabling question-conditioned in-context compression, unlike existing methods that rely on full video visual features. This selectively extracts relevant information, significantly reducing memory token requirements. Through extensive experiments on a new dataset based on InfiniBench for long-term video understanding, and standard benchmarks used for existing methods' evaluation, we demonstrate the effectiveness of our proposed IQViC framework and its superiority over state-of-the-art methods in terms of video understanding accuracy and memory efficiency.

Figures

Figures reproduced from arXiv: 2412.09907 by the authors.

Figure 1
Figure 1. Comparison of conventional and proposed methods fo [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The architecture of the proposed IQViC framework i [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Training method of the proposed IQViC framework in [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Question and answer examples from the InfiniBench- [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Examples of QA pairs from the InfiniBench and InfiniB [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Curation process for the InfiniBench dataset. [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Prompt for GPT-4o to curate the QA pairs in the Infini [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

63 extracted references · 32 canonical work pages

  1. [1]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Askell, et al. Language models are few-shot learners. In Advances in Neural Infor- mation Processing Systems , volume 33, pages 1877–1901. Curran Associates, Inc., 2020. 1

    work page 1901

  2. [2]

    GPT-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. arXiv preprint arXiv:2303.08774 , 2023

    arXiv 2023

  3. [3]

    LLaMA: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ ee Lacroix, Baptiste Rozi` ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    arXiv 2023

  4. [4]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    arXiv 2023

  5. [5]

    Lillicrap, Jean-Baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrit- twieser, et al

    Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy P . Lillicrap, Jean-Baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrit- twieser, et al. Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024. 1

    arXiv 2024

  6. [6]

    Large language models: A survey

    Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. Large language models: A survey. arXiv preprint arXiv:2402.06196, 2024. 1

    arXiv 2024

  7. [7]

    A survey of large language mod- els

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Y upeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language mod- els. arXiv preprint arXiv:2303.18223, 2023

    arXiv 2023

  8. [8]

    Large language models: a comprehensive survey of its applications, challenges, limitations, and future prospects

    Muhammad Usman Hadi, Qasem Al Tashi, Abbas Shah, Rizwan Qureshi, Amgad Muneer, Muhammad Irfan, Anas Zafar, Muhammad Bilal Shaikh, Naveed Akhtar, Jia Wu, et al. Large language models: a comprehensive survey of its applications, challenges, limitations, and future prospects. Authorea Preprints, 2024

    2024

Show all 63 references
  1. [9]

    Retrieval-augmented generation for large language models: A survey

    Y unfan Gao, Y un Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Y uxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997, 2023

    2023 arXiv

  2. [10]

    Large language models in finance: A survey

    Yinheng Li, Shaofei Wang, Han Ding, and Hang Chen. Large language models in finance: A survey. In Proceed- ings of the fourth ACM International Conference on AI in Finance, pages 374–382, 2023

    2023

  3. [11]

    ChatGPT for good? on opportunities and challenges of large language models for education

    Enkelejda Kasneci, Kathrin Seßler, Stefan K¨ uchemann , Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan G¨ unnemann, Eyke H¨ ullermeier, et al. ChatGPT for good? on opportunities and challenges of large language models for education. Learning a...

    2023

  4. [12]

    An image is worth 16x16 words: Trans- formers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesniko v, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv preprint ...

    2010 arXiv

  5. [13]

    BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In In- ternational Conference on Machine Learning, pages 19730– 19742. PMLR, 2023. 1, 2

    2023

  6. [14]

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Albert Li, Pascale Fung, and Steven C. H. Hoi. InstructBLIP: To- wards general-purpose vision-language models with instru c- tion tuning. arXiv preprint arXiv: 2305.06500 , 2023. 1, 2

    2023 arXiv

  7. [15]

    Transformers in vision: A survey

    Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. Transformers in vision: A survey. ACM Computing Surveys, 54(10s):1–41, 2022. 1

    2022

  8. [16]

    Multimodal few-shot learning with frozen language models

    Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Es- lami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems, 34:200–212, 2021. 1

    2021

  9. [17]

    Video understanding with large language models: A survey

    Y unlong Tang, Jing Bi, Siting Xu, Luchuan Song, Susan Liang, Teng Wang, Daoan Zhang, Jie An, Jingyang Lin, Rongyi Zhu, et al. Video understanding with large language models: A survey. arXiv preprint arXiv:2312.17432 , 2023. 1, 2

    2023

  10. [18]

    From seconds to hours: Reviewing multimodal large language models on comprehensive long video understanding

    Heqing Zou, Tianze Luo, Guiyang Xie, Fengmao Lv, Guangcong Wang, Juanyang Chen, Zhuochen Wang, Han- sheng Zhang, Huaijian Zhang, et al. From seconds to hours: Reviewing multimodal large language models on comprehensive long video understanding. arXiv preprint arXiv:2409.18938,...

    2024 arXiv

  11. [19]

    A survey on multimodal large language models

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. arXiv preprint arXiv:2306.13549, 2023. 1

    2023 arXiv

  12. [20]

    Vision-language models for vision tasks: A survey

    Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-language models for vision tasks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence ,

  13. [21]

    MovieChat: From dense token to sparse memory for long video understanding

    Enxin Song, Wenhao Chai, Guanhong Wang, Y ucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, et al. MovieChat: From dense token to sparse memory for long video understanding. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and P...

    2024

  14. [22]

    MA-LMM: Memory-augmented large multimodal model for long-term video understanding

    Bo He, Hengduo Li, Y oung Kyun Jang, Menglin Jia, Xue- fei Cao, Ashish Shah, Abhinav Shrivastava, and Ser-Nam Lim. MA-LMM: Memory-augmented large multimodal model for long-term video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogn...

    2024

  15. [23]

    Flash-VStream: Memory- based real-time understanding for long video streams

    Haoji Zhang, Yiqin Wang, Yansong Tang, Y ong Liu, Jiashi Feng, Jifeng Dai, and Xiaojie Jin. Flash-VStream: Memory- based real-time understanding for long video streams. arXiv preprint arXiv:2406.08085, 2024. 2, 3, 5, 6, 1 9

    2024 arXiv

  16. [24]

    Evolving conceptions of memory storage, selective attention, and their mutual constraints within t he human information-processing system

    Nelson Cowan. Evolving conceptions of memory storage, selective attention, and their mutual constraints within t he human information-processing system. Psychological bul- letin, 104(2):163, 1988. 2, 4

    1988

  17. [25]

    Gorillas in o ur midst: Sustained inattentional blindness for dynamic even ts

    Daniel J Simons and Christopher F Chabris. Gorillas in o ur midst: Sustained inattentional blindness for dynamic even ts. Perception, 28(9):1059–1074, 1999. 2, 4

    1999

  18. [26]

    A comprehensive survey on pretrained foundation mod- els: A history from BERT to ChatGPT

    Ce Zhou, Qian Li, Chen Li, Jun Y u, Yixin Liu, Guangjing Wang, Kai Zhang, Cheng Ji, Qiben Yan, Lifang He, et al. A comprehensive survey on pretrained foundation mod- els: A history from BERT to ChatGPT. arXiv preprint arXiv:2302.09419, 2023. 2

    2023 arXiv

  19. [27]

    Towards reason- ing in large language models: A survey

    Jie Huang and Kevin Chen-Chuan Chang. Towards reason- ing in large language models: A survey. arXiv preprint arXiv:2212.10403, 2022. 2

    2022 arXiv

  20. [28]

    BLIP: Bootstrapping language-image pre-training for uni- fied vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping language-image pre-training for uni- fied vision-language understanding and generation. In In- ternational Conference on Machine Learning , pages 12888– 12900. PMLR, 2022. 2

    2022

  21. [29]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Y uheng Li, and Y ong Jae Lee. Improved baselines with visual instruction tuning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024. 2, 4, 5

    2024

  22. [30]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Y ong Jae Lee. Visual instruction tuning. Advances in Neural Information Processing Systems, 36, 2024. 2, 5, 6, 7

    2024

  23. [31]

    Learn- ing transferable visual models from natural language super - vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super - vision. In International Conference on Machine Learnin...

    2021

  24. [32]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhang- hao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Y ong- hao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P . Xing. Vicuna: An open-source chatbot impressing GPT-4 with 90%* ChatGPT quality, March 2023. 2, 5

    2023

  25. [33]

    LLaV A-NeXT: Improved reasoning, OCR, and world knowledge, January 2024

    Haotian Liu, Chunyuan Li, Y uheng Li, Bo Li, Y uanhan Zhang, Sheng Shen, and Y ong Jae Lee. LLaV A-NeXT: Improved reasoning, OCR, and world knowledge, January 2024. 2

    2024

  26. [34]

    Chat-UniVi: Unified visual representation em- powers large language models with image and video un- derstanding

    Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao , and Li Y uan. Chat-UniVi: Unified visual representation em- powers large language models with image and video un- derstanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1...

    2024

  27. [35]

    VideoLLaMA 2: Advancing spatial- temporal modeling and audio understanding in video-llms

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Y ongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. VideoLLaMA 2: Advancing spatial- temporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.07476, 2024

    2024 arXiv

  28. [36]

    LLaMA-VID: An image is worth 2 tokens in large language models

    Yanwei Li, Chengyao Wang, and Jiaya Jia. LLaMA-VID: An image is worth 2 tokens in large language models. In European Conference on Computer Vision , pages 323–340. Springer, 2024. 2, 1, 3

    2024

  29. [37]

    Video-ChatGPT: Towards detailed video understanding via large vision and language models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fa- had Shahbaz Khan. Video-ChatGPT: Towards detailed video understanding via large vision and language models. In Pro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics , volume 1, pages 12585...

    2024

  30. [38]

    VideoChat: Chat-centric video understanding

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Y u Qiao. VideoChat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355, 2023

    2023 arXiv

  31. [39]

    Video-LLaMA: An instruction-tuned audio-visual language model for video u n- derstanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-LLaMA: An instruction-tuned audio-visual language model for video u n- derstanding. In Proceedings of the 2023 Conference on Em- pirical Methods in Natural Language Processing: System Demonstrations, pages 543–553, 2023. 2, 1, 3

    2023

  32. [40]

    Long context transfer from language to vision

    Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Y uanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv preprint arXiv:2406.16852, 2024. 2, 3

    2024 arXiv

  33. [41]

    MM-Vid: Advanc- ing video understanding with GPT-4V(ision)

    Kevin Lin, Faisal Ahmed, Linjie Li, Chung-Ching Lin, Ehsan Azarnasab, Zhengyuan Yang, Jianfeng Wang, Lin Liang, Zicheng Liu, Y umao Lu, et al. MM-Vid: Advanc- ing video understanding with GPT-4V(ision). arXiv preprint arXiv:2310.19773, 2023. 3

    arXiv

  34. [42]

    Artemis: Towards referential understanding in com- plex videos

    Jihao Qiu, Y uan Zhang, Xi Tang, Lingxi Xie, Tianren Ma, Pengyu Yan, David Doermann, Qixiang Ye, and Y unjie Tian. Artemis: Towards referential understanding in com- plex videos. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. 2, 3

    2024

  35. [43]

    Language modeling is compression

    Gr´ egoire Del´ etang, Anian Ruoss, Paul-Ambroise Duquenne, Elliot Catt, Tim Genewein, Christopher Mattern, Jordi Grau - Moya, Li Kevin Wenliang, Matthew Aitchison, Laurent Orseau, et al. Language modeling is compression. arXiv preprint arXiv:2309.10668, 2023. 3

    2023 arXiv

  36. [44]

    Learning to compress prompts with gist tokens

    Jesse Mu, Xiang Lisa Li, and Noah Goodman. Learning to compress prompts with gist tokens. In Thirty-seventh Con- ference on Neural Information Processing Systems , 2023. 3

    2023

  37. [45]

    Adapting language models to compress con- texts

    Alexis Chevalier, Alexander Wettig, Anirudh Ajith, an d Danqi Chen. Adapting language models to compress con- texts. arXiv preprint arXiv:2305.14788, 2023. 3

    2023 arXiv

  38. [46]

    In-context autoencoder for context compression in a large language model

    Tao Ge, Hu Jing, Lei Wang, Xun Wang, Si-Qing Chen, and Furu Wei. In-context autoencoder for context compression in a large language model. In The Twelfth International Con- ference on Learning Representations, 2024. 3, 4, 5

    2024

  39. [47]

    LoRA: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Y uanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. 5

    2021 arXiv

  40. [48]

    Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ ar, and C

    Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ ar, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. In European Conference on Computer Vision, pages 740–755. Springer, 2014. 5

    2014

  41. [49]

    GQA: a new dataset for compositional question answering over real - world images

    Drew A Hudson and Christopher D Manning. GQA: a new dataset for compositional question answering over real - world images. arXiv preprint arXiv:1902.09506, 2019. 5 10

    1902 arXiv

  42. [50]

    Towards VQA models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Y u Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards VQA models that can read. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8317–8326, 2019. 5

    2019

  43. [51]

    OCR-VQA: Visual question answer- ing by reading text in images

    Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. OCR-VQA: Visual question answer- ing by reading text in images. In 2019 International Con- ference on Document Analysis and Recognition , pages 947–

    2019

  44. [52]

    Shamma, Michael S

    Ranjay Krishna, Y uke Zhu, Oliver Groth, Justin Johnson , Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalan- tidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual Genome: Connecting language and vision using crowdsourced dense image annotations...

    2017

  45. [53]

    ActivityNet: A large-scale video benchmark for human activity understanding

    Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem , and Juan Carlos Niebles. ActivityNet: A large-scale video benchmark for human activity understanding. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 961–970, 2015. 5

    2015

  46. [54]

    Decoupled weight de- cay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight de- cay regularization. In International Conference on Learning Representations, 2019. 5

    2019

  47. [55]

    In- finiBench: A comprehensive benchmark for large multi- modal models in very long video understanding

    Kirolos Ataallah, Chenhui Gou, Eslam Abdelrahman, Khushbu Pahwa, Jian Ding, and Mohamed Elhoseiny. In- finiBench: A comprehensive benchmark for large multi- modal models in very long video understanding. arXiv preprint arXiv:2406.19875, 2024. 5, 6, 1

    2024

  48. [56]

    MLVU: A comprehensive benchmark for multi-task long video understanding

    Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Y ongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. MLVU: A comprehensive benchmark for multi-task long video understanding. arXiv preprint arXiv:2406.04264, 2024. 6, 1

    2024 arXiv

  49. [57]

    LVBench: An extreme long video understanding benchmark

    Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiao- han Zhang, Ji Qi, Xiaotao Gu, Shiyu Huang, Bin Xu, Y uxiao Dong, et al. LVBench: An extreme long video understanding benchmark. arXiv preprint arXiv:2406.08035, 2024. 6, 1

    2024 arXiv

  50. [58]

    NExT-QA: Next phase of question-answering to explaining temporal actions

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua . NExT-QA: Next phase of question-answering to explaining temporal actions. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition , pages 9777–9786, 2021. 6

    2021

  51. [59]

    Video question answer- ing via gradually refined attention over appearance and mo- tion

    Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Y ueting Zhuang. Video question answer- ing via gradually refined attention over appearance and mo- tion. In Proceedings of the 25th ACM International Confer- ence on Multimedia, pages 1645–1653, 2017. 6

    2017

  52. [60]

    Video-MME: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Y uhan Dai, Y ondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Y unhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen, Rongrong Ji, and Xing Sun. Video-MME: The first-ever compr...

    2024 arXiv

  53. [61]

    LongVILA: Scaling long-context visual language models for long videos

    Fuzhao Xue, Y ukang Chen, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Y unhao Fang, Haotian Tang, Shang Yang, Zhijian Liu, et al. LongVILA: Scaling long-context visual language models for long videos. arXiv preprint arXiv:2408.10188, 2024. 3 11 IQViC: In-context, Question Ad...

    2024 arXiv

  54. [62]

    Sheldon,

    InfiniBench-Vision To evaluate the long-term video understanding capabilitie s of the proposed method, we introduce InfiniBench-Vision, a subset of the InfiniBench [ 55]. This section describes the motivation behind creating the InfiniBench-Vision dataset , its curation process, a...

    1980

  55. [63]

    Invisible Gorilla

    Limitation This study has several limitations, which also represent ex - citing avenues for future research. First, achieving accu- rate responses to new video questions, as highlighted by the “Invisible Gorilla” experiment [ 25], necessitates dynamic memory updates tailored t...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.