Pith. sign in

REVIEW 3 major objections 6 minor 6 cited by

Streaming Video Understanding and Multi-round Interaction with Memory-enhanced Knowledge

T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a training-free hierarchical memory lets an existing video LLM carry on real-time multi-round dialogue about a video stream, reaching 64.7% accuracy on its new StreamBench benchmark and 32 frames per second processing.

desk verdict A useful streaming video QA system and a new benchmark, but the headline accuracy is undefined without threshold T and the hyperparameters appear to be tuned on the test set. read the letter →

arxiv 2501.13468 v1 pith:AP6LRYXT submitted 2025-01-23 cs.CV cs.AI

Haomiao Xiong , Zongxin Yang , Jiazuo Yu , Yunzhi Zhuge , Lu Zhang , Jiawen Zhu , Huchuan Lu This is my paper · ORCID
classification cs.CVcs.AI
keywords streamingvideounderstandinghierarchicalmemorymulti-rounddialoguetraining-freeLLMquestionansweringStreamBenchlong
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to establish that a video-language model can carry on real-time, multi-round conversation about an ongoing video stream without any training, by managing what it has seen in a hierarchical memory. It introduces StreamChat, built on the LongVA video-language model, together with a new benchmark, StreamBench, of 306 long videos with six question types arranged as multi-turn dialogues. On StreamBench, StreamChat's slow setting reaches 64.7% accuracy online, 8.3 points above the previous best streaming method, while its fast setting processes 32 frames per second and all settings answer within 0.9 seconds. The claim matters because real-time video agents, such as robots or wearable assistants, need to answer fresh questions about long past footage without blowing up memory or waiting for the whole video.

What carries the argument

The load-bearing mechanism is the hierarchical memory: a short-term memory $M_s$ built by Ebbinghaus-weighted random sampling of recent vision embeddings, a long-term memory $M_l$ organized as a tree whose nodes are $k$-means clusters of chunked vision features paired with captions, and a dialogue memory $M_d$ of encoded question-answer pairs. A selective frame stacking thread using Lucas-Kanade optical flow drops redundant frames before encoding, and three parallel threads (frame stacking, memory formation, contextual summarization) decouple feature extraction from response generation, which is what keeps latency under 0.9 seconds.

What would settle it

Split StreamBench by video into tuning and held-out halves, tune the four memory parameters only on the tuning half, then measure StreamChat's held-out accuracy relative to Video-online; if the 8.3-point advantage does not survive the held-out comparison, the reported edge was an artifact of in-sample parameter selection.

Watch

Extended reading notes

Core claim

The central discovery is that a training-free memory system can give an existing video LLM streaming and multi-round abilities. StreamChat stores recent video features in a short-term memory, compresses older footage into a tree of clustered visual features with text captions as a long-term memory, and keeps dialogue history in a separate encoded memory; at query time it retrieves the most similar nodes and captions and feeds them to the LLM. The paper reports that this raises StreamBench online accuracy from the previous best streaming method's 56.4% to 64.7% for the Slow setting and 63.8% for the Base setting, while cutting request processing delay to about 0.9 seconds and reaching 32 FPS.

Load-bearing premise

The headline numbers assume that the memory settings (frame threshold, chunk length, group size, clustering goal) chosen after inspecting StreamBench accuracy will perform as well on videos never used to pick those settings, yet the paper does not describe a validation/test split of StreamBench.

Editorial extensions

If this is right

  • StreamChat can be applied to any existing video LLM without training, so streaming capability becomes a memory-and-retrieval problem rather than a finetuning problem.
  • The six task types in StreamBench let different memory components be credited separately: long-term memory improves long-term recall, short-term memory improves recent events, and dialogue memory improves conversational interaction.
  • At 32 FPS with sub-0.9-second request processing delay, the system is fast enough for real-time robotics or assistant interaction, not just offline benchmark evaluation.
  • Ablations show the memory components are complementary: combining long- and short-term memory adds 0.9% average accuracy over either one alone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if the memory parameters were fixed without peeking at StreamBench, the approach would likely transfer to other long-video streams, but the paper does not report such a held-out split, so the 8.3-point margin should be read as in-sample.
  • Editorial inference: the retrieval-by-caption design means accuracy is capped by caption quality and by the granularity of the $k$-means clusters; small and briefly appearing objects are already listed among failure cases.
  • Editorial inference: the method suggests a general recipe: external hierarchical memory plus retrieval can substitute for training in other streaming modalities, such as audio or sensor streams, whenever a capable pretrained model exists.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes StreamChat, a training-free framework built on the LongVA video-LLM, and StreamBench, a new benchmark for streaming video understanding with multi-round interaction. StreamChat uses a hierarchical memory system (short-term, long-term tree-structured, and dialogue memory) and a parallel three-thread scheduling strategy to support real-time online QA. The authors report 64.7% accuracy on StreamBench for the Slow variant, an 8.3% improvement over the previous best streaming method Video-online, 32 FPS processing for the Fast variant, and sub-0.9s response delay. They also report offline accuracy gains on MSVD, MSRVTT, ActivityNet, and NExT-QA, and provide ablations of the memory components and of the memory hyperparameters.

Significance. If the reported numbers are reliable, the paper makes a valuable contribution: it demonstrates that a frozen video-LLM augmented with a carefully designed memory and scheduling system can support genuine multi-turn streaming dialogue at interactive frame rates, and it introduces a benchmark that explicitly measures latency alongside accuracy, which the community lacks. The public availability of code and the inclusion of ablation studies over memory components and parameters are strengths. However, the central quantitative claims currently rest on two under-specified components: the accuracy threshold T in the evaluation metric and the in-sample selection of memory hyperparameters. Until those are addressed, the headline 8.3% margin and the 64.7% accuracy figure should not be taken at face value.

major comments (3)
  1. [Appendix D, Eq. (6)] The accuracy metric Acc = (1/N) * sum(I(S_i >= T)) is used for every accuracy value in Tables 4 and 5, but the threshold T is never specified anywhere in the paper or appendix. Since the LLaMA-3 judge returns integer scores S_i in [0,5], different choices of T can substantially rescale all accuracy numbers, so the reported 64.7% for StreamChat-Slow and the 8.3% gap over Video-online (56.4%) are not well-defined. Please state the exact value of T used and report the sensitivity of Table 4 to T in {2,3,4}.
  2. [§4.1, Table 3 and §4.4, Figure 7] The memory hyperparameters (t, L, g, C) for the three StreamChat variants appear to be chosen after inspecting accuracy on StreamBench itself: Figure 7 shows accuracy varying from 59.4% to 64.0% as L, g, and C change, and Table 3 fixes the final values without any described held-out validation split. As a result, the reported 64.7% is an in-sample estimate, and the performance advantage over Video-online may not generalize to a separate test split. Please either fix a default configuration before evaluation, use a dedicated validation split of StreamBench, or report cross-validated results.
  3. [§3.1.1, Eq. (3)] The long-term memory construction depends on a captioning model p_theta that generates text clues for each visual chunk, but this model is never identified or described. Because retrieval in the long-term memory tree operates on these text clues, the method is not fully reproducible without specifying p_theta, whether it is frozen, and how it is integrated with the LongVA base model. Please provide these details or a reference to the exact captioning component used.
minor comments (6)
  1. [§3.2] There is a typo in the paragraph describing system scheduling: 'slective frame stacking' should be 'selective frame stacking'.
  2. [Appendix F] The deployment section contains the typo 'GUP1' and 'GUP2'; these should be 'GPU1' and 'GPU2'.
  3. [Figure 7] The rendered figure panels appear duplicated in the manuscript, and the axes in panel (d) are not fully labeled. Please provide a clean figure with distinct panels for speed-threshold, chunk length, group size, and clustering goal.
  4. [Table 7] The sixth task is called 'Simple Factual (SF)' in §2.2, but Table 7 uses the column header 'SS' and the text refers to 'SS' in one place; this should be made consistent.
  5. [Appendix G] The expansion plan states that the current benchmark 'only has a single task type', which contradicts the six distinct question types described in §2.2. Please rephrase to clarify what is meant.
  6. [Figure 1] The figure contains a misspelling of 'Knowledge' as 'Konwledge' in the caption/legend; please correct it.

Circularity Check

1 steps flagged · score 6.0 of 10

The headline StreamBench accuracy is partly a fitted input: the memory hyperparameters (t, L, g, C) were selected using StreamBench accuracy itself, with no held-out split described, so the reported 64.7% and 8.3% margin are in-sample tuning artifacts rather than independent predictions.

  1. fitted input called prediction [Section 4.1 (Memory Configurations, Table 3) and Section 4.4 (Figure 7); headline claim in Section 1]
    "Memory Configurations. To adapt the model to various application scenarios, we configure three versions with different memory settings: Base, Fast, and Slow. These variants adjust key memory parameters, including threshold (t), chunk length (L), group size (g), and clustering goals (C), as summarized in Tab. 3. ... In Fig. 7 (b-d), we evaluate how these factors influence online video understanding tasks, using the Base model with t=0.35. ... higher processing speeds are discouraged due to their detrimental impact on model performance (64.0%→60.7%)."

    The central quantitative claim is 'delivering a 64.7% accuracy on STREAMBENCH for online settings, which is an 8.3% improvement over the previous best' (Section 1). That 64.7% is produced by the Slow configuration, whose four memory parameters (t=0.13, L=35, g=15, C=5) were chosen after inspecting StreamBench accuracy in Table 3, Figure 7, and the surrounding ablations. The paper never describes a validation or test split of StreamBench, so the reported accuracy is measured on the same benchmark that was used to select the configuration. The headline margin is therefore partly a consequence of fitting model configuration to the evaluation set, not an independent prediction of generalization.

full rationale

The StreamChat framework itself is not circular: its forward pipeline is described by concrete mechanisms (optical-flow thresholding, k-means clustering, captioning, tree-structured retrieval, dialogue memory encoding), and the offline evaluations on MSVD, MSRVTT, ActivityNet, and NExT-QA use external benchmarks that are independent of the method's design. The one significant circularity is the selection of the memory hyperparameters on the same StreamBench set that is used for the headline 64.7% accuracy and the 8.3% improvement. No held-out split or pre-registered configuration is described, so the top-line number is partly a fitted input. The missing threshold T in Eq. (6) is a serious reproducibility and correctness flaw, but it is not a circular-derivation issue and is not counted in the score. Self-citations are not load-bearing here: LongVA is a third-party foundation model, and no uniqueness theorem or prior-work claim is used to forbid alternatives. Score 6 reflects one central claim that is partially forced by in-sample tuning, while the method's other benchmark results remain externally grounded.

Assumptions & free parameters 7 free parameters · 6 assumptions · 0 invented entities

The system introduces no new physical entities. Its load-bearing elements are tuned parameters (optical flow threshold, memory chunk sizes, clustering goals, short-term memory sizes) and untested modeling assumptions about which frames, clusters, captions, and judge outputs preserve the information needed for correct answers. The unstated accuracy threshold T is a free parameter in the metric itself.

free parameters (7)
  • optical flow threshold t = 0.13 (Slow), 0.35 (Base), 0.58 (Fast)
    Controls frame filtering; selected via accuracy/speed tradeoff on StreamBench (Table 3, Fig. 7a).
  • chunk length L = 35 (Slow), 25 (Base), 30 (Fast)
    Long-memory chunk size; chosen by accuracy/latency on StreamBench (Fig. 7b).
  • group size g = 15 (Slow), 10 (Base), 15 (Fast)
    Number of child nodes grouped into parent in memory tree; tuned on StreamBench (Fig. 7c).
  • clustering goal C = 5 for all versions
    Number of k-means clusters per chunk; tuned on StreamBench (Fig. 7d).
  • short-term memory size S = 5
    Number of vision embeddings randomly kept in short-term memory; set in Implementation Details (Section 4.1).
  • candidate length C (short-term) = 20
    Number of recent vision embeddings considered for short-term memory; set in Implementation Details (Section 4.1).
  • accuracy threshold T in Acc metric = not reported
    Accuracy is defined as I(S_i >= T) in Eq. (6) but T never specified, so reported Acc values are underdetermined.
assumptions (6)
  • domain assumption LongVA (used as frozen foundation) provides sufficiently capable video-language reasoning.
    The whole system inherits LongVA's visual understanding; if LongVA fails on a task, StreamChat inherits the failure. Section 3, Appendix F.
  • domain assumption Lucas-Kanade optical flow magnitude reliably identifies frames worth storing.
    Used in selective frame stacking (Eq. 1); no validation that motion magnitude correlates with question-relevant information.
  • domain assumption K-means clustering and MLLM captioning of video chunks preserves information needed to answer later questions.
    Long-term memory construction (Eq. 3-5) compresses video features; the paper does not measure how much question-answerable information survives compression.
  • ad hoc to paper The Ebbinghaus forgetting curve is an appropriate basis for random selection of short-term memory units.
    Borrowed from cognitive psychology without evidence that normalized forgetting probabilities improve video QA; the sigma_i distribution is not specified.
  • domain assumption LLaMA-3-8B judge provides correct semantic scores for open-ended video QA.
    All accuracy and coherence numbers depend on this judge (Appendix D); no human agreement or calibration is reported.
  • ad hoc to paper The unspecified captioning model p_theta can generate reliable text clues for chunks.
    Eq. (3) calls p_theta(x_i|K_i) but does not identify the MLLM or prompt; retrieval quality rests on these captions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Streaming Video Understanding and Multi-round Interaction with Memory-enhanced Knowledge." pith.science (2026). https://pith.science/paper/AP6LRYXT

@misc{pith2026250113468,
  author       = {Pith},
  title        = {Pith review of: Streaming Video Understanding and Multi-round Interaction with Memory-enhanced Knowledge},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AP6LRYXT}},
  note         = {Machine review of arXiv:2501.13468}
}
abstract

Recent advances in Large Language Models (LLMs) have enabled the development of Video-LLMs, advancing multimodal learning by bridging video data with language tasks. However, current video understanding models struggle with processing long video sequences, supporting multi-turn dialogues, and adapting to real-world dynamic scenarios. To address these issues, we propose StreamChat, a training-free framework for streaming video reasoning and conversational interaction. $\StreamChat$ leverages a novel hierarchical memory system to efficiently process and compress video features over extended sequences, enabling real-time, multi-turn dialogue. Our framework incorporates a parallel system scheduling strategy that enhances processing speed and reduces latency, ensuring robust performance in real-world applications. Furthermore, we introduce StreamBench, a versatile benchmark that evaluates streaming video understanding across diverse media types and interactive scenarios, including multi-turn interactions and complex reasoning tasks. Extensive evaluations on StreamBench and other public benchmarks demonstrate that StreamChat significantly outperforms existing state-of-the-art models in terms of accuracy and response times, confirming its effectiveness for streaming video understanding. Code is available at StreamChat: https://github.com/hmxiong/StreamChat.

Figures

Figures reproduced from arXiv: 2501.13468 by the authors.

Figure 1
Figure 1. Performance comparison between STREAMCHAT and previous Video-LLMs. Recent advancements in Large Language Models (LLMs) [1–3] have led to the development of Video￾LLMs [4–9], which aim to interpret visual scenes, ac￾tions, and narratives. These models represent significant progress in multimodal learning by bridging video data and language-based tasks, with applications spanning from content analysis to human-robot i… view at source ↗
Figure 2
Figure 2. The comparisons between StreamChat and other methods (§1). Offline methods process entire videos, leading to information loss and limited to a single interaction. Previous online methods [10, 11] enable multi-round interactions but still suffer from slow processing and answer correctly. The proposed method achieves real-time video processing, improving the efficiency and accuracy with memory support. (cf [PITH_FULL… view at source ↗
Figure 3
Figure 3. Benchmark overview (§2). Our benchmark covers 4 key domains and 16 sub-class video types. These videos exhibit a broader distribution of length, with 6 different types that are evenly distributed. • In terms of efficiency, STREAMCHAT achieves a processing speed of 32 FPS, marking a sixfold increase over existing methods. Additionally, it maintains text generation latency under 0.9 seconds, showcasing significant adv… view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Overview of StreamChat (§3), which comprises three main components: (i) Selective frame stacking, which prepares vision features for processing, including encoding frames and filling the vision buffer; (ii) Memory formation, where vision features are organized into str…
Figure 5
Figure 5. Figure 5: The hierarchical memory storage (§3.1). (a) Long-short term memory, where the long memory tree Ml and short-term memory Ms are constructed along the video time line. (b) The dialogue memory Md is updated after each inference conversation for managing the dialogue histo…
Figure 6
Figure 6. Figure 6: An inference example of StreamChat (§4.3). Given a question, our system retrieves the most related information in a long memory tree and dialogue histories based on the highest cosine similarity [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Analysis of memory parameters. (a) The influence between speed and threshold; Impact of (b) chunk length and (c) group size on performance and latency; (d) Effect of clustering goal on performance and VRAM. 5 RELATED WORK Multi-modal Language Models (MLMs). Recent deve…
Figure 8
Figure 8. Figure 8: The date preparation pipeline utilized in [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Visualization of egocentric video analysis. [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: Visualization of web video analysis. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 11
Figure 11. Figure 11: Visualization of working video analysis. [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]
Figure 12
Figure 12. Figure 12: Visualization of movie video analysis. C RETRIEVAL ALGORITHM Inspired by the retrieval argumentation system [43], our approach enhances the model’s capability to address complex queries by retrieving the most relevant information from long-term memory for contextual s…
Figure 13
Figure 13. Figure 13: Visualization of failure cases. or hallucinations in the model’s output when prompts are adjusted to incorporate information from the memory mechanism. By integrating LongVA with our proposed system, we successfully extend its capabilities to en￾compass streaming vide…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. FOLIO: Focused Semantic Memory for Streaming Video Understanding

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Entity-centered focus-guided streaming memory lifts Qwen3-VL-8B to 82.0/69.1 Perception/Backward on OVO-Bench and 74.5 on StreamingBench while cutting writer tokens by ~32%.

  2. Q-GeoMem: Question-Guided Geometric Memory for Video Spatial Reasoning

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    Question-guided dual geometric memories with relevance-novelty utility reportedly reach state-of-the-art video spatial reasoning on two in-domain and five out-of-distribution benchmarks.

  3. Position: Modular Memory is the Key to Continual Learning Agents

    cs.LG 2026-03 conditional novelty 6.0 of 10

    A modular memory combining in-context learning and in-weight learning is proposed as the key to continual learning agents.

  4. AdsQA: Towards Advertisement Video Understanding

    cs.CV 2025-09 conditional novelty 6.0 of 10

    AdsQA adds an ad-video question-answering benchmark and ReAd-R, a GRPO-trained model that beats 7B baselines but not larger closed models.

  5. Vad-R1: Towards Video Anomaly Reasoning via Perception-to-Cognition Chain-of-Thought

    cs.CV 2025-05 conditional novelty 6.0 of 10

    A two-stage trained MLLM with a perception-to-cognition chain-of-thought and a self-verification RL reward outperforms prior models on video anomaly detection and reasoning.

  6. Diffractive electroproduction of light vector particles: leading Fock-state contribution in the presence of significant higher Fock-state effects

    hep-ph 2025-08 unverdicted novelty 5.0 of 10

    The paper claims the leading quark-antiquark approximation in the color dipole model only matches HERA data for rho/gamma above Q^2 of 20 GeV^2 and for phi above Q^2 of 10 GeV^2, unlike J/psi.

Reference graph

Works this paper leans on

54 extracted references · 4 canonical work pages · cited by 6 Pith papers

  1. [1]

    Chatgpt: Optimizing language models for dialogue

    OpenAI. Chatgpt: Optimizing language models for dialogue. https://openai.com/ blog/chatgpt, 2022. Accessed on: November 30, 2022

    work page 2022

  2. [2]

    Llama-vid: An image is worth 2 tokens in large language models

    Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. arXiv preprint arXiv:2311.17043, 2023

    arXiv 2023

  3. [3]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    arXiv 2024

  4. [4]

    Video-llava: Learning united visual representation by alignment before projection

    Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122, 2023

    arXiv 2023

  5. [5]

    Chat-univi: Unified visual representation empowers large language models with image and video understanding

    Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation empowers large language models with image and video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13700–13710, 2024

    2024

  6. [6]

    Video-chatgpt: Towards detailed video understanding via large vision and language models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424, 2023

    arXiv 2023

  7. [7]

    Moviechat: From dense token to sparse memory for long video understanding

    Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, et al. Moviechat: From dense token to sparse memory for long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18221–18232, 2024

    2024

  8. [8]

    Freeva: Offline mllm as training-free video assistant

    Wenhao Wu. Freeva: Offline mllm as training-free video assistant. arXiv preprint arXiv:2405.07798, 2024

    arXiv 2024

Show all 54 references
  1. [9]

    Pllava: Parameter-free llava extension from images to videos for video dense captioning

    Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. Pllava: Parameter-free llava extension from images to videos for video dense captioning. arXiv preprint arXiv:2404.16994, 2024

    2024 arXiv

  2. [10]

    Flash-vstream: Memory-based real-time understanding for long video streams

    Haoji Zhang, Yiqin Wang, Yansong Tang, Yong Liu, Jiashi Feng, Jifeng Dai, and Xiaojie Jin. Flash-vstream: Memory-based real-time understanding for long video streams. arXiv preprint arXiv:2406.08085, 2024

    2024 arXiv

  3. [11]

    Videollm-online: Online video large language model for streaming video

    Joya Chen, Zhaoyang Lv, Shiwei Wu, Kevin Qinghong Lin, Chenan Song, Difei Gao, Jia-Wei Liu, Ziteng Gao, Dongxing Mao, and Mike Zheng Shou. Videollm-online: Online video large language model for streaming video. In Proceedings of the IEEE/CVF Conference on Computer Vision and P...

    2024

  4. [12]

    Msr-vtt: A large video description dataset for bridging video and language

    Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5288–5296, 2016

    2016

  5. [13]

    Egoschema: A diagnostic benchmark for very long-form video language understanding

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding. Advances in Neural Information Processing Systems, 36, 2024

    2024

  6. [14]

    Activitynet-qa: A dataset for understanding complex web videos via question answering

    Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 9127–9134, 2019

    2019

  7. [15]

    Next-qa: Next phase of question- answering to explaining temporal actions

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question- answering to explaining temporal actions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9777–9786, 2021. 11 Published as a conference paper a...

    2021

  8. [16]

    Video question answering via gradually refined attention over appearance and motion

    Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In Proceedings of the 25th ACM international conference on Multimedia, pages 1645–1653, 2017

    2017

  9. [17]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075, 2024

    2024 arXiv

  10. [18]

    Mlvu: A comprehensive benchmark for multi-task long video understanding

    Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding. arXiv preprint arXiv:2406.04264, 2024

    2024 arXiv

  11. [19]

    Youtube-8m: A large-scale video classification benchmark

    Sami Abu-El-Haija, Nisarg Kothari, Joonseok Lee, Paul Natsev, George Toderici, Balakrishnan Varadarajan, and Sudheendra Vijayanarasimhan. Youtube-8m: A large-scale video classification benchmark. arXiv preprint arXiv:1609.08675, 2016

    2016 arXiv

  12. [20]

    Long context transfer from language to vision

    Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv preprint arXiv:2406.16852, 2024

    2024 arXiv

  13. [21]

    An iterative image registration technique with an application to stereo vision

    Bruce D Lucas and Takeo Kanade. An iterative image registration technique with an application to stereo vision. In IJCAI’81: 7th international joint conference on Artificial intelligence , volume 2, pages 674–679, 1981

    1981

  14. [22]

    A proposed system and its control processes

    Richard C Atkinson. A proposed system and its control processes. The Psychology of Learning and Motivation, 2, 1968

    1968

  15. [23]

    Memory: A contribution to experimental psychology

    Hermann Ebbinghaus. Memory: A contribution to experimental psychology. Annals of neurosciences, 20(4):155, 2013

    2013

  16. [24]

    Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers

    Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. Advances in Neural Information Processing Systems, 33:5776–5788, 2020

    2020

  17. [25]

    Billion-scale similarity search with gpus

    Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. IEEE Transactions on Big Data, 7(3):535–547, 2019

    2019

  18. [26]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pa...

    2021

  19. [27]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  20. [28]

    Llava-next: Improved reasoning, ocr, and world knowledge, January 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024

    2024

  21. [29]

    Direct preference optimization of video large multimodal models from language model reward

    Ruohong Zhang, Liangke Gui, Zhiqing Sun, Yihao Feng, Keyang Xu, Yuanhan Zhang, Di Fu, Chunyuan Li, Alexander Hauptmann, Yonatan Bisk, et al. Direct preference optimization of video large multimodal models from language model reward. arXiv preprint arXiv:2404.01258, 2024

    2024 arXiv

  22. [30]

    Minicpm-v: A gpt-4v level mllm on your phone

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800, 2024

    2024 arXiv

  23. [31]

    Vila: On pre-training for visual language models

    Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26689–26699, 2024

    2024

  24. [32]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vis...

    2024

  25. [33]

    Internlm-xcomposer-2.5: A versatile 12 Published as a conference paper at ICLR 2025 large vision language model supporting long-contextual input and output

    Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang, Linke Ouyang, et al. Internlm-xcomposer-2.5: A versatile 12 Published as a conference paper at ICLR 2025 large vision language model supporting long-contextual input and ou...

    2025 arXiv

  26. [34]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  27. [35]

    Opt: Open pre-trained transformer language models

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022

    2022 arXiv

  28. [36]

    Scaling instruction-finetuned language models

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25(70):1–53, 2024

    2024

  29. [37]

    Palm 2 technical report

    Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023

    2023 arXiv

  30. [38]

    Gpt-4 technical report

    R OpenAI. Gpt-4 technical report. arxiv 2303.08774. View in Article, 2(5), 2023

    2023 arXiv

  31. [39]

    Visual instruction tuning.Advances in neural information processing systems, 36, 2024

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36, 2024

    2024

  32. [40]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024

    2024

  33. [41]

    Quo vadis, action recognition? a new model and the kinetics dataset

    Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299–6308, 2017

    2017

  34. [42]

    Audio visual scene-aware dialog

    Huda Alamri, Vincent Cartillier, Abhishek Das, Jue Wang, Anoop Cherian, Irfan Essa, Dhruv Batra, Tim K Marks, Chiori Hori, Peter Anderson, et al. Audio visual scene-aware dialog. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7558–7...

    2019

  35. [43]

    Retrieval-augmented generation for ai-generated content: A survey

    Penghao Zhao, Hailin Zhang, Qinhan Yu, Zhengren Wang, Yunteng Geng, Fangcheng Fu, Ling Yang, Wentao Zhang, and Bin Cui. Retrieval-augmented generation for ai-generated content: A survey. arXiv preprint arXiv:2402.19473, 2024

    2024 arXiv

  36. [44]

    Query rewriting for retrieval-augmented large language models

    Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, and Nan Duan. Query rewriting for retrieval-augmented large language models. arXiv preprint arXiv:2305.14283, 2023

    2023 arXiv

  37. [45]

    Generate rather than retrieve: Large language models are strong context generators

    Wenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. Generate rather than retrieve: Large language models are strong context generators. arXiv preprint arXiv:2209.10063, 2022

    2022 arXiv

  38. [46]

    Enhancing retrieval-augmented large language models with iterative retrieval-generation synergy

    Zhihong Shao, Yeyun Gong, Yelong Shen, Minlie Huang, Nan Duan, and Weizhu Chen. Enhancing retrieval-augmented large language models with iterative retrieval-generation synergy. arXiv preprint arXiv:2305.15294, 2023

    2023 arXiv

  39. [47]

    Knowledgpt: Enhancing large language models with retrieval and storage access on knowledge bases

    Xintao Wang, Qianwen Yang, Yongting Qiu, Jiaqing Liang, Qianyu He, Zhouhong Gu, Yanghua Xiao, and Wei Wang. Knowledgpt: Enhancing large language models with retrieval and storage access on knowledge bases. arXiv preprint arXiv:2308.11761, 2023

    2023 arXiv

  40. [48]

    Promptagator: Few-shot dense retrieval from 8 examples

    Zhuyun Dai, Vincent Y Zhao, Ji Ma, Yi Luan, Jianmo Ni, Jing Lu, Anton Bakalov, Kelvin Guu, Keith B Hall, and Ming-Wei Chang. Promptagator: Few-shot dense retrieval from 8 examples. arXiv preprint arXiv:2209.11755, 2022

    2022 arXiv

  41. [49]

    Recitation-augmented language models

    Zhiqing Sun, Xuezhi Wang, Yi Tay, Yiming Yang, and Denny Zhou. Recitation-augmented language models. arXiv preprint arXiv:2210.01296, 2022

    2022 arXiv

  42. [50]

    Self-rag: Learning to retrieve, generate, and critique through self-reflection

    Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflection. arXiv preprint arXiv:2310.11511, 2023

    2023 arXiv

  43. [51]

    Ra-dit: Retrieval-augmented dual instruction tuning

    Xi Victoria Lin, Xilun Chen, Mingda Chen, Weijia Shi, Maria Lomeli, Rich James, Pedro Rodriguez, Jacob Kahn, Gergely Szilvasy, Mike Lewis, et al. Ra-dit: Retrieval-augmented dual instruction tuning. arXiv preprint arXiv:2310.01352, 2023. 13 Published as a conference paper at ICLR 2025

    arXiv

  44. [52]

    Fine-tuning or retrieval? comparing knowledge injection in llms

    Oded Ovadia, Menachem Brief, Moshik Mishaeli, and Oren Elisha. Fine-tuning or retrieval? comparing knowledge injection in llms. arXiv preprint arXiv:2312.05934, 2023

    2023 arXiv

  45. [53]

    Dense x retrieval: What retrieval granularity should we use? arXiv preprint arXiv:2312.06648, 2023

    Tong Chen, Hongwei Wang, Sihao Chen, Wenhao Yu, Kaixin Ma, Xinran Zhao, Dong Yu, and Hongming Zhang. Dense x retrieval: What retrieval granularity should we use? arXiv preprint arXiv:2312.06648, 2023

    2023 arXiv

  46. [54]

    role": "system

    Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, and Yanlin Wang. Memorybank: Enhancing large language models with long-term memory. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19724–19731, 2024. 14 Published as a conference paper at ICLR 20...

    2024

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.