Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

AuroraLong: Bringing RNNs Back to Efficient Open-Ended Video Understanding

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Replacing the transformer with a linear RNN cuts long-video memory ~34x without losing accuracy.

desk verdict A useful efficiency-first architecture for long-video QA, but the accuracy and efficiency claims live in disjoint regimes and the margins are partly tuned on test benchmarks. read the letter →

arxiv 2507.02591 v3 pith:EORAJE6J submitted 2025-07-03 cs.CV

Weili Xu , Enxin Song , Wenhao Chai , Xuexiang Wen , Tian Ye , Gaoang Wang This is my paper
classification cs.CV
keywords longvideounderstandinglinearRNNRWKVtokenmergingmultimodalLLMefficientinferencequestionanswering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the transformer language model inside a video understanding system can be replaced by a linear recurrent network without sacrificing accuracy, while sharply cutting memory and compute. Its AuroraLong model, with only 2B parameters and trained entirely on public data, matches or beats transformer-based video models of similar or larger size on long-video benchmarks such as MLVU, MovieChat-1K, and MVBench. The key step is compressing each frame's visual tokens to about ten percent and feeding them to the recurrent network in ascending order of merge size, so the compact token stream preserves the information needed to answer questions. If this holds, it lowers the hardware barrier for long-video question answering and suggests recurrent language models can serve as practical backbones for multimodal systems.

What carries the argument

The load-bearing object is the Sorted Visual Token Merge (S-ToMe): in each ViT layer, patch tokens are split into two sets, the most similar cross-set pairs are merged according to a retention threshold of 0.1, and the surviving tokens are sorted in ascending order of the number of original patches they represent before being projected into the RWKV LLM. This simultaneously cuts the token count so long videos fit within RWKV's 4k pretrained context and imposes an order the recurrent network can exploit, since larger merged patches carry the frame's key content and the SigLIP encoder's positional embeddings preserve spatial layout.

What would settle it

Run AuroraLong on a long-video QA benchmark where the correct answer requires reading a small, single-frame detail such as a sign, number, or tiny object, and compare against the same model given 100% token retention; if accuracy collapses at the 0.1 retention ratio, the merge discards critical spatial information. Alternatively, plot the RWKV backbone's perplexity on text sequences of 8k to 16k tokens and show a sharp degradation that correlates with a drop on long video questions, which would contradict the paper's claim that generalizability aligns with the loss curve.

Watch

Extended reading notes

Core claim

AuroraLong is a LLaVA-style model in which the transformer LLM is replaced by RWKV-v6-Finch, a linear RNN whose hidden state has constant size, so memory use no longer scales quadratically with the number of video frames. To fit a long video into the RNN's 4k pretrained context, the paper merges similar visual tokens within each frame down to a retention ratio of about 0.1 (roughly 60 tokens per frame) and then sorts the merged tokens in ascending order of how many original patches they represent. The paper reports that this 2B model, trained only on public data, matches or outperforms transformer-based video models up to 20 times larger on MLVU, MovieChat-1K, MVBench, and dense captioning, while using about 34x less GPU memory at 1024 frames and running about 8x faster than a comparable-size transformer model. It also claims to be the first to use a fully recurrent LLM backbone in a LLaVA-like architecture for open-ended video question answering.

Load-bearing premise

The central claim depends on the assumption that compressing each frame to about ten percent of its visual tokens and ordering those tokens by size still leaves the information needed to answer video questions, and that the recurrent model's known weakness on sequences longer than its 4k training context does not degrade performance on ten-minute videos.

Editorial extensions

If this is right

  • A 2B video model trained only on public data can come out ahead of much larger transformer models on several open-ended video benchmarks, including 26B InternVL-1.5 on MVBench and Gemini-1.5-Pro on VDC.
  • GPU memory for video understanding stops growing quadratically: AuroraLong holds roughly constant memory as frames increase, reaching 16k frames on a 24GB GPU instead of the roughly 100-frame ceiling of transformer baselines.
  • Token merging at a 10% retention ratio not only preserves accuracy but improves some captioning tasks, indicating that spatial redundancy in video frames is exploitable by recurrent models.
  • Training on merged tokens at ratio 0.1 in both single-image and video stages is cheaper and performs better than selectively merging only multi-image samples.
  • The sandwich prompting strategy, which places image tokens between instruction tokens, transfers from image-based VisualRWKV to video question answering.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the result holds, the practical limit for 'long video' input becomes set by the vision encoder's linear cost and the RNN's state capacity rather than by KV-cache memory; one testable extension is pushing the recipe to hour-long videos and checking whether the recurrent state saturates.
  • The ascending-size ordering suggests a general design principle for feeding spatially structured inputs into recurrent language models: order tokens by information density rather than by scan order, a hypothesis that could be tested on other linear RNN backbones such as Mamba.
  • The paper's own limitation section concedes weak multiple-choice reasoning, so a natural extension is scaling RWKV or using a hybrid that keeps local attention to see whether the efficiency gain survives with stronger reasoning.
  • Because the paper attributes merge-robustness to overfitting of the recurrent state on short visual contexts, one could test whether fine-tuning on longer contexts reduces the benefit of aggressive merging; the paper does not run that experiment.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes AuroraLong, a video large multimodal model that replaces the transformer-based LLM in a LLaVA-style architecture with the RWKV-v6 linear RNN backbone. The vision side uses SigLIP with a two-layer MLP connector; the main novelty is a training-free sorted visual token merge (S-ToMe) that reduces tokens per frame to about 60 (retention ratio 0.1) and reorders merged tokens by ascending patch count before feeding them to the recurrent LLM, plus sandwich prompting and a three-stage public-data training recipe. The paper reports competitive or better results than several larger transformer video LMMs on short and long video QA/captioning benchmarks, claims a 34x GPU-memory advantage at 1,024 frames and an 8x speed-up at 1 fps, and states it is the first to use a fully recurrent LLM backbone in an open-ended video LMM.

Significance. If the central claim holds, AuroraLong would be an important demonstration that linear RNN language models can replace transformers in video multimodal systems without a major accuracy penalty, potentially lowering the compute and memory barrier for long-video understanding. The paper has real strengths: a simple and well-motivated token-merging and reordering mechanism, ablations of the retention ratio and token order, training restricted to public data, and direct comparisons across multiple short- and long-video benchmarks. However, the accuracy and efficiency claims are validated in disjoint input-length regimes, and key hyperparameters are selected on the same benchmark sets used to report the final numbers. These gaps currently weaken the headline claim that the model is both accurate and dramatically more efficient for long videos. With the missing long-context accuracy evidence and an evaluation-protocol fix, the contribution would be significant for efficient video understanding.

major comments (3)
  1. [Section 4.1.3, Table 2, Figure 1]
  2. [Section 4.2.1, Section 4.2.2, Figure 5, Table 4]
  3. [Tables 1, 2, and 3]
minor comments (5)
  1. [Figure 1 caption]
  2. [Table 1]
  3. [Section 4.1.3]
  4. [Algorithm 1]
  5. [Figure 6]

Circularity Check

1 steps flagged · score 4.0 of 10

The retention ratio and token-order choices are selected on the very benchmarks later reported as final accuracy, so part of the accuracy claim is a fit to the test sets; efficiency and MVBench/MLVU results remain independent.

  1. fitted input called prediction [Sec. 3.2.2, Sec. 3.3, Sec. 4.2.1, Fig. 5, Table 4]
    "We experiment with several sorting orders, and select the ascending order for its superior performance. … As shown in Figure 5, for most tasks, AURORA LONG reaches performance peak even with a visual token kept ratio of only 0.1."

    The final model fixes two design choices on the basis of ablations run directly on the same test benchmarks that are later reported as evidence of competitive performance. Section 3.2.2 selects ascending token order using, per Table 4, ANet, VATEX, VDC, and MovieChat-1K; Section 3.3 sets the visual token retention ratio to 0.1, and Section 4.2.1 validates that ratio on ANet, VATEX, VDC, and MovieChat-1K (Fig. 5).

full rationale

The paper's central contribution pairs efficiency with competitiveness on long-video benchmarks. The efficiency claim is genuinely independent: Figure 1 directly measures GPU memory versus frame count, and the constant-state RWKV design gives a structural, not fitted, memory advantage. The accuracy claim, however, is partially circular for the four benchmarks used to select both the 0.1 retention ratio and the ascending token order: the same ANet, VATEX, VDC, and MovieChat-1K sets appear in the ablation that fixes these choices and in the final result tables. This is benchmark-driven hyperparameter selection rather than a by-construction equivalence, and it does not force the results on MLVU (a long-video benchmark not shown in the ablation) or MVBench (a short-video benchmark also absent from the tuning curves). I do not see load-bearing circularity in the self-citations: MovieChat-1K and VDC are public benchmarks, and citing AuroraCap for the training recipe is normal methodology reuse. The unsupported assertion about the RWKV loss curve (Sec. 4.1.3) and the absence of accuracy evidence at 1,024 frames are real evidentiary gaps, but they are correctness risks, not circular reasoning. Overall, the reported accuracy is partly fitted on four benchmarks, while the efficiency result and several benchmark results retain independent content, so a moderate score of 4 is appropriate.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

No new physical or architectural entities are postulated. The free parameters are the token retention ratio, the token ordering choice, and the training frame count, all selected using evaluation benchmarks. The axioms are standard background plus domain assumptions about RNN long-context behavior and the informativeness of merged tokens.

free parameters (3)
  • visual token retention ratio r = 0.1
    Set to 0.1 during language-stage training based on ablations in Fig. 5 that show peak or near-peak performance on ANet, VATEX, VDC, and MovieChat-1K. This is a hyperparameter fitted to test benchmarks.
  • sorted token order = ascending by merged token size
    Selected in Table 4 because ascending order gives the best scores on ANet, VATEX, VDC, and MovieChat-1K. Categorical choice tuned on evaluation benchmarks.
  • number of training frames = up to 60
    Chosen to fit within RWKV's 4k pretrained context with 0.1 retention; no systematic ablation is shown for this choice.
assumptions (5)
  • standard math RWKV-6 recurrence correctly models sequential dependencies at arbitrary inference length
    The paper adopts the RWKV equations from [66] without derivation; this is standard background for the proposed method.
  • domain assumption Linear RNN LLMs overfit to pretrained context length and degrade beyond it, per [17]
    Used to justify token merging; this is an empirical claim from prior work, not re-established here. Location: Section 3.2.2.
  • domain assumption Merging similar visual tokens to a retention ratio of 0.1 preserves enough information for video QA
    The central premise of the method; supported only by ablations on evaluation benchmarks, not by a separate validation set. Location: Section 4.2.1.
  • ad hoc to paper Ascending sorting of merged tokens improves RWKV's use of its implicit positional encoding
    Proposed by the authors and justified only by post hoc Table 4 results and an informal claim about data-dependent token shift. Location: Section 4.2.2.
  • domain assumption LLM-assisted evaluation (LMMs-Eval) produces comparable scores across models
    All open-ended QA scores rely on automatic LLM judges; differences may reflect judge bias rather than model capability. Location: Section 4.1.2.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AuroraLong: Bringing RNNs Back to Efficient Open-Ended Video Understanding." pith.science (2026). https://pith.science/paper/EORAJE6J

@misc{pith2026250702591,
  author       = {Pith},
  title        = {Pith review of: AuroraLong: Bringing RNNs Back to Efficient Open-Ended Video Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EORAJE6J}},
  note         = {Machine review of arXiv:2507.02591}
}
read the original abstract

The challenge of long video understanding lies in its high computational complexity and prohibitive memory cost, since the memory and computation required by transformer-based LLMs scale quadratically with input sequence length. We propose AuroraLong to address this challenge by replacing the LLM component in MLLMs with a linear RNN language model that handles input sequence of arbitrary length with constant-size hidden states. To further increase throughput and efficiency, we combine visual token merge with linear RNN models by reordering the visual tokens by their sizes in ascending order. Despite having only 2B parameters and being trained exclusively on public data, AuroraLong achieves performance comparable to Transformer-based models of similar size trained on private datasets across multiple video benchmarks. This demonstrates the potential of efficient, linear RNNs to democratize long video understanding by lowering its computational entry barrier. To our best knowledge, we are the first to use a linear RNN based LLM backbone in a LLaVA-like model for open-ended video understanding.

Figures

Figures reproduced from arXiv: 2507.02591 by the authors.

Figure 1
Figure 1. GPU memory cost in gigabyte (GB) (y-axis) v.s. frame [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Visualization of the Sorted Token Merge (S-ToMe) algorithm used in A [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. AURORALONG prompting strategy overview. Following VisualRWKV [35], we adopt sandwich prompting strategy, which places image tokens in the middle of instruction tokens. the visual modeling capabilities of linear attention models like RWKV [65, 66] and Mamba [22, 31] by bidirectionally scanning visual tokens, leading to additional computation overhead. Therefore, we propose a simpler, training-free vi￾sual token reord… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Compared to transformer models of similar size, A [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Ablation study of token merging in short video question answering on ANet [ [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Comparison between different training strategy in Lan [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. $M^3-Verse$: A "Spot the Difference" Challenge for Large Multimodal Models

    cs.CV 2025-12 conditional novelty 6.0 of 10

    A new benchmark tests whether large multimodal models can compare paired 'before and after' videos to detect scene changes, and finds current models perform near random.

Reference graph

Works this paper leans on

120 extracted references · 6 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. 1, 2

    arXiv 2023

  2. [2]

    Hired: Attention-guided token dropping for efficient inference of high-resolution vision-language models in resource-constrained environments

    Kazi Hasan Ibn Arif, JinYi Yoon, Dimitrios S Nikolopou- los, Hans Vandierendonck, Deepu John, and Bo Ji. Hired: Attention-guided token dropping for efficient inference of high-resolution vision-language models in resource-constrained environments. arXiv preprint arXiv:2408.10945, 2024. 2

    arXiv 2024

  3. [3]

    In- finibench: A comprehensive benchmark for large multi- modal models in very long video understanding

    Kirolos Ataallah, Chenhui Gou, Eslam Abdelrahman, Khushbu Pahwa, Jian Ding, and Mohamed Elhoseiny. In- finibench: A comprehensive benchmark for large multi- modal models in very long video understanding. arXiv preprint arXiv:2406.19875, 2024. 2

    arXiv 2024

  4. [4]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025. 2, 4

    arXiv 2025

  5. [5]

    Paligemma: A versatile 3b vlm for transfer

    Lucas Beyer, Andreas Steiner, Andr ´e Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, et al. Paligemma: A versatile 3b vlm for transfer. arXiv preprint arXiv:2407.07726, 2024. 1

    arXiv 2024

  6. [6]

    Vane-bench: Video anomaly evaluation benchmark for conversational lmms

    Rohit Bharadwaj, Hanan Gani, Muzammal Naseer, Fa- had Shahbaz Khan, and Salman Khan. Vane-bench: Video anomaly evaluation benchmark for conversational lmms. arXiv preprint arXiv:2406.10326, 2024. 2

    arXiv 2024

  7. [7]

    Token merging: Your ViT but faster

    Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your ViT but faster. In International Conference on Learning Representations, 2023. 2, 3

    2023

  8. [8]

    Activitynet: A large-scale video benchmark for human activity understanding

    Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In Proceed- ings of the ieee conference on computer vision and pattern recognition, pages 961–970, 2015. 6, 7

    2015

Show all 120 references
  1. [9]

    Matryoshka multimodal models

    Mu Cai, Jianwei Yang, Jianfeng Gao, and Yong Jae Lee. Matryoshka multimodal models. arXiv preprint arXiv:2405.17430, 2024. 1

    2024 arXiv

  2. [10]

    View transformer layers from online optimization perspective, 2025

    Wenhao Chai and Weili Xu. View transformer layers from online optimization perspective, 2025. 2

    2025

  3. [11]

    Auroracap: Ef- ficient, performant video detailed captioning and a new benchmark

    Wenhao Chai, Enxin Song, Yilun Du, Chenlin Meng, Vashisht Madhavan, Omer Bar-Tal, Jeng-Neng Hwang, Saining Xie, and Christopher D Manning. Auroracap: Ef- ficient, performant video detailed captioning and a new benchmark. arXiv preprint arXiv:2410.03051, 2024. 4, 5, 6, 7, 8

    arXiv

  4. [12]

    Video mamba suite: State space model as a ver- satile alternative for video understanding

    Guo Chen, Yifei Huang, Jilan Xu, Baoqi Pei, Zhe Chen, Zhiqi Li, Jiahao Wang, Kunchang Li, Tong Lu, and Limin Wang. Video mamba suite: State space model as a ver- satile alternative for video understanding. arXiv preprint arXiv:2403.09626, 2024. 3

    2024 arXiv

  5. [13]

    Sharegpt4video: Improving video understanding and generation with better captions

    Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, et al. Sharegpt4video: Improving video understanding and generation with better captions. arXiv preprint arXiv:2406.04325, 2024. 2, 6, 7

    2024 arXiv

  6. [14]

    An image is worth 1/2 tokens after layer 2: Plug-and-play inference ac- celeration for large vision-language models

    Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Jun- yang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference ac- celeration for large vision-language models. arXiv preprint arXiv:2403.06764, 2024. 2

    2024 arXiv

  7. [15]

    Motion- llm: Understanding human behaviors from human motions and videos

    Ling-Hao Chen, Shunlin Lu, Ailing Zeng, Hao Zhang, Benyou Wang, Ruimao Zhang, and Lei Zhang. Motion- llm: Understanding human behaviors from human motions and videos. arXiv preprint arXiv:2405.20340, 2024. 2

    2024 arXiv

  8. [17]

    Stuffed mamba: State col- lapse and state capacity of rnn-based long-context model- ing

    Yingfa Chen, Xinrong Zhang, Shengding Hu, Xu Han, Zhiyuan Liu, and Maosong Sun. Stuffed mamba: State col- lapse and state capacity of rnn-based long-context model- ing. arXiv preprint arXiv:2410.07145, 2024. 2, 3, 7

    2024

  9. [18]

    InternVL: Scaling up vision founda- tion models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Zhong Muyan, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. InternVL: Scaling up vision founda- tion models and aligning for generic visual-linguistic tasks. arXiv preprint arXiv:2312.14238, 2023. 6

    2023 arXiv

  10. [19]

    Expanding performance boundaries of open-source multimodal models with model, data, and test- time scaling

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhang- wei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test- time scaling. arXiv preprint arXiv:2412.05271, 2024. 5, 6, 7

    2024 arXiv

  11. [20]

    How far are we to gpt-4v? clos- ing the gap to commercial multimodal models with open- source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhang- wei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? clos- ing the gap to commercial multimodal models with open- source suites. arXiv preprint arXiv:2404.16821, 2024. 5, 6, 7

    2024 arXiv

  12. [21]

    Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.07476, 2024. 1

    2024 arXiv

  13. [22]

    Transformers are ssms: Gener- alized models and efficient algorithms through structured state space duality

    Tri Dao and Albert Gu. Transformers are ssms: Gener- alized models and efficient algorithms through structured state space duality. arXiv preprint arXiv:2405.21060, 2024. 2, 4

    2024 arXiv

  14. [23]

    Griffin: Mixing gated linear recurrences with local attention for efficient language models.arXiv preprint arXiv:2402.19427, 2024

    Soham De, Samuel L Smith, Anushan Fernando, Alek- sandar Botev, George Cristian-Muraru, Albert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srini- vasan, et al. Griffin: Mixing gated linear recurrences with local attention for efficient language models.arXiv prepri...

    2024 arXiv

  15. [24]

    Gate-variants of gated re- current unit (gru) neural networks

    Rahul Dey and Fathi M Salem. Gate-variants of gated re- current unit (gru) neural networks. In 2017 IEEE 60th international midwest symposium on circuits and systems (MWSCAS), pages 1597–1600. IEEE, 2017. 2

    2017

  16. [25]

    Towards event-oriented long video under- standing

    Yifan Du, Kun Zhou, Yuqi Huo, Yifan Li, Wayne Xin Zhao, Haoyu Lu, Zijia Zhao, Bingning Wang, Weipeng Chen, and Ji-Rong Wen. Towards event-oriented long video under- standing. arXiv preprint arXiv:2406.14129, 2024. 2

    2024 arXiv

  17. [26]

    Vision-rwkv: Efficient and scalable vi- sual perception with rwkv-like architectures.arXiv preprint arXiv:2403.02308, 2024

    Yuchen Duan, Weiyun Wang, Zhe Chen, Xizhou Zhu, Lewei Lu, Tong Lu, Yu Qiao, Hongsheng Li, Jifeng Dai, and Wenhai Wang. Vision-rwkv: Efficient and scalable vi- sual perception with rwkv-like architectures.arXiv preprint arXiv:2403.02308, 2024. 3

    2024 arXiv

  18. [27]

    Videoagent: A memory-augmented mul- timodal agent for video understanding

    Yue Fan, Xiaojian Ma, Rujie Wu, Yuntao Du, Jiaqi Li, Zhi Gao, and Qing Li. Videoagent: A memory-augmented mul- timodal agent for video understanding. In European Con- ference on Computer Vision, pages 75–92. Springer, 2025. 2

    2025

  19. [28]

    Were rnns all we needed? arXiv preprint arXiv:2410.01201, 2024

    Leo Feng, Frederick Tung, Mohamed Osama Ahmed, Yoshua Bengio, and Hossein Hajimirsadegh. Were rnns all we needed? arXiv preprint arXiv:2410.01201, 2024. 2

    2024 arXiv

  20. [29]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075, 2024. 2

    2024 arXiv

  21. [30]

    Long short-term memory

    Alex Graves and Alex Graves. Long short-term memory. Supervised sequence labelling with recurrent neural net- works, pages 37–45, 2012. 2

    2012

  22. [31]

    Mamba: Linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023. 2, 4

    2023 arXiv

  23. [32]

    Shot2story20k: A new benchmark for comprehen- sive understanding of multi-shot videos

    Mingfei Han, Linjie Yang, Xiaojun Chang, and Heng Wang. Shot2story20k: A new benchmark for comprehen- sive understanding of multi-shot videos. arXiv preprint arXiv:2312.10300, 2023. 2

    2023 arXiv

  24. [33]

    Ma-lmm: Memory-augmented large multimodal model for long-term video understanding

    Bo He, Hengduo Li, Young Kyun Jang, Menglin Jia, Xuefei Cao, Ashish Shah, Abhinav Shrivastava, and Ser-Nam Lim. Ma-lmm: Memory-augmented large multimodal model for long-term video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogniti...

    2024

  25. [34]

    Masked autoencoders are scal- able vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll´ar, and Ross Girshick. Masked autoencoders are scal- able vision learners. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 16000–16009, 2022. 8

    2022

  26. [35]

    Visualrwkv: Exploring recurrent neural networks for visual language models

    Haowen Hou, Peigen Zeng, Fei Ma, and Fei Richard Yu. Visualrwkv: Exploring recurrent neural networks for visual language models. arXiv preprint arXiv:2406.13362, 2024. 3, 4, 7

    2024 arXiv

  27. [36]

    Token compensator: Altering in- ference cost of vision transformer without re-tuning

    Shibo Jie, Yehui Tang, Jianyuan Guo, Zhi-Hong Deng, Kai Han, and Yunhe Wang. Token compensator: Altering in- ference cost of vision transformer without re-tuning. In European Conference on Computer Vision , pages 76–94. Springer, 2025. 2

    2025

  28. [37]

    Chat-univi: Unified visual representation em- powers large language models with image and video under- standing

    Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation em- powers large language models with image and video under- standing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13...

    2024

  29. [38]

    Exploring enhanced contextual infor- mation for video-level object tracking

    Ben Kang, Xin Chen, Simiao Lai, Yang Liu, Yi Liu, and Dong Wang. Exploring enhanced contextual infor- mation for video-level object tracking. arXiv preprint arXiv:2412.11023, 2024. 3

    2024 arXiv

  30. [39]

    Transformers are rnns: Fast autore- gressive transformers with linear attention

    Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Franc ¸ois Fleuret. Transformers are rnns: Fast autore- gressive transformers with linear attention. In International conference on machine learning, pages 5156–5165. PMLR,

  31. [40]

    Rethinking posi- tional encoding in language pre-training

    Guolin Ke, Di He, and Tie-Yan Liu. Rethinking posi- tional encoding in language pre-training. arXiv preprint arXiv:2006.15595, 2020. 3

    2006 arXiv

  32. [41]

    Video token merging for long-form video under- standing

    Seon-Ho Lee, Jue Wang, Zhikang Zhang, David Fan, and Xinyu Li. Video token merging for long-form video under- standing. arXiv preprint arXiv:2410.23782, 2024. 2

    2024 arXiv

  33. [42]

    Minimax-01: Scaling foundation models with lightning attention

    Aonian Li, Bangwei Gong, Bo Yang, Boji Shan, Chang Liu, Cheng Zhu, Chunhao Zhang, Congchao Guo, Da Chen, Dong Li, et al. Minimax-01: Scaling foundation models with lightning attention. arXiv preprint arXiv:2501.08313,

    arXiv

  34. [43]

    Lmms-eval: Accelerating the development of large multimoal models, 2024

    Bo Li, Peiyuan Zhang, Kaichen Zhang, Fanyi Pu, Xinrun Du, Yuhao Dong, Haotian Liu, Yuanhan Zhang, Ge Zhang, Chunyuan Li, and Ziwei Liu. Lmms-eval: Accelerating the development of large multimoal models, 2024. 5

    2024

  35. [44]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 5, 6, 7

    2024 arXiv

  36. [45]

    Aria: An open multimodal native mixture- of-experts model

    Dongxu Li, Yudong Liu, Haoning Wu, Yue Wang, Zhiqi Shen, Bowen Qu, Xinyao Niu, Guoyin Wang, Bei Chen, and Junnan Li. Aria: An open multimodal native mixture- of-experts model. arXiv preprint arXiv:2410.05993, 2024. 2

    2024 arXiv

  37. [46]

    Mvbench: A comprehensive multi-modal video under- standing benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video under- standing benchmark. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 2219...

    2024

  38. [47]

    Videomamba: State space model for efficient video understanding

    Kunchang Li, Xinhao Li, Yi Wang, Yinan He, Yali Wang, Limin Wang, and Yu Qiao. Videomamba: State space model for efficient video understanding. In European Conference on Computer Vision, pages 237–255. Springer,

  39. [48]

    Independently recurrent neural network (indrnn): Building a longer and deeper rnn

    Shuai Li, Wanqing Li, Chris Cook, Ce Zhu, and Yanbo Gao. Independently recurrent neural network (indrnn): Building a longer and deeper rnn. In Proceedings of the IEEE con- ference on computer vision and pattern recognition , pages 5457–5466, 2018. 2

    2018

  40. [49]

    Mamba- nd: Selective state space modeling for multi-dimensional data

    Shufan Li, Harkanwar Singh, and Aditya Grover. Mamba- nd: Selective state space modeling for multi-dimensional data. In European Conference on Computer Vision, pages 75–92. Springer, 2024. 3

    2024

  41. [50]

    Llama-vid: An image is worth 2 tokens in large language models

    Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. arXiv preprint arXiv:2311.17043, 2023. 2

    2023 arXiv

  42. [51]

    Llama-vid: An image is worth 2 tokens in large language models

    Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. In European Conference on Computer Vision, pages 323–340. Springer, 2025. 6, 7

    2025

  43. [52]

    Video-llava: Learning united visual represen- tation by alignment before projection

    Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual represen- tation by alignment before projection. arXiv preprint arXiv:2311.10122, 2023. 5

    2023 arXiv

  44. [53]

    Vila: On pre-training for visual language models

    Ji Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. arXiv preprint arXiv:2312.07533, 2023. 6, 7

    2023 arXiv

  45. [54]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural infor- mation processing systems, 36:34892–34916, 2023. 1

    2023

  46. [55]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024. 3

    2024

  47. [56]

    Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024. 1, 6, 7

    2024

  48. [57]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural infor- mation processing systems, 36, 2024. 4

    2024

  49. [58]

    Et bench: Towards open-ended event-level video-language understanding

    Ye Liu, Zongyang Ma, Zhongang Qi, Yang Wu, Ying Shan, and Chang Wen Chen. Et bench: Towards open-ended event-level video-language understanding. arXiv preprint arXiv:2409.18111, 2024. 2

    2024 arXiv

  50. [59]

    Videomam- bapro: A leap forward for mamba in video understanding

    Hui Lu, Albert Ali Salah, and Ronald Poppe. Videomam- bapro: A leap forward for mamba in video understanding. arXiv preprint arXiv:2406.19006, 2024. 3

    2024 arXiv

  51. [60]

    Vista-llama: Reliable video narra- tor via equal distance to visual tokens

    Fan Ma, Xiaojie Jin, Heng Wang, Yuchen Xian, Jiashi Feng, and Yi Yang. Vista-llama: Reliable video narra- tor via equal distance to visual tokens. arXiv preprint arXiv:2312.08870, 2023. 2

    2023 arXiv

  52. [61]

    Video token sparsification for efficient multimodal llms in autonomous driving

    Yunsheng Ma, Amr Abdelraouf, Rohit Gupta, Ziran Wang, and Kyungtae Han. Video token sparsification for efficient multimodal llms in autonomous driving. arXiv preprint arXiv:2409.11182, 2024. 2

    2024 arXiv

  53. [62]

    Video-chatgpt: Towards detailed video understanding via large vision and language models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424, 2023. 2, 6, 7

    2023 arXiv

  54. [63]

    Egoschema: A diagnostic benchmark for very long- form video language understanding

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long- form video language understanding. Advances in Neural Information Processing Systems, 36:46212–46244, 2023. 2

    2023

  55. [64]

    Videomamba: Spatio-temporal selective state space model

    Jinyoung Park, Hee-Seon Kim, Kangwook Ko, Minbeom Kim, and Changick Kim. Videomamba: Spatio-temporal selective state space model. In European Conference on Computer Vision, pages 1–18. Springer, 2024. 3

    2024

  56. [65]

    Rwkv: Reinventing rnns for the transformer era

    Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, et al. Rwkv: Reinventing rnns for the transformer era. arXiv preprint arXiv:2305.13048, 2023. 2, 4, 8

    2023 arXiv

  57. [66]

    Eagle and finch: Rwkv with matrix-valued states and dynamic re- currence

    Bo Peng, Daniel Goldstein, Quentin Anthony, Alon Al- balak, Eric Alcaide, Stella Biderman, Eugene Cheah, Xingjian Du, Teddy Ferdinan, Haowen Hou, et al. Eagle and finch: Rwkv with matrix-valued states and dynamic re- currence. arXiv preprint arXiv:2404.05892, 2024. 2, 3, 4, 5, 7, 8

    2024 arXiv

  58. [67]

    Rwkv-7” goose” with expressive dynamic state evolution

    Bo Peng, Ruichong Zhang, Daniel Goldstein, Eric Alcaide, Xingjian Du, Haowen Hou, Jiaju Lin, Jiaxing Liu, Janna Lu, William Merrill, et al. Rwkv-7” goose” with expressive dynamic state evolution. arXiv preprint arXiv:2503.14456,

    arXiv

  59. [68]

    Vl-mamba: Ex- ploring state space models for multimodal learning

    Yanyuan Qiao, Zheng Yu, Longteng Guo, Sihan Chen, Zijia Zhao, Mingzhen Sun, Qi Wu, and Jing Liu. Vl-mamba: Ex- ploring state space models for multimodal learning. arXiv preprint arXiv:2403.13600, 2024. 3

    2024 arXiv

  60. [69]

    Lightning attention-2: A free lunch for handling unlimited sequence lengths in large language models

    Zhen Qin, Weigao Sun, Dong Li, Xuyang Shen, Weixuan Sun, and Yiran Zhong. Lightning attention-2: A free lunch for handling unlimited sequence lengths in large language models. arXiv preprint arXiv:2401.04658, 2024. 3

    2024 arXiv

  61. [70]

    Various lengths, constant speed: Efficient language modeling with lightning attention

    Zhen Qin, Weigao Sun, Dong Li, Xuyang Shen, Weixuan Sun, and Yiran Zhong. Various lengths, constant speed: Efficient language modeling with lightning attention. arXiv preprint arXiv:2405.17381, 2024. 3

    2024 arXiv

  62. [71]

    Automated as- sistance for creative writing with an rnn language model

    Melissa Roemmele and Andrew S Gordon. Automated as- sistance for creative writing with an rnn language model. In Companion Proceedings of the 23rd International Confer- ence on Intelligent User Interfaces, pages 1–2, 2018. 2

    2018

  63. [72]

    Bidirectional recur- rent neural networks

    Mike Schuster and Kuldip K Paliwal. Bidirectional recur- rent neural networks. IEEE transactions on Signal Process- ing, 45(11):2673–2681, 1997. 2

    1997

  64. [73]

    Llava-prumerge: Adaptive token reduc- tion for efficient large multimodal models

    Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. Llava-prumerge: Adaptive token reduc- tion for efficient large multimodal models. arXiv preprint arXiv:2403.15388, 2024. 1

    2024

  65. [74]

    Disan: Directional self-attention network for rnn/cnn-free language understanding

    Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, Shirui Pan, and Chengqi Zhang. Disan: Directional self-attention network for rnn/cnn-free language understanding. In Pro- ceedings of the AAAI conference on artificial intelligence ,

  66. [75]

    Longvu: Spatiotemporal adaptive compression for long video-language understanding

    Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishnan Varadarajan, Florian Bordes, et al. Longvu: Spatiotemporal adaptive compression for long video-language understanding. arXiv preprint arXiv:2410.17434, 2024. 2

    arXiv

  67. [76]

    Video-xl: Extra-long vision language model for hour-scale video understanding

    Yan Shu, Zheng Liu, Peitian Zhang, Minghao Qin, Jun- jie Zhou, Zhengyang Liang, Tiejun Huang, and Bo Zhao. Video-xl: Extra-long vision language model for hour-scale video understanding. arXiv preprint arXiv:2409.14485 ,

    arXiv

  68. [77]

    Moviechat: From dense token to sparse memory for long video understand- ing

    Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Xun Guo, Tian Ye, Yan Lu, Jenq-Neng Hwang, et al. Moviechat: From dense token to sparse memory for long video understand- ing. arXiv preprint arXiv:2307.16449 , 2023. 2, 5, 6, 7, 8

    2023 arXiv

  69. [78]

    Moviechat+: Question-aware sparse memory for long video question answering

    Enxin Song, Wenhao Chai, Tian Ye, Jenq-Neng Hwang, Xi Li, and Gaoang Wang. Moviechat+: Question-aware sparse memory for long video question answering. arXiv preprint arXiv:2404.17176, 2024. 2

    2024 arXiv

  70. [79]

    Video-mmlu: A massive multi- discipline lecture understanding benchmark

    Enxin Song, Wenhao Chai, Weili Xu, Jianwen Xie, Yuxuan Liu, and Gaoang Wang. Video-mmlu: A massive multi- discipline lecture understanding benchmark. arXiv preprint arXiv:2504.14693, 2025. 2

    2025 arXiv

  71. [80]

    Roformer: Enhanced transformer with rotary position embedding, 2021

    Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2021. 3, 6

    2021

  72. [81]

    Koala: Key frame-conditioned long video-llm

    Reuben Tan, Ximeng Sun, Ping Hu, Jui-hsien Wang, Hanieh Deilamsalehy, Bryan A Plummer, Bryan Russell, and Kate Saenko. Koala: Key frame-conditioned long video-llm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13581– 13591, 2024. 2

    2024

  73. [82]

    Dycoke: Dynamic compression of tokens for fast video large language models

    Keda Tao, Can Qin, Haoxuan You, Yang Sui, and Huan Wang. Dycoke: Dynamic compression of tokens for fast video large language models. arXiv preprint arXiv:2411.15024, 2024. 2

    2024 arXiv

  74. [83]

    Llama: Open and efficient foundation language mod- els

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth ´ee Lacroix, Bap- tiste Rozi `ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language mod- els. arXiv preprint arXiv:2302.13971, 2023. 1, 2, 3

    2023 arXiv

  75. [84]

    Attention is all you need

    A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017. 3

    2017

  76. [86]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. 1, 5, 6, 7

    2024 arXiv

  77. [87]

    Cogvlm: Visual expert for pretrained language models

    Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. Cogvlm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.03079, 2023. 3

    2023 arXiv

  78. [88]

    Lvbench: An extreme long video un- derstanding benchmark

    Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xi- aohan Zhang, Ji Qi, Shiyu Huang, Bin Xu, Yuxiao Dong, Ming Ding, et al. Lvbench: An extreme long video un- derstanding benchmark. arXiv preprint arXiv:2406.08035,

    arXiv

  79. [89]

    Vatex: A large-scale, high-quality multilingual dataset for video-and-language research

    Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. Vatex: A large-scale, high-quality multilingual dataset for video-and-language research. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4581–4591, 2019. 6, 7, 8

    2019

  80. [90]

    Videoagent: Long-form video understand- ing with large language model as agent

    Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung-Levy. Videoagent: Long-form video understand- ing with large language model as agent. arXiv preprint arXiv:2403.10517, 2024. 2

    2024 arXiv

  81. [91]

    Longvlm: Efficient long video under- standing via large language models

    Yuetian Weng, Mingfei Han, Haoyu He, Xiaojun Chang, and Bohan Zhuang. Longvlm: Efficient long video under- standing via large language models. In European Confer- ence on Computer Vision, pages 453–470. Springer, 2025. 3

    2025

  82. [92]

    Longvideobench: A benchmark for long-context inter- leaved video-language understanding

    Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context inter- leaved video-language understanding. arXiv preprint arXiv:2407.15754, 2024. 2, 5

    2024 arXiv

  83. [93]

    Motionbank: A large-scale video motion benchmark with disentangled rule-based annotations

    Liang Xu, Shaoyang Hua, Zili Lin, Yifan Liu, Feipeng Ma, Yichao Yan, Xin Jin, Xiaokang Yang, and Wenjun Zeng. Motionbank: A large-scale video motion benchmark with disentangled rule-based annotations. arXiv preprint arXiv:2410.13790, 2024. 2

    arXiv

  84. [94]

    Pllava : Parameter-free llava extension from images to videos for video dense captioning, 2024

    Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. Pllava : Parameter-free llava extension from images to videos for video dense captioning, 2024. 2

    2024

  85. [95]

    Longvila: Scaling long-context visual language models for long videos

    Fuzhao Xue, Yukang Chen, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhijian Liu, et al. Longvila: Scaling long-context visual language models for long videos. arXiv preprint arXiv:2408.10188, 2024. 2

    2024 arXiv

  86. [96]

    xgen-mm (blip-3): A family of open large multimodal models

    Le Xue, Manli Shu, Anas Awadalla, Jun Wang, An Yan, Senthil Purushwalkam, Honglu Zhou, Viraj Prabhu, Yu- tong Dai, Michael S Ryoo, et al. xgen-mm (blip-3): A family of open large multimodal models. arXiv preprint arXiv:2408.08872, 2024. 6

    2024

  87. [97]

    Qwen2 technical report

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024. 1, 2

    2024 arXiv

  88. [98]

    Gated linear attention trans- formers with hardware-efficient training

    Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention trans- formers with hardware-efficient training. arXiv preprint arXiv:2312.06635, 2023. 2

    2023 arXiv

  89. [99]

    Gated delta networks: Improving mamba2 with delta rule

    Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule. arXiv preprint arXiv:2412.06464, 2024. 2

    2024 arXiv

  90. [100]

    Parallelizing linear transformers with the delta rule over sequence length

    Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing linear transformers with the delta rule over sequence length. Advances in Neural Information Processing Systems, 37:115491–115522, 2025. 2

    2025

  91. [101]

    mplug-owl: Modularization empowers large language models with multimodality

    Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14178, 2023. 6, 7

    2023 arXiv

  92. [102]

    Activitynet-qa: A dataset for understanding complex web videos via question answering

    Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. In Proceedings of the AAAI Conference on Artificial Intel- ligence, pages 9127–9134, 2019. 8

    2019

  93. [103]

    Movie101v2: Improved movie narration benchmark

    Zihao Yue, Yepeng Zhang, Ziheng Wang, and Qin Jin. Movie101v2: Improved movie narration benchmark. arXiv preprint arXiv:2404.13370, 2024. 2

    2024 arXiv

  94. [104]

    Timesuite: Improving mllms for long video understanding via grounded tuning

    Xiangyu Zeng, Kunchang Li, Chenting Wang, Xinhao Li, Tianxiang Jiang, Ziang Yan, Songze Li, Yansong Shi, Zhen- grong Yue, Yi Wang, et al. Timesuite: Improving mllms for long video understanding via grounded tuning. arXiv preprint arXiv:2410.19702, 2024. 2

    arXiv

  95. [105]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975–11986, 2023. 3

    2023

  96. [106]

    A simple llm framework for long-range video question-answering

    Ce Zhang, Taixi Lu, Md Mohaiminul Islam, Ziyang Wang, Shoubin Yu, Mohit Bansal, and Gedas Bertasius. A simple llm framework for long-range video question-answering. arXiv preprint arXiv:2312.17235, 2023. 2

    2023 arXiv

  97. [107]

    Mm-narrator: Narrating long-form videos with multimodal in-context learning

    Chaoyi Zhang, Kevin Lin, Zhengyuan Yang, Jianfeng Wang, Linjie Li, Chung-Ching Lin, Zicheng Liu, and Li- juan Wang. Mm-narrator: Narrating long-form videos with multimodal in-context learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition...

    2024

  98. [108]

    Video-llama: An instruction-tuned audio-visual language model for video understanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306.02858, 2023. 2, 7

    2023 arXiv

  99. [109]

    Lmms-eval: Reality check on the evaluation of large multi- modal models, 2024

    Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuan- han Zhang, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Lmms-eval: Reality check on the evaluation of large multi- modal models, 2024. 5

    2024

  100. [110]

    Internlm-xcomposer-2.5: A versatile large vision language model supporting long-contextual in- put and output

    Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang, Linke Ouyang, et al. Internlm-xcomposer-2.5: A versatile large vision language model supporting long-contextual in- put and output. arXiv preprint arXiv:2407.03320, 2024. 2

    2024 arXiv

  101. [112]

    Long context transfer from language to vision

    Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv preprint arXiv:2406.16852, 2024. 2, 6, 7

    2024 arXiv

  102. [113]

    [cls] attention is all you need for training- free visual token pruning: Make vlm inference faster.arXiv preprint arXiv:2412.01818, 2024

    Qizhe Zhang, Aosong Cheng, Ming Lu, Zhiyong Zhuo, Minqi Wang, Jiajun Cao, Shaobo Guo, Qi She, and Shang- hang Zhang. [cls] attention is all you need for training- free visual token pruning: Make vlm inference faster.arXiv preprint arXiv:2412.01818, 2024. 2

    2024 arXiv

  103. [114]

    Llama-adapter: Efficient fine-tuning of language models with zero-init attention

    Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Ao- jun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303.16199, 2023. 2

    2023 arXiv

  104. [115]

    ınftybench: Extending long context evaluation beyond 100k tokens

    Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang, Zhiyuan Liu, et al. ınftybench: Extending long context evaluation beyond 100k tokens. In ACL (1), 2024. 2

    2024

  105. [116]

    Llava-next: A strong zero-shot video understanding model,

    Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava-next: A strong zero-shot video understanding model,

  106. [117]

    Gated slot attention for efficient linear- time sequence modeling

    Yu Zhang, Songlin Yang, Ruijie Zhu, Yue Zhang, Leyang Cui, Yiqiao Wang, Bolun Wang, Freda Shi, Bailin Wang, Wei Bi, et al. Gated slot attention for efficient linear- time sequence modeling. arXiv preprint arXiv:2409.07146,

    arXiv

  107. [118]

    Distilling vision-language models on millions of videos

    Yue Zhao, Long Zhao, Xingyi Zhou, Jialin Wu, Chun-Te Chu, Hui Miao, Florian Schroff, Hartwig Adam, Ting Liu, Boqing Gong, et al. Distilling vision-language models on millions of videos. arXiv preprint arXiv:2401.06129, 2024. 1

    2024 arXiv

  108. [119]

    Mlvu: A comprehensive benchmark for multi-task long video understanding

    Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding. arXiv preprint arXiv:2406.04264, 2024. 2, 5, 6

    2024 arXiv

  109. [120]

    Streaming dense video captioning

    Xingyi Zhou, Anurag Arnab, Shyamal Buch, Shen Yan, Austin Myers, Xuehan Xiong, Arsha Nagrani, and Cordelia Schmid. Streaming dense video captioning. arXiv preprint arXiv:2404.01297, 2024. 1, 2

    2024 arXiv

  110. [121]

    Rethinking visual dependency in long-context reason- ing for large vision-language models

    Yucheng Zhou, Zhi Rao, Jun Wan, and Jianbing Shen. Rethinking visual dependency in long-context reason- ing for large vision-language models. arXiv preprint arXiv:2410.19732, 2024. 2

    2024 arXiv

  111. [122]

    Vision mamba: Efficient visual representation learning with bidirectional state space model

    Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model. In International Conference on Machine Learning,

  112. [123]

    Apollo: An exploration of video understanding in large multimodal models

    Orr Zohar, Xiaohan Wang, Yann Dubois, Nikhil Mehta, Tong Xiao, Philippe Hansen-Estruch, Licheng Yu, Xiao- fang Wang, Felix Juefei-Xu, Ning Zhang, et al. Apollo: An exploration of video understanding in large multimodal models. arXiv preprint arXiv:2412.10360, 2024. 7

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.