Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Look Every Frame All at Once: Video-Ma$^2$mba for Efficient Long-form Video Understanding with Multi-Axis Gradient Checkpointing

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that a Mamba-2 backbone plus multi-axis gradient checkpointing cuts long-video LMM activation memory from $\Theta(LS)$ to $\Theta(S)$, enabling 0.8M-token training and 2M-token inference on one 80 GB GPU.

desk verdict The memory analysis for the proposed bi-axis checkpointing is the real contribution, but the accuracy claims are oversold and the exact-recompute description leaves the causal conv in Mamba-2 unaccounted for. read the letter →

arxiv 2411.19460 v1 pith:H57AYK3O submitted 2024-11-29 cs.CV cs.AIcs.LG

Hosu Lee , Junho Kim , Hyunjun Kim , Yong Man Ro This is my paper · ORCID
classification cs.CVcs.AIcs.LG
keywords videounderstandingstatespacemodelsMamba-2gradientcheckpointinglong-contextlanguagelargemultimodalmemoryefficiency1FPSsampling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Video-Ma2mba claims that a large multimodal model for video can stop sampling a handful of frames and instead ingest every frame at 1 FPS, because replacing the Transformer backbone with Mamba-2 turns quadratic memory into linear memory. To make that feasible in training, the paper introduces Multi-Axis Gradient Checkpointing (MA-GC), which stores activations on a grid along both the layer axis and the sequence axis and reduces activation memory from $\Theta(LS)$ to $\Theta(S)$ for a fixed number of layers $L$. If the claim holds, a 3.1B-parameter model can train on 0.8M-token sequences and answer from up to 2M tokens on a single 80 GB GPU, corresponding to more than two hours of video at 1 FPS. On Video-MME and LongVideoBench it scores 45.2 and 43.0 respectively, matching or beating several 7B-8B Transformer-based video models.

What carries the argument

The key mechanism is the multi-axis checkpoint grid. Layer-wise checkpoints store the input to every $l$-th layer; sequence-wise checkpoints store the hidden state every $s$-th time step; their intersections define grid cells that are recomputed forward and then backpropagated through during the backward pass. This is possible only because Mamba-2 (a structured state space model) advances one step at a time and needs the immediately previous state, unlike attention, which needs all past keys and values. The memory formula $M = LS/l + LS/s + ls$ and its minimizer $l=s=\sqrt[3]{LS}$ are what carry the theoretical claim of $\Theta(S)$ memory.

What would settle it

Measure peak activation memory for Mamba-2-2.7B at $S=2^{19}$ with MA-GC on an 80 GB A100 in BF16; the paper predicts roughly 42.2 GB, so a measured value above 80 GB (or failure at $S=2^{20}$) would refute the $\Theta(S)$ bound. Separately, retraining Video-Ma2mba-3.1B with 32-frame sampling under otherwise identical conditions and comparing Video-MME would test whether the 0.6-point advantage of 1 FPS over 32 frames is real.

Watch

Extended reading notes

Core claim

The central discovery is that Mamba-2's Markov hidden-state recurrence, $h_t = A_t h_{t-1} + B_t x_t$, makes gradient checkpointing possible on a second axis: instead of storing activations only every $l$ layers, the model also stores sequence states every $s$ time steps, so gradients are reconstructed cell-by-cell on an $L\times S$ grid. With the checkpoint intervals chosen to minimize $M = LS/l + LS/s + ls$, the memory bound becomes $\Theta((LS)^{2/3})$ in general and $\Theta(S)$ in the long-sequence regime $L^2 \le S$, down from $\Theta(LS)$ without such checkpointing. The paper then builds Video-Ma2mba on this memory engine, training a 3.1B Mamba-2-based LMM on dense 1-FPS video data, and reports that it processes up to 0.8M tokens during training, up to 2M at inference, and achieves 45.2 on Video-MME and 43.0 on LongVideoBench.

Load-bearing premise

The whole argument depends on the assumption that a Mamba-2 hidden state is a complete and exact summary of the past, so the backward pass can be rebuilt from checkpoints spaced every $s$ steps, and on the empirical premise that full 1-FPS input is worth the added memory engineering; if the recurrence cannot be resumed exactly, or if 1 FPS gains are negligible, the central claim unravels.

Editorial extensions

If this is right

  • A single 80 GB GPU can train Video-Ma2mba-3.1B on sequences of 0.8M tokens, and the same model can answer with up to 2M input tokens, so videos beyond two hours at 1 FPS are in reach without frame dropping.
  • Memory for Mamba-2 activations grows linearly with sequence length rather than quadratically, so doubling video duration roughly doubles activation memory.
  • The 3.1B model reaches 45.2 on Video-MME and 43.0 on LongVideoBench, outperforming most 7B-8B Transformer-based video LMM baselines despite being smaller.
  • Including the intermediate long-video knowledge stage (Stage 1.5) and 1 FPS full-frame input together add about 1.8 points over the no-Stage-1.5 baseline on Video-MME.
  • MA-GC costs about 35% throughput but extends trainable sequence length by up to 32 times compared with no checkpointing, making the memory-for-time trade-off explicit.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The bi-axis checkpointing recipe should transfer to any recurrent or linear-RNN backbone with a Markov state, so the $O(S)$ memory result is likely a general tool for SSM-based LMMs, not a video-specific trick.
  • An independent repeated-run comparison is needed to see whether the 0.6-point gap between 1 FPS and 32-frame sampling is stable; if it is not, the main proven contribution would be memory engineering rather than accuracy.
  • Combining MA-GC with targeted frame selection or memory banks could address the paper's own note that salience-based sampling may be better for very long videos; MA-GC would then handle memory while retrieval handles relevance.
  • Because throughput drops 35%, the method's practical value is best tested on tasks where dense temporal coverage matters more than wall-clock speed, such as long-duration activity localization or narrative reasoning.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Video-Ma2mba, a video large multi-modal model that replaces the Transformer backbone with a Mamba-2 state-space model, and introduces Multi-Axis Gradient Checkpointing (MA-GC) to reduce training activation memory from O(L·S) to O(S). The model is trained in three stages, including a long-video knowledge stage on the authors' SceneWalk dataset, and is evaluated on Video-MME, LongVideoBench, ActivityNetQA, VideoChatGPT, and MVBench. The main claimed contributions are (i) a linear-complexity backbone for long video, (ii) a bi-axis gradient checkpointing scheme with a memory analysis in Eqs. (4)-(11), and (iii) empirical results showing that a 3.1B model can process sequences up to 0.8M tokens during training and 2M tokens at inference on a single 80GB GPU.

Significance. If the central claims hold, the memory engineering result is useful: the paper provides a parameter-free scaling analysis, reports measured peak memory across sequence lengths, and the qualitative trend in Table 3 is consistent with a sublinear memory improvement over no checkpointing. The system-level demonstration of 2M-token inference on one GPU is also noteworthy. However, the accuracy comparisons are overstated: there is no same-scale Transformer baseline trained under identical conditions, and the headline numbers are essentially tied with existing 7B-8B models (e.g., 45.2 vs 45.3 for SliME on Video-MME). More importantly, the exact-recompute premise of MA-GC is not justified for actual Mamba-2 blocks because the causal Conv1d is ignored, and the claimed advantage over existing sequence-axis checkpointing is not established. The novelty and validity of the core method therefore require substantive revision.

major comments (3)
  1. [Section 3.2, Algorithm 1] The exact-recompute premise of MA-GC is not supported for real Mamba-2 blocks. Eq. (1) simplifies the block to h_t = A_t h_{t-1} + B_t x_t, and Algorithm 1 saves only the SSD hidden state h as the sequence-wise checkpoint. In the cited Mamba-2 architecture [11], a causal Conv1d is applied to the input before the SSD, so the block output at time t depends on x_{t-k+1},...,x_t. Starting from a checkpoint at time s with only h_s and x_s, the recomputation at s+1 cannot reproduce the convolution. If the implementation caches convolution buffers, Eq. (4) undercounts memory and the description is incomplete; if it does not, the recomputed forward pass and the resulting gradients are inexact. The paper never states that the Conv1d is removed from the backbone. Since the trained results in Tables 1-2 and the memory measurements in Table 3 rest on this step, the central methodological claim needs either a corrected memory model or an explicit statement of the implemented variant.
  2. [Section 3.3, Eqs. (4)-(11)] The claimed improvement over existing gradient checkpointing omits the sequence-axis checkpointing for RNNs from Chen et al. [6], which the paper itself cites. For a fixed stack of L layers, storing sequence checkpoints every s steps requires Θ(S + LS/s + Ls) peak memory (input activations, hidden states, and recomputation cell); choosing s = √S gives Θ(S), and the constant is no larger than what MA-GC achieves with its optimal layer interval l = L in Eq. (4), namely S + L√S + L√S. Thus the layer axis of MA-GC does not reduce the asymptotic memory below the existing sequence-axis method; it only adds the S-term for layer-wise checkpoints. The comparison in Table 3 is only against layer-grouping baselines ('GC on' and 'Sqrt GC'), not against time-axis checkpointing from [6]. The paper should compare against sequence-axis-only checkpointing and clarify what the layer axis contributes beyond Chen et al.
  3. [Section 4.2, Tables 1 and 4] The accuracy claims are overstretched relative to the evidence. Table 1 compares only against published numbers from models with different training data, backbones, and evaluation protocols; no same-scale Transformer baseline trained under identical conditions is provided. On Video-MME, Video-Ma2mba-3.1B (45.2) is statistically tied with SliME-8B (45.3), so the abstract's 'substantial advantages over existing frameworks' is unsupported. In Table 4, the key motivational comparison between full 1 FPS input and 32-frame sampling (45.2 vs 44.6) is a 0.6-point gap with no error bars or repeated runs. The text in Section 4.2 additionally reports a '+1.7 points (4.1%) improvement' that does not match any clean row of Table 4 (the closest all-stage comparison is 45.2 vs 43.7, a 1.5-point gap). The evidence that processing all frames at 1 FPS materially improves accuracy over dense-but-sampled input is not yet convincing.
minor comments (5)
  1. [Table 3] In the 1.3B MA-GC row, the entry at S=2^13 appears as '.2.1' and should be '2.1'.
  2. [Section 3.5 vs Table 6] The text states that models are trained for one epoch at each stage, while Table 6 reports 'train epochs 2' for all stages; this contradiction should be resolved.
  3. [Section 4.2, Table 4] The reported '+1.7 points (4.1%)' improvement does not correspond to a direct comparison in the table; please specify which rows are being compared.
  4. [Section 3.3 vs Table 3] The measured MA-GC values in Table 3 increase by a factor of roughly 1.6 per doubling of S, which appears more consistent with a sublinear Θ((LS)^{2/3}) trend than with the claimed Θ(S) boundary solution; the authors should report the actual l and s used for each row and reconcile the measured scaling with Eq. (10).
  5. [Section 4.1] The evaluation uses gpt-3.5-turbo-0125 for response scoring, but no temperature, decoding parameters, or prompt template for the scorer are given; this limits reproducibility.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the MA-GC Θ(S) memory reduction is derived from a parameter-free grid-checkpointing model and validated by measured peak memory, not by fitting; the only self-referential element is the SceneWalk long-video dataset [19] used for Stage 1.5 training, which is not load-bearing for the memory or benchmark claims.

full rationale

The paper's central memory claim (Eq. 10: M*=Θ(S)) follows from minimizing Eq. (4), M = LS/l + LS/s + ls, a parameter-free accounting of layer-wise checkpoints, sequence-wise checkpoints, and grid-cell activations; no fitted constant enters the asymptotic result, and Table 3 reports measured peak memory against an external hardware baseline rather than reproducing a fitted target. The benchmark evaluations use external datasets (Video-MME, LongVideoBench, ActivityNetQA, VideoChatGPT, MVBench), so the accuracy claims are not circular. The only same-author citation is [19] (SceneWalk), used as Stage 1.5 long-video training data in Section 3.5; even though this dataset comes from the same group, it is an input to training and Table 4 provides an in-paper ablation for the Stage 1.5 gain, so it is not load-bearing evidence for the memory derivation. The Discussion's admission that selective frame sampling 'may prove to be a more effective modeling strategy' weakens the motivation but is a stated limitation, not a circular step. A possible inexactness in exact recomputation if the causal Conv1d of the Mamba-2 block is not checkpointed is a correctness/implementation concern under the paper's assumptions, not a reduction of the derivation to its own inputs; it is therefore outside the circularity definition used here.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central memory claim rests on the Markovian SSM recurrence, the proportional-cost memory model, and standard calculus; these are reasonable domain assumptions. The only self-referential input is the SceneWalk dataset from the same authors. No free parameters are fitted to the reported benchmark results; the checkpoint intervals l and s are algorithmic hyperparameters chosen via the derived optimum.

free parameters (2)
  • Layer checkpoint interval l = ≈(LS)^{1/3}
    Chosen to minimize Eq (4); affects the claimed O(S) memory bound. An algorithmic hyperparameter, not fitted to benchmark data.
  • Sequence checkpoint interval s = ≈(LS)^{1/3}, rounded to a multiple of 256
    Chosen to minimize Eq (4) subject to the SSD scan chunk constraint; affects the claimed O(S) memory bound.
assumptions (5)
  • domain assumption Mamba-2 SSD hidden-state recurrence is Markovian: h_t depends only on h_{t-1} and x_t (Eq 1).
    Required for sequence-axis checkpointing with exact recomputation (Sec 3.2).
  • domain assumption Activation memory is proportional to element counts in SSM states and checkpoints (Eqs 4, 12-16).
    The memory model ignores allocator overhead, fragmentation, and non-SSM components; Table 7 constants are derived from model config.
  • standard math Extreme Value Theorem and Fermat's Theorem applied to integer-valued l,s via continuous relaxation (Sec 3.3).
    Used to derive the critical point; integer rounding is handled by boundary cases.
  • domain assumption SceneWalk dataset (same authors, [19]) provides valid long-video knowledge for Stage 1.5.
    Training data quality is assumed; dataset not publicly released.
  • domain assumption Benchmarks Video-MME, LongVideoBench, etc. are reliable measures of video understanding.
    Standard external evaluation; no manual verification by authors.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Look Every Frame All at Once: Video-Ma$^2$mba for Efficient Long-form Video Understanding with Multi-Axis Gradient Checkpointing." pith.science (2026). https://pith.science/paper/H57AYK3O

@misc{pith2026241119460,
  author       = {Pith},
  title        = {Pith review of: Look Every Frame All at Once: Video-Ma$^2$mba for Efficient Long-form Video Understanding with Multi-Axis Gradient Checkpointing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/H57AYK3O}},
  note         = {Machine review of arXiv:2411.19460}
}
abstract

With the growing scale and complexity of video data, efficiently processing long video sequences poses significant challenges due to the quadratic increase in memory and computational demands associated with existing transformer-based Large Multi-modal Models (LMMs). To address these issues, we introduce Video-Ma$^2$mba, a novel architecture that incorporates State Space Models (SSMs) within the Mamba-2 framework, replacing the attention mechanisms. This allows the LMMs to scale linearly in terms of time and memory requirements, making it feasible to handle long-duration video content. Furthermore, we enhance the memory efficiency introducing the Multi-Axis Gradient Checkpointing (MA-GC) method, which strategically manages memory by retaining only essential activations across multiple computational axes. Our approach significantly reduces the memory footprint compared to standard gradient checkpointing. Empirical analyses show that Video-Ma$^2$mba can process extensive video sequences-equivalent to millions of tokens or over two hours of continuous sequences at 1 FPS-on a single GPU. By maintaining a detailed capture of temporal dynamics, our model improves the accuracy and relevance of responses in long video understanding tasks, demonstrating substantial advantages over existing frameworks.

Figures

Figures reproduced from arXiv: 2411.19460 by the authors.

Figure 1
Figure 1. Memory usage comparison across sequence lengths [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of MA-GC grid structure. Checkpoints are [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The overall summarization for the training stages of [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Qualitative examples on Video-MME [13] with Video-Ma2mba-3.1B. Question: What is the woman doing in the outdoor field during the daytime? Answer: The woman is participating in a pole vaulting competition. Prediction: She is running on the track, holding a pole vault po…
Figure 5
Figure 5. Figure 5: Qualitative examples from the Generative Subset of VideoChatGPT [ [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Qualitative examples on LongVideoBench [ [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ReFoCUS: Reinforcement-guided Frame Optimization for Contextual Understanding

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A reinforcement-learned frame selection policy, trained with reward margins from a reference video-LLM, improves video QA accuracy of LLaVA-OV and InternVL3 across several benchmarks.

Reference graph

Works this paper leans on

50 extracted references · 19 canonical work pages · cited by 1 Pith paper

  1. [11]

    Transformers are ssms: General- ized models and efficient algorithms through structured state space duality

    Tri Dao and Albert Gu. Transformers are ssms: General- ized models and efficient algorithms through structured state space duality. ArXiv, abs/2405.21060, 2024. 2, 3, 7

    arXiv 2024

  2. [6]

    Training deep nets with sublinear memory cost

    Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. ArXiv, abs/1604.06174, 2016. 2, 3, 4

    arXiv 2016

  3. [1]

    Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 1(2):3, 2023. 7

    arXiv 2023

  4. [2]

    Frozen in time: A joint video and image encoder for end-to-end retrieval

    Max Bain, Arsha Nagrani, G ¨ul Varol, and Andrew Zisser- man. Frozen in time: A joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF inter- national conference on computer vision , pages 1728–1738,

  5. [3]

    Lan- guage models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Lan- guage models are few-shot learners. Advances in neural in- formation processing systems, 33:1877–1901, 2020. 1, 2

    1901

  6. [4]

    Sharegpt4video: Improving video understand- ing and generation with better captions

    Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, et al. Sharegpt4video: Improving video understand- ing and generation with better captions. arXiv preprint arXiv:2406.04325, 2024. 7

    arXiv 2024

  7. [5]

    Extending context window of large language models via positional interpolation

    Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuan- dong Tian. Extending context window of large language models via positional interpolation. ArXiv, abs/2306.15595,

    arXiv

  8. [7]

    Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.07476, 2024. 7

    arXiv 2024

Show all 50 references
  1. [8]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhang- hao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yong- hao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023. 2

    2023

  2. [9]

    InstructBLIP: Towards general-purpose vision- language models with instruction tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards general-purpose vision- language models with instruction tuning. In Advances in Neural Information Processing Systems, 2023. 1

    2023

  3. [10]

    Carbonell, Quoc V

    Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc V . Le, and Ruslan Salakhutdinov. Transformer-xl: At- tentive language models beyond a fixed-length context. In Annual Meeting of the Association for Computational Lin- guistics, 2019. 2

    2019

  4. [12]

    Jeffrey L. Elman. Finding structure in time. Cogn. Sci., 14: 179–211, 1990. 3

    1990

  5. [13]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075, 2024. 7, 3

    2024 arXiv

  6. [14]

    Gemini, 2023

    Google. Gemini, 2023. 1

    2023

  7. [15]

    Mamba: Linear-time sequence mod- eling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence mod- eling with selective state spaces. ArXiv, abs/2312.00752,

    arXiv

  8. [16]

    Ma-lmm: Memory-augmented large multimodal model for long-term video understanding

    Bo He, Hengduo Li, Young Kyun Jang, Menglin Jia, Xue- fei Cao, Ashish Shah, Abhinav Shrivastava, and Ser-Nam Lim. Ma-lmm: Memory-augmented large multimodal model for long-term video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni...

    2024

  9. [17]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. 6

    2021 arXiv

  10. [18]

    Chat-univi: Unified visual representation em- powers large language models with image and video un- derstanding

    Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation em- powers large language models with image and video un- derstanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 137...

    2024

  11. [19]

    Sa- lova: Segment-augmented long video assistant for targeted retrieval and routing in long-form video analysis

    Junho Kim, Hyunjun Kim, Hosu Lee, and Yong Man Ro. Sa- lova: Segment-augmented long video assistant for targeted retrieval and routing in long-form video analysis. arXiv preprint arXiv:2411.16173, 2024. 6

    2024 arXiv

  12. [20]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 2

    2024 arXiv

  13. [21]

    Mvbench: A comprehensive multi-modal video understand- ing benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understand- ing benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195–...

    2024

  14. [22]

    Llama-vid: An image is worth 2 tokens in large language models

    Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. In European Conference on Computer Vision, pages 323–340. Springer, 2025. 3, 7

    2025

  15. [23]

    Video-llava: Learning united visual rep- resentation by alignment before projection

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual rep- resentation by alignment before projection. arXiv preprint arXiv:2311.10122, 2023. 1, 2, 7

    2023 arXiv

  16. [24]

    Vila: On pre-training for vi- sual language models

    Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Moham- mad Shoeybi, and Song Han. Vila: On pre-training for vi- sual language models. InProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 26689–26699, 2024. 2

    2024

  17. [25]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744, 2023. 1, 5, 7

    2023 arXiv

  18. [26]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. InAdvances in Neural Information Processing Systems, 2023. 1, 5, 6

    2023

  19. [27]

    Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024. 6 9

    2024

  20. [28]

    St-llm: Large language models are effective tem- poral learners

    Ruyang Liu, Chen Li, Haoran Tang, Yixiao Ge, Ying Shan, and Ge Li. St-llm: Large language models are effective tem- poral learners. In European Conference on Computer Vision, pages 1–18. Springer, 2025. 7

    2025

  21. [29]

    Video-chatgpt: Towards detailed video understanding via large vision and language models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fa- had Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424, 2023. 1, 3, 7

    2023 arXiv

  22. [30]

    OpenAI. ChatGPT. https://openai.com/blog/ chatgpt/, 2023. 1

    2023

  23. [31]

    Gpt-4 technical report, 2023

    OpenAI. Gpt-4 technical report, 2023. 1, 7

    2023

  24. [32]

    GPT-4V(ision) System Card, 2023

    OpenAI. GPT-4V(ision) System Card, 2023. 7

    2023

  25. [33]

    Hello gpt-4o, 2024

    OpenAI. Hello gpt-4o, 2024. 7

    2024

  26. [34]

    Per- ception test: A diagnostic benchmark for multimodal video models

    Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Re- casens, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Mateusz Malinowski, Yi Yang, Carl Doersch, et al. Per- ception test: A diagnostic benchmark for multimodal video models. Advances in Neural Information Processing ...

    2024

  27. [35]

    Smith, and Mike Lewis

    Ofir Press, Noah A. Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. ArXiv, abs/2108.12409, 2021. 2

    2021 arXiv

  28. [36]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  29. [37]

    Zero: Memory optimizations toward training trillion parameter models

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Confer- ence for High Performance Computing, Networking, Storage and Analysis, pages 1–16. IEEE, 2020. 6

    2020

  30. [38]

    Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context

    Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrit- twieser, et al. Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context. arXiv...

    2024 arXiv

  31. [39]

    Moviechat: From dense token to sparse memory for long video understanding

    Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, et al. Moviechat: From dense token to sparse memory for long video understanding. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pa...

    2024

  32. [40]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. ArXiv, abs/2104.09864, 2021. 2

    2021 arXiv

  33. [41]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 1, 2

    2023 arXiv

  34. [42]

    Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N

    Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Neural Infor- mation Processing Systems, 2017. 1, 2

    2017

  35. [43]

    Longvideobench: A benchmark for long-context inter- leaved video-language understanding

    Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context inter- leaved video-language understanding. arXiv preprint arXiv:2407.15754, 2024. 7, 4

    2024 arXiv

  36. [44]

    Next-qa: Next phase of question-answering to explaining temporal actions

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question-answering to explaining temporal actions. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 9777–9786, 2021. 6, 1

    2021

  37. [45]

    Pllava: Parameter-free llava extension from images to videos for video dense captioning

    Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. Pllava: Parameter-free llava extension from images to videos for video dense captioning. arXiv preprint arXiv:2404.16994, 2024. 7

    2024 arXiv

  38. [46]

    Activitynet-qa: A dataset for understanding complex web videos via question answering

    Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yuet- ing Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. In Proceedings of the AAAI Conference on Artificial Intelli- gence, pages 9127–9134, 2019. 6, 7, 1

    2019

  39. [47]

    Video-llama: An instruction-tuned audio-visual language model for video un- derstanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video un- derstanding. arXiv preprint arXiv:2306.02858, 2023. 7

    2023 arXiv

  40. [48]

    Long context transfer from language to vision

    Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv preprint arXiv:2406.16852, 2024. 2

    2024 arXiv

  41. [49]

    Video instruction tuning with synthetic data

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Zi- wei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713, 2024. 6, 1

    2024 arXiv

  42. [50]

    Beyond llava-hd: Diving into high-resolution large multimodal models

    Yi-Fan Zhang, Qingsong Wen, Chaoyou Fu, Xue Wang, Zhang Zhang, Liang Wang, and Rong Jin. Beyond llava-hd: Diving into high-resolution large multimodal models. arXiv preprint arXiv:2406.08487, 2024. 3, 7 10 Look Every Frame All at Once: Video-Ma2mba for Efficient Long-form Vide...

    2024 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.