Pith. sign in

REVIEW 5 major objections 6 minor 3 cited by

Token reduction in hybrid video VLMs is compression, not deletion: keeping 25% of visual tokens matches baseline accuracy while prefilling runs 3.8–4.2x faster.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-08-02 20:09 UTC pith:5TDINR7X

load-bearing objection Practical token reduction for hybrid VLMs with real speedups, but the advertised state-retention probe is missing and the train-time gains are confounded. the 5 major comments →

arxiv 2603.00198 v2 pith:5TDINR7X submitted 2026-02-27 cs.CV cs.AI

Stateful Token Reduction for Long-Video Hybrid VLMs

Jindong Jiang , Amala Sanjay Deshmukh , Kateryna Chumachenko , Karan Sapra , Zhiding Yu , Guilin Liu , Andrew Tao , Pavlo Molchanov
show 2 more authors
Jan Kautz Wonmin Byeon
This is my paper
classification cs.CV cs.AI
keywords stateful token reductionhybrid Mamba–Transformerlong-video VLMtoken pruningprogressive reduction scheduleimplicit attentionquery-conditioned importanceprefilling acceleration
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper studies aggressive token reduction for long-video vision–language models that mix Transformer attention with Mamba-style state-space layers. Its central claim is that reduction in such hybrids is stateful: Mamba's recurrent state accumulates information from all earlier tokens, so when a token is pruned a compressed summary of it persists, making reduction behave like compression rather than deletion. The paper supports this with layer-wise analysis showing importance is concentrated in a small token set within a layer but unstable across layers, motivating a progressive low-to-high reduction schedule and a query-conditioned importance score usable in both attention and Mamba blocks. Empirically, holding 25% of visual tokens yields 3.8–4.2x prefilling speedups with near-baseline accuracy, and light finetuning under reduction pushes accuracy above the unreduced baseline. The same treatment of a Transformer-only model shows the conventional speed–accuracy trade-off.

Core claim

On the paper's own terms, the discovery is that token reduction in Mamba–Transformer hybrids is a form of compression. Unrolling the selective-scan recurrence exposes an attention-like weight per token, with a content-alignment term \bar b_j^T c_t and a cumulative decay; the paper scores Mamba tokens by the alignment alone, dropping the decay because it biases importance toward recent positions. Using these scores, it finds that importance is sparse within layers but that importance rankings are poorly correlated across layers, so aggressive early pruning is unsafe; it therefore schedules reduction progressively, keeping more tokens early and pruning harder deeper in the network. At a 25% vi

What carries the argument

The load-bearing mechanism is Mamba's recurrent state. The unrolled update expresses each output as y_t = sum_j w_{t,j} x_j with w_{t,j} equal to a product of decays from j+1 to t times the alignment \bar b_j^T c_t. The paper treats \bar b_i as a key and c_m as a query, defines an implicit-attention importance score s_i = (1/MG) sum |\bar b_i^T c_m| over text positions and Mamba groups, and omits the decay term to avoid recency bias. This score enables token ranking inside non-attention layers; a progressive sigmoid or step retention schedule then allocates pruning later in the network, where the accumulated state can absorb dropped tokens.

Load-bearing premise

The paper's central claim rests on the assumption that Mamba's recurrent state preserves meaningful, recoverable information about discarded tokens; the representation-based probe promised in the abstract does not appear in the main text or appendix, leaving only indirect benchmark tolerance as evidence that reduction is compression rather than deletion.

What would settle it

Run the probe the abstract promises: after pruning a known set of visual tokens, train linear classifiers to decode which token was pruned from the Mamba state, and compare with the same probe applied to a Transformer's residual stream at matched depth. If retained information is no larger in the hybrid, the stateful-compression mechanism and the claimed hybrid advantage collapse, even if the speedups remain.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • At a 25% token budget, prefilling time-to-first-token drops 3.8–4.2x with near-baseline accuracy on three long-video QA benchmarks.
  • Light finetuning under reduction lets the hybrid model beat its unreduced baseline, while a Transformer baseline loses accuracy at the same compression.
  • Token pruning should be deferred: early single-layer reduction costs 1.2–2.3 points on average, while later progressive reduction is roughly neutral or positive.
  • Query-conditioned selection outperforms average pooling in Mamba layers by about 3.5 points at 25% compression.
  • Reduction overhead stays under one second while extending feasible video length well beyond 256 frames before out-of-memory.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If stateful compression holds, token reduction could be scheduled by what the Mamba state actually contains rather than by layer index alone; a state-saturation probe could tell when pruning is safe.
  • The same implicit-attention scoring should transfer to other recurrent or linear-attention architectures, where pruning may also behave as compression rather than deletion.
  • A decisive follow-up is the representation-probing experiment the abstract promises but the paper never reports: directly measure how much discriminative information about a pruned token remains decodable from Mamba state versus a Transformer residual stream at matched depth. Without it, the compression explanation rests on benchmark tolerance.
  • The finetuning gains under reduction suggest token reduction acts as a regularizer for long-video QA; a testable extension is whether reduction-then-finetune improves generalization on unseen video lengths.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper studies query-conditioned token reduction for hybrid Mamba–Transformer video VLMs. It defines importance scores for attention layers (Eq. 1) and for Mamba layers via an implicit-attention proxy (Eq. 3), analyzes layerwise sparsity and cross-layer importance stability on a 32-frame sample, and proposes low-to-high progressive reduction schedules. On Nemotron-Nano-V2 VL 12B at a ~25% visual-token budget, it reports 3.8–4.2x prefilling TTFT speedups with near-baseline test-time accuracy and above-baseline accuracy with light finetuning; comparisons with Qwen3-VL 8B are used to argue that hybrids tolerate aggressive reduction better than Transformers. The central explanatory claim is that Mamba's recurrent state retains information from pruned tokens, so reduction behaves as compression rather than dropping.

Significance. If substantiated, the central claim would be significant: it identifies an architectural property—stateful memory in Mamba blocks—that changes the speed–accuracy trade-off of token reduction and provides a practical recipe for aggressive reduction in hybrid video VLMs. The empirical scope is broad (three long-video benchmarks, TTFT/latency, component-wise analysis), the schedule parameters are fully enumerated in Appendix A.3, and the ablations cover layer patterns and reduction types. The speedup measurements are concrete and internally consistent. However, the load-bearing mechanistic interpretation is not directly tested: the abstract promises a representation-based probing experiment that does not appear in the main text or appendices, and the train-time 'improvement above baseline' is confounded by a mismatch in training frames. The paper also does not provide code, though the experimental description is reasonably detailed.

major comments (5)
  1. [Abstract; §2.2] The abstract promises 'a representation-based probing method measuring how much information from discarded tokens is retained,' but no such experiment appears in the main text or the appendix. §2.2's 'Mamba Memory State Is Key to Token Reduction' paragraph supports the state-retention claim only with benchmark tolerance and with Figure 6 heatmaps of importance scores; neither measures whether information from pruned tokens persists in the recurrent state. This is not a cosmetic issue: the paper's central mechanism—and the claimed hybrid-specific advantage over Transformers—depends on that state actually preserving information about discarded tokens. Please add a genuine probe (e.g., train linear probes on Mamba states before/after pruning and compare with Transformer residual-stream states) or explicitly re-frame the stateful-compression account as a hypothesis rather than a supported co
  2. [§3.1; Tables 1–2] The claim that train-time reduction improves accuracy over the no-reduction baseline is confounded by training length. The setup states: 'the no-reduction baseline uses 96 frames in the final stage (without reduction), which is the largest that fits in GPU memory,' while Stage 3 with token reduction uses 'up to 128 frames.' Thus the positive deltas in Table 1 (e.g., All: +1.37 avg) and Table 2 could be due to the longer 128-frame training rather than to token reduction. Please provide a matched control, such as a no-reduction baseline trained with the same 128-frame data (or a reduced model trained with 96 frames), or explicitly report both comparisons. Without a matched control, the 'improving above baseline' claim is not attributable to reduction.
  3. [§2.2; Figure 1] The sparsity/stability analysis is based on 'an 32 frames video sample from the validation set'—a single sample. Figure 1 reports no error bars or aggregation over multiple videos, and the text's threshold ('values around 0.5 and below') is used without justification. The claimed difference in cross-layer consistency between hybrid and Transformer models is load-bearing for the proposed early-vs-progressive scheduling strategy. Please aggregate over at least several videos and report standard errors or per-video ranges, or clearly label Figure 1 as an illustrative case study rather than a quantitative result.
  4. [§2.1, Eq. (3); §A.2.3] The decision to drop the position-dependent decay term from the Mamba importance score is justified only by visual inspection of heatmaps ('we empirically observe' in §2.1 and Figure 6). A quantitative ablation would be much stronger: e.g., compare selection with the full implicit-attention weight |w_{t,j}| (Eq. 2) versus the decay-free score (Eq. 3) at identical budgets, reporting accuracy and latency. As written, the choice appears post hoc, and the later statement that omitting decay 'produces a stronger selection signal' is not directly evidenced by the reported experiments.
  5. [§3; Tables 2, 7] The hybrid-vs-Transformer comparison is not fully controlled. Nemotron-Nano-V2 VL is 12B while Qwen3-VL is 8B, and the two models differ in architecture family, vision backbone, and layer composition. Table 2's note says token reduction is applied at 'all attention layers (6 layers)' for both models, but §3.1 describes Qwen3-VL 8B as a 36-layer Transformer; it is unclear whether 6 layers or all 36 layers were reduced for Qwen. The conclusion that 'hybrid models benefit from aggressive reduction, whereas Transformers exhibit the standard trade-off' may reflect model scale, number of reduced layers, or training recipe rather than state memory. Please add a matching-scale or matching-layer-count control, or explicitly discuss these confounds.
minor comments (6)
  1. [Throughout] Typos and grammar issues: 'agressive', 'proning', 'measur', 'fineturning', 'degredation', 'vatiants', 'donates' (Tables 1 and 5 should read 'denotes').
  2. [Table 2] The note 'all attention layers (6 layers)' is inconsistent with the 36-layer Qwen3-VL description in §3.1; please clarify the exact reduction positions for the Transformer model.
  3. [Figures 2–3] The layer-pattern diagrams in Figure 2 are hard to read because the block labels (M/P/A) are small and the reduction locations are not visually distinct. Consider annotating the reduction points explicitly. Figure 3 would benefit from a legend entry for each schedule type as printed.
  4. [§A.3] The schedule table lists duplicate rows for 'All Attn+1M' under Mamba+Attn (Step Decay) with identical compression rates; one of the rows appears to be intended for 'All Attn+2M.' Please correct.
  5. [Figure 6] The caption does not state that a logarithmic color space is used, although the text mentions it; please include this detail in the caption so the reader can interpret the heatmaps.
  6. [§A.3] The sigmoid schedule fixes k=20 and adjusts x0 to hit target budgets, but there is no sensitivity analysis for k or x0. A brief ablation would clarify whether the low-to-high shape, rather than the specific sigmoid parameters, is responsible for the results.

Circularity Check

0 steps flagged

No load-bearing circularity: the token-reduction method is defined independently of the benchmark outcomes, and the main weakness is a missing probe, which is an evidence gap rather than a circular step.

full rationale

The paper's derivation chain is not circular. The query-conditioned importance scores (Eqs. 1 and 3) are defined from model internals and are not constructed from the benchmark accuracies they are used to predict; token selection is top-K by these scores, and the reported speedups and accuracies come from external benchmarks (VideoMME, LongVideoBench, LVBench). The progressive schedule is parameterized to hit token budgets (e.g., x0 ∈ {0.11, 0.24, 0.41} to reach ~25/35/50%), but the budgets are experimental conditions, not fitted predictions of accuracy. The 'stateful compression' claim is at least grounded in the Mamba recurrence (Eqs. 4–7), where earlier tokens enter the state S_t; this is a mathematical property, and the empirical claim of tolerance is benchmarked. No load-bearing argument reduces to a self-citation: [10], [24], [25], and [27] are model/training/dataset references, not proofs of the token-reduction claim. Two non-circular concerns remain: (i) the abstract promises 'a representation-based probing method measuring how much information from discarded tokens is retained,' but no such probe appears in the main text or appendix, so the mechanistic explanation is under-evidenced; and (ii) the train-time 'improves above baseline' comparisons may be confounded because Stage 3 finetuning uses up to 128 frames while 'the no-reduction baseline uses 96 frames in the final stage.' These are evidence/comparability problems, not definitional circularity.

Axiom & Free-Parameter Ledger

2 free parameters · 5 axioms · 0 invented entities

The method relies on two hand-set schedule parameters, the post-hoc choice to drop the Mamba decay term, and the unverified mechanistic premise that Mamba state retains pruned-token information. No new physical entities are introduced. The most important ledger item is the state-retention premise: it is asserted in the abstract, promised to be probed, but never directly tested in the paper.

free parameters (2)
  • sigmoid schedule steepness k = 20
    Fixed by hand; §A.3 Table 8. Controls sharpness of progressive reduction; not derived from data.
  • sigmoid midpoint x0 = 0.11 / 0.24 / 0.41 for 25% / 35% / 50% budgets
    Chosen per target compression rate, §A.3. Determines when aggressive pruning begins; hand-tuned to hit target budgets.
axioms (5)
  • standard math The Mamba-2 recurrence can be unrolled as a weighted sum with weights w_{t,j} = (prod A_u) b_j^T c_t.
    Background from [15]; this is a standard derivation and the paper reproduces it in Eq. (7).
  • ad hoc to paper Absorbing Δ_j into b_j and dropping the decay factor prod A_u yields a better token-importance signal than the full implicit attention weight.
    Figure 6 shows some Mamba layers concentrate full weights on recent tokens; no quantitative criterion is given for omitting decay beyond visual inspection.
  • domain assumption Token importance rankings measured on one 32-frame validation sample generalize to the video benchmarks.
    Section 2.2 uses 'a 32 frames video sample'; no variance, no multi-video statistics, so the stability/sparsity conclusions may not be representative.
  • domain assumption The recurrent state S_t in Mamba preserves enough information from pruned tokens to make aggressive reduction behave like compression rather than dropping.
    This is the central mechanism. The abstract says it is supported by representation-based probing, but no probing experiment appears in the text.
  • domain assumption Finetuning under the reduction schedule is comparable to a baseline without reduction, even though the baseline did not receive the same Stage-3 finetuning or frame lengths.
    Section 3.1 says the no-reduction baseline uses 96 frames and no Stage-3 token-reduction finetuning, while reduced models use up to 128 frames on EAGLE-Video-110K, confounding the accuracy gains.

reviewed 2026-08-02 · how reviews work

0 comments
read the original abstract

Token reduction accelerates long-video vision--language models (VLMs), but existing methods target Transformers, where reduction is treated as token pruning. We study token reduction in hybrid Mamba--Transformer VLMs and find that it is \emph{stateful}: Mamba layers maintain a recurrent state that accumulates information from earlier tokens, allowing discarded tokens to persist, so reduction behaves more like compression than dropping.We support this view with a representation-based probing method measuring how much information from discarded tokens is retained, and analyze layer-wise sparsity and cross-layer importance stability. Our findings show importance is sparse within layers but unstable across layers, making aggressive early pruning unreliable while hybrids remain robust to later reduction.Motivated by this, we propose a hybrid-aware token reduction framework with a low-to-high progressive schedule and a unified query-conditioned importance score for attention and Mamba layers. For Mamba, excluding the position-dependent decay from the recurrence produces a stronger selection signal. Across long-video benchmarks, our method achieves $3.8{\times}$--$4.2{\times}$ prefilling speedups at a 25% token budget while maintaining near-baseline accuracy and improving with light finetuning. Hybrid models benefit from aggressive reduction, improving both efficiency and accuracy, whereas Transformers exhibit the standard trade-off. Our method also outperforms prior baselines on the same hybrid backbone and combines effectively with visual redundancy reduction methods.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. CodecSight: Leveraging Video Codec Signals for Efficient Streaming VLM Inference

    cs.DC 2026-04 unverdicted novelty 6.0

    CodecSight reuses video codec signals for online patch pruning before the vision transformer and selective KV-cache refresh in the LLM, delivering up to 3x higher throughput and 87% lower GPU compute than prior baseli...

  2. cuRAMSES: Scalable AMR Optimizations for Large-Scale Cosmological Simulations

    astro-ph.GA 2026-04 conditional novelty 5.0

    cuRAMSES replaces Hilbert-curve domain decomposition with recursive k-section partitioning and adds Morton-key hashing plus spatial binning to cut communication volume and accelerate feedback routines by up to 260x wh...

  3. cuRAMSES: Scalable AMR Optimizations for Large-Scale Cosmological Simulations

    astro-ph.GA 2026-04 conditional novelty 5.0

    Recursive k-section domain decomposition, Morton-key hashing, and GPU dispatch cut communication and memory bottlenecks in RAMSES while preserving conservation to ~0.5%.

Reference graph

Works this paper leans on

48 extracted references · 16 linked inside Pith · cited by 2 Pith papers

  1. [1]

    An image is worth 1/2 tokens after layer 2: Plug-and- play inference acceleration for large vision-language models, 2024

    Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and- play inference acceleration for large vision-language models, 2024

    2024

  2. [2]

    Sparsevlm: Visual token sparsifi- cation for efficient vision-language model inference, 2025

    Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, and Shanghang Zhang. Sparsevlm: Visual token sparsifi- cation for efficient vision-language model inference, 2025

    2025

  3. [3]

    Pyramiddrop: Accel- erating your large vision-language models via pyra- mid visual redundancy reduction.arXiv preprint arXiv:2410.17247, 2024

    Long Xing, Qidong Huang, Xiaoyi Dong, Jiajie Lu, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, Jiaqi Wang, Feng Wu, et al. Pyramiddrop: Accel- erating your large vision-language models via pyra- mid visual redundancy reduction.arXiv preprint arXiv:2410.17247, 2024

    Pith/arXiv arXiv 2024

  4. [4]

    Beyond text-visual attention: Exploiting visual cues for effective token pruning in vlms

    Qizhe Zhang, Aosong Cheng, Ming Lu, Renrui Zhang, Zhiyong Zhuo, Jiajun Cao, Shaobo Guo, Qi She, and Shanghang Zhang. Beyond text-visual attention: Exploiting visual cues for effective token pruning in vlms. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 20857–20867, 2025

    2025

  5. [5]

    Metok: Multi-stage event-based token compression for efficient long video understanding.arXiv preprint arXiv:2506.02850, 2025

    Mengyue Wang, Shuo Chen, Kristian Kersting, Volker Tresp, and Yunpu Ma. Metok: Multi-stage event-based token compression for efficient long video understanding.arXiv preprint arXiv:2506.02850, 2025

    arXiv 2025

  6. [6]

    Flexselect: Flexible token selection for efficient long video understanding.arXiv preprint arXiv:2506.00993, 2025

    Yunzhu Zhang, Yu Lu, Tianyi Wang, Fengyun Rao, Yi Yang, and Linchao Zhu. Flexselect: Flexible token selection for efficient long video understanding.arXiv preprint arXiv:2506.00993, 2025

    Pith/arXiv arXiv 2025

  7. [7]

    Multi-granular spatio-temporal token merging for training-free ac- celeration of video llms

    Jeongseok Hyun, Sukjun Hwang, Su Ho Han, Taeoh Kim, Inwoong Lee, Dongyoon Wee, Joon-Young Lee, Seon Joo Kim, and Minho Shim. Multi-granular spatio-temporal token merging for training-free ac- celeration of video llms. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 23990–24000, 2025

    2025

  8. [8]

    Dyntok: Dy- namic compression of visual tokens for efficient and effective video understanding.arXiv preprint arXiv:2506.03990, 2025

    Hongzhi Zhang, Jingyuan Zhang, Xingguang Ji, Qi Wang, and Fuzheng Zhang. Dyntok: Dy- namic compression of visual tokens for efficient and effective video understanding.arXiv preprint arXiv:2506.03990, 2025

    Pith/arXiv arXiv 2025

  9. [9]

    Nvidia nemotron nano 2: An accurate and efficient hybrid mamba-transformer reasoning model, 2025

    NVIDIA, :, Aarti Basant, Abhijit Khairnar, Abhijit Paithankar, Abhinav Khattar, Adithya Renduchin- tala, Aditya Malte, Akhiad Bercovich, Akshay Haz- are, Alejandra Rico, Aleksander Ficek, Alex Kon- dratenko, Alex Shaposhnikov, Alexander Bukharin, Ali Taghibakhshi, et al. Nvidia nemotron nano 2: An accurate and efficient hybrid mamba-transformer reasoning ...

    2025

  10. [10]

    Nvidia nemotron nano v2 vl, 2025

    NVIDIA, :, Amala Sanjay Deshmukh, Kateryna Chu- machenko, Tuomas Rintamaki, Matthieu Le, Tyler Poon, Danial Mohseni Taheri, Ilia Karmanov, Guilin Liu, Jarno Seppanen, Guo Chen, Karan Sapra, Zhid- ing Yu, Adi Renduchintala, Charles Wang, Peter Jin, Arushi Goel, Mike Ranzinger, Lukas Voegtle, Philipp Fischer, et al. Nvidia nemotron nano v2 vl, 2025

    2025

  11. [11]

    Mamba: Linear-time se- quence modeling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time se- quence modeling with selective state spaces. InFirst conference on language modeling, 2024

    2024

  12. [12]

    Video- mme: The first-ever comprehensive evaluation bench- mark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video- mme: The first-ever comprehensive evaluation bench- mark of multi-modal llms in video analysis. InPro- ceedings of the Computer Vision and Pattern Recog- nition Conference, pages 24108–24118, 2025

    2025

  13. [13]

    Longvideobench: A benchmark for long-context interleaved video-language understanding.Advances in Neural Information Processing Systems, 37:28828– 28857, 2024

    Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding.Advances in Neural Information Processing Systems, 37:28828– 28857, 2024. 10 Stateful Token Reduction for Long-Video Hybrid VLMs

    2024

  14. [14]

    Lvbench: An extreme long videounderstandingbenchmark

    Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Ming Ding, Xiaotao Gu, Shiyu Huang, Bin Xu, et al. Lvbench: An extreme long videounderstandingbenchmark. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 22958–22967, 2025

    2025

  15. [15]

    Transformers are ssms: Generalized models and efficient algorithms through structured state space duality, 2024

    Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality, 2024

    2024

  16. [16]

    Transformers are rnns: Fast autoregressive transformers with linear atten- tion

    Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pap- pas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear atten- tion. InInternational conference on machine learning, pages 5156–5165. PMLR, 2020

    2020

  17. [17]

    Understanding transformer from the perspective of associativememory.arXiv preprint arXiv:2505.19488, 2025

    Shu Zhong, Mingyu Xu, Tenglong Ao, and Guang Shi. Understanding transformer from the perspective of associativememory.arXiv preprint arXiv:2505.19488, 2025

    Pith/arXiv arXiv 2025

  18. [18]

    A new measure of rank correla- tion.Biometrika, 30(1-2):81–93, 1938

    Maurice G Kendall. A new measure of rank correla- tion.Biometrika, 30(1-2):81–93, 1938

    1938

  19. [19]

    Why stacking sliding windows can’t see very far.https://guangxuanx.com/blog/ stacking-swa.html, 2025

    Guangxuan Xiao. Why stacking sliding windows can’t see very far.https://guangxuanx.com/blog/ stacking-swa.html, 2025

    2025

  20. [20]

    Sparsevila: Decou- pling visual sparsity for efficient vlm inference

    Samir Khaki, Junxian Guo, Jiaming Tang, Shang Yang, Yukang Chen, Konstantinos N Plataniotis, Yao Lu, Song Han, and Zhijian Liu. Sparsevila: Decou- pling visual sparsity for efficient vlm inference. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pages 23784–23794, 2025

    2025

  21. [21]

    Acceler- ating multimodal large language models by searching optimal vision token reduction

    Shiyu Zhao, Zhenting Wang, Felix Juefei-Xu, Xide Xia, Miao Liu, Xiaofang Wang, Mingfu Liang, Ning Zhang, Dimitris N Metaxas, and Licheng Yu. Acceler- ating multimodal large language models by searching optimal vision token reduction. InProceedings of the Computer Vision and Pattern Recognition Confer- ence, pages 29869–29879, 2025

    2025

  22. [22]

    Siglip 2: Multilingual vision-language encoders with improved semantic un- derstanding, localization, and dense features.arXiv preprint arXiv:2502.14786, 2025

    Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language encoders with improved semantic un- derstanding, localization, and dense features.arXiv preprint arXiv:2502.14786, 2025

    Pith/arXiv arXiv 2025

  23. [23]

    Qwen3-vl technical report, 2025

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, ChangGao, ChunjiangGe, WenbinGe, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, et al. Qwen3-vl technical report, 2025

    2025

  24. [24]

    Nemotron-h: A family of accurate and efficient hybrid mamba- transformer models, 2025

    NVIDIA, :, Aaron Blakeman, Aarti Basant, Abhinav Khattar, Adithya Renduchintala, Akhiad Bercovich, Aleksander Ficek, Alexis Bjorlin, Ali Taghibakhshi, Amala Sanjay Deshmukh, Ameya Sunil Mahabalesh- warkar, Andrew Tao, Anna Shors, Ashwath Aithal, Ashwin Poojary, Ayush Dattagupta, Balaram Bud- dharaju, Bobby Chen, Boris Ginsburg, Boxin Wang, Brandon Norick,...

    2025

  25. [25]

    Storm: Token-efficient long video understanding for multimodal llms

    Jindong Jiang, Xiuyu Li, Zhijian Liu, Muyang Li, Guo Chen, Zhiqi Li, De-An Huang, Guilin Liu, Zhid- ing Yu, Kurt Keutzer, et al. Storm: Token-efficient long video understanding for multimodal llms. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pages 5830–5841, 2025

    2025

  26. [26]

    Video in- struction tuning with synthetic data.arXiv preprint arXiv:2410.02713, 2024

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Ze- jun Ma, Ziwei Liu, and Chunyuan Li. Video in- struction tuning with synthetic data.arXiv preprint arXiv:2410.02713, 2024

    Pith/arXiv arXiv 2024

  27. [27]

    Ea- gle 2.5: Boosting long-context post-training for frontier vision-language models.arXiv preprint arXiv:2504.15271, 2025

    Guo Chen, Zhiqi Li, Shihao Wang, Jindong Jiang, Yicheng Liu, Lidong Lu, De-An Huang, Wonmin Byeon, Matthieu Le, Tuomas Rintamaki, et al. Ea- gle 2.5: Boosting long-context post-training for frontier vision-language models.arXiv preprint arXiv:2504.15271, 2025

    arXiv 2025

  28. [28]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Rad- ford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    Pith/arXiv arXiv 2024

  29. [29]

    Gemini: a family of highly capable mul- timodal models.arXiv preprint arXiv:2312.11805, 2023

    GeminiTeam, RohanAnil, SebastianBorgeaud, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Milli- can, et al. Gemini: a family of highly capable mul- timodal models.arXiv preprint arXiv:2312.11805, 2023

    Pith/arXiv arXiv 2023

  30. [30]

    Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    Pith/arXiv arXiv 2024

  31. [31]

    Qwen2-vl: Enhancing vision- language model’s perception of the world at any res- olution.arXiv preprint arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhi- hao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision- language model’s perception of the world at any res- olution.arXiv preprint arXiv:2409.12191, 2024

    Pith/arXiv arXiv 2024

  32. [32]

    Nvila: Efficient frontier visual language models

    Zhijian Liu, Ligeng Zhu, Baifeng Shi, Zhuoyang Zhang, Yuming Lou, Shang Yang, Haocheng Xi, Shiyi Cao, Yuxian Gu, Dacheng Li, et al. Nvila: Efficient frontier visual language models. InProceedings of the Computer Vision and Pattern Recognition Con- ference, pages 4122–4134, 2025

    2025

  33. [33]

    Apollo: An exploration of video understanding 11 Stateful Token Reduction for Long-Video Hybrid VLMs in large multimodal models

    Orr Zohar, Xiaohan Wang, Yann Dubois, Nikhil Mehta, Tong Xiao, Philippe Hansen-Estruch, Licheng Yu, Xiaofang Wang, Felix Juefei-Xu, Ning Zhang, et al. Apollo: An exploration of video understanding 11 Stateful Token Reduction for Long-Video Hybrid VLMs in large multimodal models. InProceedings of the Computer Vision and Pattern Recognition Confer- ence, pa...

    2025

  34. [34]

    Vide- ollama 3: Frontier multimodal foundation models for image and video understanding.arXiv preprint arXiv:2501.13106, 2025

    Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, et al. Vide- ollama 3: Frontier multimodal foundation models for image and video understanding.arXiv preprint arXiv:2501.13106, 2025

    Pith/arXiv arXiv 2025

  35. [35]

    Oryx mllm: On- demand spatial-temporal understanding at arbitrary resolution.arXiv preprint arXiv:2409.12961, 2024

    Zuyan Liu, Yuhao Dong, Ziwei Liu, Winston Hu, Jiwen Lu, and Yongming Rao. Oryx mllm: On- demand spatial-temporal understanding at arbitrary resolution.arXiv preprint arXiv:2409.12961, 2024

    Pith/arXiv arXiv 2024

  36. [36]

    M-llm based video frame selection for efficient video understanding

    Kai Hu, Feng Gao, Xiaohan Nie, Peng Zhou, Son Tran, Tal Neiman, Lingyun Wang, Mubarak Shah, Raffay Hamid, Bing Yin, et al. M-llm based video frame selection for efficient video understanding. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 13702–13712, 2025

    2025

  37. [37]

    Longvu: Spatiotemporal adaptive compres- sion for long video-language understanding.arXiv preprint arXiv:2410.17434, 2024

    Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishnan Varadarajan, Florian Bor- des, et al. Longvu: Spatiotemporal adaptive compres- sion for long video-language understanding.arXiv preprint arXiv:2410.17434, 2024

    Pith/arXiv arXiv 2024

  38. [38]

    Slowfast-llava-1.5: A family of token-efficient video large language models for long-form video understanding.arXiv preprint arXiv:2503.18943, 2025

    Mingze Xu, Mingfei Gao, Shiyu Li, Jiasen Lu, Zhe Gan, Zhengfeng Lai, Meng Cao, Kai Kang, Yinfei Yang, and Afshin Dehghan. Slowfast-llava-1.5: A family of token-efficient video large language models for long-form video understanding.arXiv preprint arXiv:2503.18943, 2025

    Pith/arXiv arXiv 2025

  39. [39]

    Scaling video-language models to 10k frames via hierarchical differential distillation.arXiv preprint arXiv:2504.02438, 2025

    Chuanqi Cheng, Jian Guan, Wei Wu, and Rui Yan. Scaling video-language models to 10k frames via hierarchical differential distillation.arXiv preprint arXiv:2504.02438, 2025

    Pith/arXiv arXiv 2025

  40. [40]

    Dycoke: Dynamic compression of tokens for fast video large language models

    Keda Tao, Can Qin, Haoxuan You, Yang Sui, and Huan Wang. Dycoke: Dynamic compression of tokens for fast video large language models. InProceedings of the Computer Vision and Pattern Recognition Con- ference, pages 18992–19001, 2025

    2025

  41. [41]

    Visionselector: End-to-end learnable visual token compression for efficient multimodal llms.arXiv preprint arXiv:2510.16598, 2025

    Jiaying Zhu, Yurui Zhu, Xin Lu, Wenrui Yan, Dong Li, Kunlin Liu, Xueyang Fu, and Zheng-Jun Zha. Visionselector: End-to-end learnable visual token compression for efficient multimodal llms.arXiv preprint arXiv:2510.16598, 2025

    arXiv 2025

  42. [42]

    Visa: Group-wise visual token selection and aggregation via graph summarization for efficient mllms infer- ence

    Pengfei Jiang, Hanjun Li, Linglan Zhao, Fei Chao, Ke Yan, Shouhong Ding, and Rongrong Ji. Visa: Group-wise visual token selection and aggregation via graph summarization for efficient mllms infer- ence. InProceedings of the 33rd ACM International Conference on Multimedia, pages 11130–11139, 2025

    2025

  43. [43]

    Laco: Efficient layer-wise com- pression of visual tokens for multimodal large lan- guage models.arXiv preprint arXiv:2507.02279, 2025

    Juntao Liu, Liqiang Niu, Wenchao Chen, Jie Zhou, and Fandong Meng. Laco: Efficient layer-wise com- pression of visual tokens for multimodal large lan- guage models.arXiv preprint arXiv:2507.02279, 2025

    Pith/arXiv arXiv 2025

  44. [44]

    Visionzip: Longer is better but not necessary in vision language models

    Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. Visionzip: Longer is better but not necessary in vision language models. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 19792–19802, 2025

    2025

  45. [45]

    Parallelizing linear transformers with the delta rule over sequence length.Advances in neural information processing systems, 37:115491– 115522, 2024

    Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing linear transformers with the delta rule over sequence length.Advances in neural information processing systems, 37:115491– 115522, 2024. 12 Stateful Token Reduction for Long-Video Hybrid VLMs A. Appendix A.1. The Implicit Attention Mechanism of Mamba We derive the implicit atten...

    2024

  46. [46]

    Content alignment: 𝑏⊤ 𝑗 𝑐𝑡 measures the rele- vance between the input at position𝑗 (encoded by 𝑏𝑗) and the query at position𝑡 (encoded by 𝑐𝑡)

  47. [47]

    Input gating:Δ 𝑗 controls how strongly position 𝑗writes to the state

  48. [48]

    Attn Only

    Temporal decay:∏︀𝑡 𝑢=𝑗+1 ¯𝐴𝑢 exponentially de- cays contributions from distant positions. Equation (7) suggests an intuitive role for𝑏 and 𝑐, we next formalize this by mapping the recurrence directly to the key-query formulation of Linear At- tention. This connection justifies our use of these projections for token selection.. A.2. Token Selection via Mam...

This paper was first reviewed by deepseek-v4-flash on August 2, 2026.