Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

MMInference: Accelerating Pre-filling for Long-Context VLMs via Modality-Aware Permutation Sparse Attention

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read MMInference claims that modality-aware permutation sparse attention accelerates long-context VLM pre-filling by up to 8.3x at 1M tokens while matching full-attention accuracy on the benchmarks tested.

desk verdict Credible training-free VLM prefill accelerator; permutation-based grid/modality handling is a real extension of MInference, but the 1M accuracy claim needs scoping and the EgoSchema calibration leak should be fixed. read the letter →

arxiv 2504.16083 v2 pith:BD4CA7UC submitted 2025-04-22 cs.CV cs.LG

Yucheng Li , Huiqiang Jiang , Chengruidong Zhang , Qianhui Wu , Xufang Luo , Surin Ahn , Amir H. Abdi , Dongsheng Li
show 3 more authors
Jianfeng Gao Yuqing Yang Lili Qiu
This is my paper · ORCID
classification cs.CVcs.LG
keywords sparseattentionlong-contextvisionlanguagemodelspre-fillaccelerationpermutation-basedmodalityboundariesvideounderstandingdynamicindexingneedleinahaystack
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the slow pre-filling stage of long-context vision-language models, the phase before the model starts generating an answer, can be made roughly an order of magnitude faster with no retraining and no measurable accuracy loss on the benchmarks tested. The underlying observation is that video-driven attention in these models is not unstructured: certain heads show an evenly spaced grid of attended positions, and attention inside one modality behaves differently from attention that crosses a text-video boundary. MMInference exploits this by permuting queries, keys, and values so that scattered grid lines and same-modality regions become contiguous blocks, converting sparse memory access into dense tensor-core computation, while a cheap online estimator builds the sparse index from the last 64 queries of each modality segment. The paper reports up to 8.3x end-to-end prefill speedup at 1M tokens on a single A100 (and 12x at the kernel level) over FlashAttention-2, with video QA, captioning, and needle-in-a-haystack retrieval scores matching full attention.

What carries the argument

The load-bearing machinery is the permutation of the $Q$, $K$, and $V$ tensors guided by a dynamic sparse index. It works through three pattern families: (1) the Grid pattern for video-heavy heads, whose stride and phase are found online by scanning an approximate attention matrix built from the last 64 queries, after which the tensors are permuted so that grid lines align into contiguous blocks (Algorithm 1); (2) the Q-Boundary and 2D-Boundary heads, where row-wise or full modality-grouping permutation isolates intra-modality regions, so a sparse index built once in one visual region extrapolates across text-interrupted visual segments (Algorithms 2 and 3); and (3) the Modality-Aware Sparse Attention Search Algorithm (Algorithm 4), which assigns each head its pattern offline under a kernel-measured FLOPs budget, calibrated on a single 25K-token video sample. Computation is carried out by block-sparse FlashAttention-style kernels (built on FlashAttention-2 and the PIT dynamic sparse compilation approach) that load only the permuted blocks, keeping tensor-core compute dense.

What would settle it

Construct a long mixed-modality input in which the final 64 queries of a visual segment attend to a different key region than the earlier tokens of that segment (a scene cut or topic shift placed exactly at the segment end), and measure both the attention recall of the online-estimated masks and the needle-retrieval score. If attention recall falls far below the roughly 95% threshold the paper's own analysis treats as adequate, or retrieval drops from the reported 97.7% on V-NIAH, the last-64-queries extrapolation premise is false.

Watch

Extended reading notes

Core claim

The paper's central claim is that attention in long-context VLMs is sparse in a way that is both modality-structured and permutation-friendly, so that full-attention accuracy can be recovered by computing only a fraction of the attention matrix. It identifies three pattern families: the Grid pattern, in which video tokens attend along evenly spaced horizontal and vertical lines whose stride and phase are searched online from an approximate attention matrix; the Q-Boundary and 2D-Boundary patterns, where modality boundaries split the query dimension or both query and key dimensions; and the A-shape and Vertical-Slash patterns carried over from text-only LLMs. The method's namesake move is to permute the $Q$, $K$, and $V$ tensors so these patterns become consecutive, with row-wise permutation grouping same-modality queries, column-wise permutation aligning key strides, and 2D permutation isolating each modality pair into its own block, turning scattered sparse loads into dense tensor-core computation. Permutation is realized as index-based sparse loading inside FlashAttention-style kernels, so no weights change and no fine-tuning is needed. Offline, a modality-aware search (Algorithm 4) assigns each head its optimal pattern set under a kernel-measured FLOPs budget; online, the sparse index is estimated from the last 64 queries of each modality segment and the attention is computed with block-sparse kernels. On four VLMs and inputs up to 1.1M tokens, the paper reports up to 8.3x prefill speedup over FlashAttention-2 with benchmark scores matching full attention, including 97.7% versus 98.3% retrieval recall on V-NIAH and 91.3% versus 90.9% on its new mixed-modality NIAH test.

Load-bearing premise

The per-head sparse patterns chosen offline from a single 25K-token calibration sample, plus the online sparse indices estimated from the last 64 queries of each modality segment, must transfer to arbitrary inputs, contexts up to 1M tokens, and benchmarks the calibration never saw; if the transfer fails, the masks miss true attention mass and accuracy degrades.

Editorial extensions

If this is right

  • At 1M-token contexts the pre-fill stage runs up to 8.3x faster end-to-end and up to 12x faster at the kernel level, turning multi-minute waits on one A100 into a small fraction of the time.
  • Accuracy holds: average scores on six video understanding benchmarks stay within roughly half a point of full attention, and long-video retrieval recall remains at 97.7% versus 98.3% for full attention at 6K frames.
  • The method is training-free and model-agnostic, applying to LongVILA, Llava-Video, VideoChat-Flash, and Qwen2.5-VL, and it stacks with visual-token compression instead of competing with it.
  • Static sparse patterns degrade sharply on mixed-modality inputs, whereas MMInference keeps retrieval near full-attention levels up to roughly 1.1M tokens, evidence that handling modality boundaries, not sparsity alone, is what preserves accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The permutation principle should extend beyond prefill: the paper itself notes that 2D/3D sliding-window attention in video diffusion transformers can be reshaped into dense tensor-core tiles by an analogous permutation, which would bring the same speedup to generative video models rather than only to the prefill stage of VLMs.
  • A testable extension is per-modality index amortization: since a sparse index from one visual region extrapolates across text-separated visual segments, building the index once per modality segment and reusing it could remove even the small cost of the last-64-queries estimator at extreme context lengths.
  • The reported robustness rests on an unusually small calibration set, one EgoSchema sample of at most 25K tokens, so re-running the offline pattern search across diverse video/text mixtures (different frame rates, scene cuts, interleaving ratios) would map where per-head pattern assignments stay stable and where they break.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces MMInference, a training-free dynamic sparse attention method for accelerating the pre-filling stage of long-context vision-language models (VLMs). The authors identify a Grid attention pattern in video inputs, characterize modality-boundary patterns (Q-Boundary, 2D-Boundary, etc.), and propose permutation-based transformations plus an offline per-head pattern search (Algorithm 4) to construct dynamic sparse masks. They provide optimized Triton/FlashAttention-based kernels. Experiments on Llava-Video, LongVILA, Qwen2.5-VL, and VideoChat-Flash report accuracy close to full attention on video QA and captioning benchmarks, with speedups up to 8.3x at roughly 1M tokens on V-NIAH and MM-NIAH retrieval tasks.

Significance. If the accuracy-retention claim holds at the claimed scale, MMInference is a noteworthy systems contribution: it is training-free, model-agnostic, and demonstrates that modality-aware permutation can convert irregular multimodal sparse patterns into efficient dense-block computations. The latency measurements are concrete and reproducible in spirit, and the paper ships code. The kernel-level analyses and the Grid-pattern observation are useful for follow-up work on efficient long-video inference. However, the breadth of the accuracy claim currently exceeds the evidence: the calibration-to-evaluation overlap on EgoSchema and the restriction of 1M-token validation to synthetic needle-in-a-haystack tasks leave the 'maintaining accuracy' clause less secure than the abstract suggests.

major comments (4)
  1. [Appendix C.5, Table 1] The per-head pattern assignment is calibrated on a single EgoSchema sample of at most 25K tokens, and EgoSchema is also a headline benchmark in Table 1. This creates a selection-to-evaluation overlap: the search objective (attention recall on this one sample) directly influences the choice of sparse patterns that are then reported as improved or maintained accuracy on EgoSchema. The claim in Appendix C.5 that this calibration 'exhibits strong generalization and stability' needs quantitative support, for example by re-running the offline search on a sample from a different benchmark and reporting both configurations, or by explicitly treating EgoSchema as a development set and not a held-out evaluation.
  2. [Abstract, §4.3, §4.4, Fig. 5] The central claim of 'maintaining accuracy' at 1M tokens is supported only by V-NIAH and MM-NIAH, which test a single inserted needle at various depths in long video/text haystacks. These tasks do not exercise sustained reasoning, detailed captioning, or multi-turn mixed-modality interactions at long context, so the 1M-token evidence is narrower than the abstract's unqualified 'maintaining accuracy.' I recommend either adding long-context accuracy measurements on a broader set of tasks (for example, long-form video QA with explicit long-range reasoning or document-scale mixed-modal tasks) or explicitly qualifying the accuracy claim to retrieval-style tasks at the 1M scale.
  3. [§5 (Related Work), §4.1 (Baselines)] xAttention (Xu et al., 2025b) is cited as a recent dynamic sparse attention method for long-context VLMs, but it is not included in any comparison table or latency experiment. Since the paper's contribution is precisely a dynamic sparse attention scheme for VLMs, the absence of this closest comparable baseline leaves the claimed advantage over prior dynamic sparse VLM methods unevidenced. A direct comparison on the same benchmarks and latency setup should be added, or the exclusion should be explicitly justified.
  4. [Algorithm 4 and §4.6 (Fig. 8)] The offline search uses attention recall on one calibration sample as the selection objective, but the paper does not demonstrate that this recall-based proxy transfers to downstream task accuracy across diverse inputs, context lengths, or benchmarks. Fig. 8 tests index transfer within the same modality but not the transfer of the per-head pattern configuration selected on one EgoSchema sample to the full set of evaluation inputs. A sensitivity analysis showing pattern stability across multiple calibration samples and across context lengths (e.g., 20K, 66K, 300K, 1M tokens) would strengthen the load-bearing transfer assumption.
minor comments (5)
  1. [Appendix E.1 and E.2] There are typos: 'attenton' should be 'attention' in the sentences introducing Figs. 13 and 14.
  2. [Appendix C.5] 'egoschema' is written in lowercase in the text; use consistent capitalization for the benchmark name.
  3. [Algorithm 5] In Grid-Shape Flash Attention, the pseudocode after the softmax-like rescaling is abbreviated (the exp and normalization steps are compressed), which makes it hard to verify correctness against the FlashAttention-2 equations. Please expand the pseudocode so the rescaling and softmax updates are explicit.
  4. [Table 2] The VideoChat-Flash row reports performance without a full-attention baseline in the same table; adding Full Attention and MInference for this model would make the comparison easier to interpret.
  5. [Abstract and §3.3] The phrase 'offline search the optimal sparse patterns' is a grammatical artifact; consider rephrasing to 'offline search for the optimal sparse patterns.'

Circularity Check

1 steps flagged · score 4.0 of 10

One partial overlap: per-head patterns are calibrated on an EgoSchema sample while EgoSchema test is reported; the core approximation and speedup claims are otherwise independently benchmarked.

  1. fitted input called prediction [Appendix C.5 (Search Space); Algorithm 4; Table 1]
    "Additionally, we use only one sample as our calibration set from the egoschema task with no more than 25K tokens, which exhibits strong generalization and stability across different lengths and domains. The search time is approximately 15 minutes on a single A100. This pattern search is individually conducted for each model: Llava-Video-7B, LongVila-256Frame, and LongVila-1M."

    Algorithm 4 fits each attention head's sparse pattern by selecting the configuration that maximizes attention recall on a reference example, and the reference example is drawn from the EgoSchema task. Table 1 then reports EgoSchema test accuracy as evidence that MMInference maintains accuracy. The EgoSchema row is therefore not a fully held-out test of the searched configuration: the per-head pattern assignment was optimized on an input from the same benchmark and domain, so part of the reported score reflects the search objective's in-sample fit.

full rationale

MMInference is an approximate-attention method rather than a derived law, so most of its evaluation is honest benchmarking against full attention and sparse baselines. The speedup claim is supported by kernel-level and end-to-end latency measurements (Figs. 7 and 16), including the 8.3x figure at 1M tokens, and these do not depend on any fitted pattern. The accuracy-retention claim is supported by independent benchmarks that were not used in the offline pattern search: ActNet-QA, Next-QA, VideoMME, V-NIAH, and MM-NIAH, the latter two including 1M-token retrieval tests. The only circular burden is the calibration-to-evaluation overlap in Appendix C.5: the per-head pattern assignment is selected on one EgoSchema sample, and Table 1 then reports EgoSchema test as a headline accuracy result. This is a partial fitted-input-called-prediction issue, not a derivation-level circularity, and self-citations to MInference and PIT are implementation-level rather than load-bearing. Missing xAttention comparison is an evidence gap, not a circularity. Score 4.

Assumptions & free parameters 7 free parameters · 6 assumptions · 0 invented entities

The central contribution rests on several transfer assumptions: the grid structure of video attention, the modality boundary taxonomy, last-query index extrapolation, and per-head pattern transfer from a single calibration sample. These are empirical assumptions, disclosed only partially, and they are not machine-checked or proven. They create moderate risk because the online index is estimated from a 64-token tail and the offline pattern config comes from one sample.

free parameters (7)
  • per-head pattern assignment = varies: Grid / A-shape / Vertical-Slash / Q-Boundary / 2D-Boundary per head per model
    Selected offline in Algorithm 4 using attention recall on one EgoSchema calibration sample (Appendix C.5).
  • lastq = 64
    Length of query tail used to estimate Vertical-Slash and Grid sparse indices online (Sec 3.1, Algorithm 1, Appendix C.3).
  • Grid stride and phase = estimated online; search over frame stride and stride with max_stride=1024
    Grid pattern parameters detected from the last 64 queries; search space in Table 4 includes flags for hline, vline, and slash.
  • Vertical-Slash sizes = vertical in {1000, 2000, 3500}; slash in {200, 1024, 2048, 2500, 3096, 4096}
    Searched offline per head in Algorithm 4; listed in Table 4 and Appendix C.1.
  • A-shape local window = sink=128; local in {1024, 2048, 4096}
    Static sink tokens fixed; local window size chosen in offline search (Appendix C.1, Table 4).
  • target FLOPs budget = 1k global + 4k local tokens
    Kernel-aware search space fixed to this budget, inherited from MInference A-shape configuration (Appendix C.5).
  • offline calibration sample = one EgoSchema sample, at most 25K tokens
    Single sample used to select per-head patterns; chosen from EgoSchema, which is also an evaluation benchmark (Appendix C.5).
assumptions (6)
  • standard math Permuting query, key, and value indices in the same way, and permuting the attention mask accordingly, preserves the attention output.
    Algorithms 1 to 3 rely on row and column permutations to turn scattered sparse patterns into dense blocks; this invariance is standard linear algebra.
  • domain assumption The sparse pattern observed in the last 64 query tokens of a modality extrapolates to all query positions in that modality.
    Algorithm 1 estimates grid stride, phase, and Vertical-Slash lines from Q[-64:]; Fig 8 gives partial support, but no thorough failure analysis across heads and contexts is provided.
  • domain assumption Per-head pattern assignments found offline on one 25K-token EgoSchema sample transfer to other inputs, lengths, and benchmarks.
    Online inference uses fixed per-head patterns; transfer beyond calibration is assumed and only implicitly tested through benchmark results (Appendix C.5).
  • domain assumption Video attention heads exhibit a repeatable grid structure with evenly spaced horizontal and vertical lines.
    Sec 2.2 and Fig 3a show the pattern on LongVILA; the method's Grid head depends on this structural prior.
  • domain assumption Modality boundaries are fully captured by the No-Boundary, K-Boundary, Q-Boundary, and 2D-Boundary taxonomy and remain stable for a given model.
    Sec 2.3 defines four boundary types; Algorithms 2 and 3 dispatch on these types without a fallback for other boundary shapes.
  • ad hoc to paper Attention recall on one calibration sample is a sufficient proxy for downstream task accuracy.
    Algorithm 4 uses attention recall as the search objective; the paper does not establish a quantitative correlation between recall and the evaluated benchmarks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MMInference: Accelerating Pre-filling for Long-Context VLMs via Modality-Aware Permutation Sparse Attention." pith.science (2026). https://pith.science/paper/BD4CA7UC

@misc{pith2026250416083,
  author       = {Pith},
  title        = {Pith review of: MMInference: Accelerating Pre-filling for Long-Context VLMs via Modality-Aware Permutation Sparse Attention},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BD4CA7UC}},
  note         = {Machine review of arXiv:2504.16083}
}
read the original abstract

The integration of long-context capabilities with visual understanding unlocks unprecedented potential for Vision Language Models (VLMs). However, the quadratic attention complexity during the pre-filling phase remains a significant obstacle to real-world deployment. To overcome this limitation, we introduce MMInference (Multimodality Million tokens Inference), a dynamic sparse attention method that accelerates the prefilling stage for long-context multi-modal inputs. First, our analysis reveals that the temporal and spatial locality of video input leads to a unique sparse pattern, the Grid pattern. Simultaneously, VLMs exhibit markedly different sparse distributions across different modalities. We introduce a permutation-based method to leverage the unique Grid pattern and handle modality boundary issues. By offline search the optimal sparse patterns for each head, MMInference constructs the sparse distribution dynamically based on the input. We also provide optimized GPU kernels for efficient sparse computations. Notably, MMInference integrates seamlessly into existing VLM pipelines without any model modifications or fine-tuning. Experiments on multi-modal benchmarks-including Video QA, Captioning, VisionNIAH, and Mixed-Modality NIAH-with state-of-the-art long-context VLMs (LongVila, LlavaVideo, VideoChat-Flash, Qwen2.5-VL) show that MMInference accelerates the pre-filling stage by up to 8.3x at 1M tokens while maintaining accuracy. Our code is available at https://aka.ms/MMInference.

Figures

Figures reproduced from arXiv: 2504.16083 by the authors.

Figure 1
Figure 1. Dynamic sparse attention pipelines leverage sparse [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. (a) Latency breakdown of the pre-filling stage, with 256 tokens per frame. (b) How much element in attention needs to be [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Visualization of pre- vs. post-permutation sparsity attention patterns in VLMs. [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (15 more)
Figure 4
Figure 4. Figure 4: The framework of MMInference, encompassing both inter- and intra-modality sparse attention patterns. [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: V-NIAH (Zhang et al., 2024a) and MM-NIAH results using LongVila-Qwen2-7B-1M (Chen et al., 2025). Video (Zhang et al., 2024b) and LongVILA (Chen et al., 2025). We follow the MInference experimental setup, con￾figuring the corresponding search space while adopting op￾tim…
Figure 6
Figure 6. Figure 6: Transition of sparse patterns from textual context to visual context. (a) The vertical-slash pattern for all textual context. (b) Grid [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: End-to-End Latency [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: The sparse index does not effectively extrapolate from [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: Additional inter-modality sparse pattern. [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: The baselines of sparse attention in our experiments. [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: Permutation for the Grid Pattern. (a) Before permutation. (b) Row-wise permutation. (c) Column-wise permutation. [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]
Figure 12
Figure 12. Figure 12: Permutation for mix-modality context. (a) Mix-modality. (b) Q-wise permutation. (c) K-wise permutation. [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]
Figure 13
Figure 13. Figure 13: Video Needle In A Haystack (Zhang et al., 2024a) results using LongVila-Qwen2-7B-1M (Chen et al., 2025). E.4. VS Pattern vs. Grid Pattern Both VS pattern and Grid pattern achieve strong performance on video understanding and V-NIAH tasks. However, due to the grid atte…
Figure 14
Figure 14. Figure 14: Mixed-Modality Needle In A Haystack results using LongVila-Qwen2-7B-1M ( [PITH_FULL_IMAGE:figures/full_fig_p019_14.png]
Figure 15
Figure 15. Figure 15: Permutation-based implementation of 2D/3D sliding window attention ( [PITH_FULL_IMAGE:figures/full_fig_p019_15.png]
Figure 16
Figure 16. Figure 16: The latency breakdown of a single attention kernel for four sparse attention patterns and FlashAttention ( [PITH_FULL_IMAGE:figures/full_fig_p020_16.png]
Figure 17
Figure 17. Figure 17: Visualization of sparse attention patterns in Qwen2.5-VL with dynamic resolution input and VideoChat-Flash with visual [PITH_FULL_IMAGE:figures/full_fig_p022_17.png]
Figure 18
Figure 18. Figure 18: Visualization of sparse attention patterns in Qwen2.5-VL with dynamic resolution input and VideoChat-Flash with visual [PITH_FULL_IMAGE:figures/full_fig_p023_18.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity

    cs.CV 2025-02 conditional novelty 6.0 of 10

    Sparse VideoGen accelerates video diffusion transformers by about 2.3x with only small quality loss by classifying attention heads into spatial and temporal sparse patterns and using hardware-friendly layouts.

  2. Video-XL-2: Towards Very Long-Video Understanding Through Task-Aware KV Sparsification

    cs.CV 2025-06 conditional novelty 5.0 of 10

    Video-XL-2 cuts long-video inference cost with chunked pre-filling and query-gated dense-or-sparse KV reloading, reporting half the FLOPs and a third less decoding memory at roughly equal benchmark scores.

Reference graph

Works this paper leans on

61 extracted references · 20 canonical work pages · cited by 2 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Star attention: Efficient llm inference over long sequences

    Acharya, S., Jia, F., and Ginsburg, B. Star attention: Efficient llm inference over long sequences. ArXiv preprint, abs/2411.17116, 2024. URL https://arxiv.org/abs/2411.17116

    arXiv 2024

  3. [3]

    Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., et al. Qwen2. 5-vl technical report. ArXiv preprint, abs/2502.13923, 2025. URL https://arxiv.org/abs/2502.13923

    arXiv 2025

  4. [4]

    R., Finn, C., Kumar, A., and Levine, S

    Black, K., Nakamoto, M., Atreya, P., Walke, H. R., Finn, C., Kumar, A., and Levine, S. Zero-shot robotic manipulation with pre-trained image-editing diffusion models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=c0chJTSbci

    work page 2024

  5. [5]

    Token merging: Your vit but faster

    Bolya, D., Fu, C.-Y., Dai, X., Zhang, P., Feichtenhofer, C., and Hoffman, J. Token merging: Your vit but faster. ICLR, 2023

    work page 2023

  6. [6]

    Gr-2: A generative video-language-action model with web-scale knowledge for robot manipulation

    Cheang, C.-L., Chen, G., Jing, Y., Kong, T., Li, H., Li, Y., Liu, Y., Wu, H., Xu, J., Yang, Y., et al. Gr-2: A generative video-language-action model with web-scale knowledge for robot manipulation. ArXiv preprint, abs/2410.06158, 2024. URL https://arxiv.org/abs/2410.06158

    arXiv

  7. [7]

    An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models

    Chen, L., Zhao, H., Liu, T., Bai, S., Lin, J., Zhou, C., and Chang, B. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. ECCV, pp.\ 19--35, 2024. doi:10.1007/978-3-031-73004-7_2

    doi:10.1007/978-3-031-73004-7_2 2024

  8. [8]

    Long VILA : Scaling long-context visual language models for long videos

    Chen, Y., Xue, F., Li, D., Hu, Q., Zhu, L., Li, X., Fang, Y., Tang, H., Yang, S., Liu, Z., He, Y., Yin, H., Molchanov, P., Kautz, J., Fan, L., Zhu, Y., Lu, Y., and Han, S. Long VILA : Scaling long-context visual language models for long videos. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?i...

    work page 2025

Show all 61 references
  1. [9]

    Generating long sequences with sparse transformers

    Child, R., Gray, S., Radford, A., and Sutskever, I. Generating long sequences with sparse transformers. ArXiv preprint, abs/1904.10509, 2019. URL https://arxiv.org/abs/1904.10509

    1904 arXiv

  2. [10]

    Qwen2-audio technical report

    Chu, Y., Xu, J., Yang, Q., Wei, H., Wei, X., Guo, Z., Leng, Y., Lv, Y., He, J., Lin, J., et al. Qwen2-audio technical report. ArXiv preprint, abs/2407.10759, 2024. URL https://arxiv.org/abs/2407.10759

    2024 arXiv

  3. [11]

    Flashattention-2: Faster attention with better parallelism and work partitioning

    Dao, T. Flashattention-2: Faster attention with better parallelism and work partitioning. ICLR, 2024

    2024

  4. [12]

    Efficient-vdit: Efficient video diffusion transformers with attention tile

    Ding, H., Li, D., Su, R., Zhang, P., Deng, Z., Stoica, I., and Zhang, H. Efficient-vdit: Efficient video diffusion transformers with attention tile. arXiv preprint arXiv:2502.06155, 2025

    2025 arXiv

  5. [13]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

    Fu, C., Dai, Y., Luo, Y., Li, L., Ren, S., Zhang, R., Wang, Z., Zhou, C., Shen, Y., Zhang, M., et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. ArXiv preprint, abs/2405.21075, 2024. URL https://arxiv.org/abs/2405.21075

    2024 arXiv

  6. [14]

    Vista: A generalizable driving world model with high fidelity and versatile controllability

    Gao, S., Yang, J., Chen, L., Chitta, K., Qiu, Y., Geiger, A., Zhang, J., and Li, H. Vista: A generalizable driving world model with high fidelity and versatile controllability. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://o...

    2024

  7. [15]

    Neighborhood attention transformer

    Hassani, A., Walton, S., Li, J., Li, S., and Shi, H. Neighborhood attention transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 6185--6194, 2023

    2023

  8. [16]

    Zipvl: Efficient large vision-language models with dynamic token sparsification and kv cache compression

    He, Y., Chen, F., Liu, J., Shao, W., Zhou, H., Zhang, K., and Zhuang, B. Zipvl: Efficient large vision-language models with dynamic token sparsification and kv cache compression. ArXiv preprint, abs/2410.08584, 2024. URL https://arxiv.org/abs/2410.08584

    2024 arXiv

  9. [17]

    Gaia-1: A generative world model for autonomous driving

    Hu, A., Russell, L., Yeo, H., Murez, Z., Fedoseev, G., Kendall, A., Shotton, J., and Corrado, G. Gaia-1: A generative world model for autonomous driving. ArXiv preprint, abs/2309.17080, 2023. URL https://arxiv.org/abs/2309.17080

    2023 arXiv

  10. [18]

    Dialoggen: Multi-modal interactive dialogue system for multi-turn text-to-image generation

    Huang, M., Long, Y., Deng, X., Chu, R., Xiong, J., Liang, X., Cheng, H., Lu, Q., and Liu, W. Dialoggen: Multi-modal interactive dialogue system for multi-turn text-to-image generation. ArXiv preprint, abs/2403.08857, 2024. URL https://arxiv.org/abs/2403.08857

    2024 arXiv

  11. [19]

    Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D

    Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. ArXiv preprint, abs/2310.06825, 2023. URL https://arxiv.org/abs/2310.06825

    2023 arXiv

  12. [20]

    H., Li, D., Lin, C.-Y., Yang, Y., and Qiu, L

    Jiang, H., Li, Y., Zhang, C., Wu, Q., Luo, X., Ahn, S., Han, Z., Abdi, A. H., Li, D., Lin, C.-Y., Yang, Y., and Qiu, L. MI nference 1.0: Accelerating pre-filling for long-context LLM s via dynamic sparse attention. In The Thirty-eighth Annual Conference on Neural Information P...

    2024

  13. [21]

    Video detail caption, 2024

    Lab, L. Video detail caption, 2024. URL https://huggingface.co/datasets/lmms-lab/VideoDetailCaption

    2024

  14. [22]

    Flexprefill: A context-aware sparse attention mechanism for efficient long-sequence inference

    Lai, X., Lu, J., Luo, Y., Ma, Y., and Zhou, X. Flexprefill: A context-aware sparse attention mechanism for efficient long-sequence inference. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=OfjIlbelrT

    2025

  15. [23]

    Videochat-flash: Hierarchical compression for long-context video modeling

    Li, X., Wang, Y., Yu, J., Zeng, X., Zhu, Y., Huang, H., Gao, J., Li, K., He, Y., Wang, C., et al. Videochat-flash: Hierarchical compression for long-context video modeling. ArXiv preprint, abs/2501.00574, 2025. URL https://arxiv.org/abs/2501.00574

    2025 arXiv

  16. [24]

    Snap KV : LLM knows what you are looking for before generation

    Li, Y., Huang, Y., Yang, B., Venkitesh, B., Locatelli, A., Ye, H., Cai, T., Lewis, P., and Chen, D. Snap KV : LLM knows what you are looking for before generation. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net...

    2024

  17. [25]

    H., Li, D., Gao, J., Yang, Y., and Qiu, L

    LI, Y., Jiang, H., Wu, Q., Luo, X., Ahn, S., Zhang, C., Abdi, A. H., Li, D., Gao, J., Yang, Y., and Qiu, L. SCB ench: A KV cache-centric analysis of long-context methods. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/f...

    2025

  18. [26]

    Baichuan-omni-1.5 technical report

    Li, Y., Liu, J., Zhang, T., Chen, S., Li, T., Li, Z., Liu, L., Ming, L., Dong, G., Pan, D., et al. Baichuan-omni-1.5 technical report. ArXiv preprint, abs/2501.15368, 2025. URL https://arxiv.org/abs/2501.15368

    2025

  19. [27]

    Retrievalattention: Accelerating long-context llm inference via vector retrieval

    Liu, D., Chen, M., Lu, B., Jiang, H., Han, Z., Zhang, Q., Chen, Q., Zhang, C., Ding, B., Zhang, K., et al. Retrievalattention: Accelerating long-context llm inference via vector retrieval. ArXiv preprint, abs/2409.10516, 2024 a . URL https://arxiv.org/abs/2409.10516

    2024 arXiv

  20. [28]

    Dynamic sparse attention for scalable transformer acceleration

    Liu, L., Qu, Z., Chen, Z., Tu, F., Ding, Y., and Xie, Y. Dynamic sparse attention for scalable transformer acceleration. IEEE Trans. Computers, pp.\ 3165--3178, 2022. doi:10.1109/TC.2022.3208206

    2022

  21. [29]

    A survey on medical large language models: Technology, application, trustworthiness, and future directions

    Liu, L., Yang, X., Lei, J., Liu, X., Shen, Y., Zhang, Z., Wei, P., Gu, J., Chu, Z., Qin, Z., et al. A survey on medical large language models: Technology, application, trustworthiness, and future directions. ArXiv preprint, abs/2406.03712, 2024 b . URL https://arxiv.org/abs/2406.03712

    2024 arXiv

  22. [30]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. Swin transformer: Hierarchical vision transformer using shifted windows. ICCV, pp.\ 9992--10002, 2021. doi:10.1109/ICCV48922.2021.00986

    2021

  23. [31]

    Moba: Mixture of block attention for long-context llms

    Lu, E., Jiang, Z., Liu, J., Du, Y., Jiang, T., Hong, C., Liu, S., He, W., Yuan, E., Wang, Y., et al. Moba: Mixture of block attention for long-context llms. arXiv preprint arXiv:2502.13189, 2025

    2025 arXiv

  24. [32]

    A., Khan, S., and Khan, F

    Maaz, M., Rasheed, H. A., Khan, S., and Khan, F. Video-chatgpt: Towards detailed video understanding via large vision and language models. ACL, pp.\ 12585--12602, 2024. doi:10.18653/V1/2024.ACL-LONG.679

    2024 doi

  25. [33]

    Egoschema: A diagnostic benchmark for very long-form video language understanding

    Mangalam, K., Akshulakov, R., and Malik, J. Egoschema: A diagnostic benchmark for very long-form video language understanding. NeurIPS, 2023

    2023

  26. [34]

    Perception test: A diagnostic benchmark for multimodal video models

    Patraucean, V., Smaira, L., Gupta, A., Recasens, A., Markeeva, L., Banarse, D., Koppula, S., Heyward, J., Malinowski, M., Yang, Y., Doersch, C., Matejovicova, T., Sulsky, Y., Miech, A., Fréchette, A., Klimczak, H., Koster, R., Zhang, J., Winkler, S., Aytar, Y., Osindero, S., D...

    2023

  27. [35]

    Perception test: A diagnostic benchmark for multimodal video models

    Patraucean, V., Smaira, L., Gupta, A., Recasens, A., Markeeva, L., Banarse, D., Koppula, S., Malinowski, M., Yang, Y., Doersch, C., et al. Perception test: A diagnostic benchmark for multimodal video models. Advances in Neural Information Processing Systems, 36, 2024

    2024

  28. [36]

    Consistency policy: Accelerated visuomotor policies via consistency distillation

    Prasad, A., Lin, K., Wu, J., Zhou, L., and Bohg, J. Consistency policy: Accelerated visuomotor policies via consistency distillation. ArXiv preprint, abs/2405.07503, 2024. URL https://arxiv.org/abs/2405.07503

    2024 arXiv

  29. [37]

    Dao, tri and haziza, daniel and massa, francisco and sizov, grigory, 2023

    Qwen, T. Dao, tri and haziza, daniel and massa, francisco and sizov, grigory, 2023. URL https://crfm.stanford.edu/2023/10/12/flashdecoding.html

    2023

  30. [38]

    Qvq: To see the world with wisdom, 2024

    Qwen, T. Qvq: To see the world with wisdom, 2024. URL https://qwenlm.github.io/blog/qvq-72b-preview/

    2024

  31. [39]

    Sparq attention: Bandwidth-efficient llm inference

    Ribar, L., Chelombiev, I., Hudlass-Galley, L., Blake, C., Luschi, C., and Orr, D. Sparq attention: Bandwidth-efficient llm inference. ICML, 2024

    2024

  32. [40]

    Longvu: Spatiotemporal adaptive compression for long video-language understanding

    Shen, X., Xiong, Y., Zhao, C., Wu, L., Chen, J., Zhu, C., Liu, Z., Xiao, F., Varadarajan, B., Bordes, F., et al. Longvu: Spatiotemporal adaptive compression for long video-language understanding. ArXiv preprint, abs/2410.17434, 2024. URL https://arxiv.org/abs/2410.17434

    arXiv

  33. [41]

    I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al

    Team, G., Georgiev, P., Lei, V. I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. ArXiv preprint, abs/2403.05530, 2024. URL https://arxiv.org/abs/2403.05530

    2024 arXiv

  34. [42]

    Triton: an intermediate language and compiler for tiled neural network computations

    Tillet, P., Kung, H.-T., and Cox, D. Triton: an intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, pp.\ 10--19, 2019

    2019

  35. [43]

    VL -cache: Sparsity and modality-aware KV cache compression for vision-language model inference acceleration

    Tu, D., Vashchilenko, D., Lu, Y., and Xu, P. VL -cache: Sparsity and modality-aware KV cache compression for vision-language model inference acceleration. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=HMrcv7Q4Ub

    2025

  36. [44]

    Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution

    Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. ArXiv preprint, abs/2409.12191, 2024 a . URL https://arxiv.org/abs/2409.12191

    2024 arXiv

  37. [45]

    Longllava: Scaling multi-modal llms to 1000 images efficiently via a hybrid architecture

    Wang, X., Song, D., Chen, S., Zhang, C., and Wang, B. Longllava: Scaling multi-modal llms to 1000 images efficiently via a hybrid architecture. ArXiv preprint, abs/2409.02889, 2024 b . URL https://arxiv.org/abs/2409.02889

    2024

  38. [46]

    Drivedreamer: Towards real-world-drive world models for autonomous driving

    Wang, X., Zhu, Z., Huang, G., Chen, X., Zhu, J., and Lu, J. Drivedreamer: Towards real-world-drive world models for autonomous driving. In European Conference on Computer Vision, pp.\ 55--72. Springer, 2024 c

    2024

  39. [47]

    Efficient vision-language models by summarizing visual tokens into compact registers

    Wen, Y., Cao, Q., Fu, Q., Mehta, S., and Najibi, M. Efficient vision-language models by summarizing visual tokens into compact registers. ArXiv preprint, abs/2410.14072, 2024. URL https://arxiv.org/abs/2410.14072

    2024 arXiv

  40. [48]

    Longvlm: Efficient long video understanding via large language models

    Weng, Y., Han, M., He, H., Chang, X., and Zhuang, B. Longvlm: Efficient long video understanding via large language models. ECCV, pp.\ 453--470, 2024. doi:10.1007/978-3-031-73414-4_26

    2024 doi

  41. [49]

    Retrieval head mechanistically explains long-context factuality

    Wu, W., Wang, Y., Xiao, G., Peng, H., and Fu, Y. Retrieval head mechanistically explains long-context factuality. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=EytBpUGB1Z

    2025

  42. [50]

    Sparse videogen: Accelerating video diffusion transformers with spatial-temporal sparsity

    Xi, H., Yang, S., Zhao, Y., Xu, C., Li, M., Li, X., Lin, Y., Cai, H., Zhang, J., Li, D., et al. Sparse videogen: Accelerating video diffusion transformers with spatial-temporal sparsity. arXiv preprint arXiv:2502.01776, 2025

    2025 arXiv

  43. [51]

    Efficient streaming language models with attention sinks

    Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Efficient streaming language models with attention sinks. ICLR, 2024

    2024

  44. [52]

    Next-qa: Next phase of question-answering to explaining temporal actions

    Xiao, J., Shang, X., Yao, A., and Chua, T.-S. Next-qa: Next phase of question-answering to explaining temporal actions. CVPR, pp.\ 9777--9786, 2021. doi:10.1109/CVPR46437.2021.00965

    2021

  45. [53]

    Xu, J., Guo, Z., He, J., Hu, H., He, T., Bai, S., Chen, K., Wang, J., Fan, Y., Dang, K., et al. Qwen2. 5-omni technical report. ArXiv preprint, abs/2503.20215, 2025 a . URL https://arxiv.org/abs/2503.20215

    2025 arXiv

  46. [54]

    Xattention: Block sparse attention with antidiagonal scoring

    Xu, R., Xiao, G., Huang, H., Guo, J., and Han, S. Xattention: Block sparse attention with antidiagonal scoring. ArXiv preprint, abs/2503.16428, 2025 b . URL https://arxiv.org/abs/2503.16428

    2025 arXiv

  47. [55]

    Visionzip: Longer is better but not necessary in vision language models

    Yang, S., Chen, Y., Tian, Z., Wang, C., Li, J., Yu, B., and Jia, J. Visionzip: Longer is better but not necessary in vision language models. ArXiv preprint, abs/2412.04467, 2024. URL https://arxiv.org/abs/2412.04467

    2024

  48. [56]

    Activitynet-qa: A dataset for understanding complex web videos via question answering

    Yu, Z., Xu, D., Yu, J., Yu, T., Zhao, Z., Zhuang, Y., and Tao, D. Activitynet-qa: A dataset for understanding complex web videos via question answering. AAAI, pp.\ 9127--9134, 2019. doi:10.1609/AAAI.V33I01.33019127

    2019 doi

  49. [57]

    Native sparse attention: Hardware-aligned and natively trainable sparse attention

    Yuan, J., Gao, H., Dai, D., Luo, J., Zhao, L., Zhang, Z., Xie, Z., Wei, Y., Wang, L., Xiao, Z., et al. Native sparse attention: Hardware-aligned and natively trainable sparse attention. arXiv preprint arXiv:2502.11089, 2025

    2025 arXiv

  50. [58]

    Long context transfer from language to vision

    Zhang, P., Zhang, K., Li, B., Zeng, G., Yang, J., Zhang, Y., Wang, Z., Tan, H., Li, C., and Liu, Z. Long context transfer from language to vision. ArXiv preprint, abs/2406.16852, 2024 a . URL https://arxiv.org/abs/2406.16852

    2024 arXiv

  51. [59]

    Fast video generation with sliding tile attention

    Zhang, P., Chen, Y., Su, R., Ding, H., Stoica, I., Liu, Z., and Zhang, H. Fast video generation with sliding tile attention. arXiv preprint arXiv:2502.04507, 2025

    2025 arXiv

  52. [60]

    Video instruction tuning with synthetic data

    Zhang, Y., Wu, J., Li, W., Li, B., Ma, Z., Liu, Z., and Li, C. Video instruction tuning with synthetic data. ArXiv preprint, abs/2410.02713, 2024 b . URL https://arxiv.org/abs/2410.02713

    2024 arXiv

  53. [61]

    Pit: Optimization of dynamic sparse deep learning models via permutation invariant transformation

    Zheng, N., Jiang, H., Zhang, Q., Han, Z., Ma, L., Yang, Y., Yang, F., Zhang, C., Qiu, L., Yang, M., et al. Pit: Optimization of dynamic sparse deep learning models via permutation invariant transformation. In Proceedings of the 29th Symposium on Operating Systems Principles, p...

    2023

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.