Pith. sign in

REVIEW 3 major objections 5 minor 49 references

Task-Aware KV Compression For Cost-Effective Long Video Understanding

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read By keeping fine detail only for task-critical chunks, Video-X2L beats uniform KV compression on long-video QA while cutting cache size by 86 percent.

desk verdict A genuinely useful, training-free KV-compression idea that is well ablated but whose headline numbers are tuned per benchmark and need a fixed-k out-of-sample rerun before they can be believed. read the letter →

arxiv 2506.21184 v1 pith:ZL6VPMZJ submitted 2025-06-26 cs.CV cs.AI

Minghao Qin , Yan Shu , Peitian Zhang , Kun Lun , Huaying Yuan , Juenjie Zhou , Shitao Xiao , Bo Zhao
show 1 more author
Zheng Liu
This is my paper
classification cs.CVcs.AI
keywords long-videounderstandingKVcachecompressionmultimodallargelanguagemodelstask-awareselectionselectivere-loadingrelevanceoracleinferenceefficiencyratio
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Long-video question answering is expensive because every frame's visual tokens become key-value (KV) cache entries the model must attend to during generation. Uniform KV compression shrinks the whole video at one fixed rate, so at high ratios it loses exactly the fine details detailed questions need. Video-X2L instead compresses each video chunk twice: once into fine-grained low-compression KVs (L-KVs) and once into abstract high-compression KVs (H-KVs). For a given question, a relevance oracle scores every chunk, and only the top-k chunks are decoded with L-KVs while the rest stay at H-KV compression. The paper reports that this task-aware redistribution outperforms uniform low compression on several long-video benchmarks while reducing the KV cache by 86.1% and speeding decoding by 4.5x.

What carries the argument

The central mechanism is bi-level KV compression followed by selective KV re-loading. A KV cache stores the key and value vectors that a transformer's attention layers compute for each token; in a KV-compressible multimodal large language model, visual summary tokens inserted into each video chunk absorb the chunk's information, and their KV states become a compressed stand-in for the chunk. Video-X2L runs this compression twice per chunk to produce L-KVs (small compression ratio, fine detail) and H-KVs (large compression ratio, abstract summary), keeps both in CPU memory after the first pass, and at decode time uses a relevance oracle to rank chunks for the given question. Only the top-k chunks' L-KVs are moved back to GPU memory, merged with the H-KVs and re-ordered by temporal position, so the final context has hybrid compression ratios. This is what carries the argument: the model sees most of the video at high compression but decodes the few chunks that matter for the question at near-full detail.

What would settle it

Pick a long video and a question whose answer depends on a detail inside a chunk that the relevance oracle ranks below the top-k, run Video-X2L with and without L-KV reloading, and compare accuracies on such items. If accuracy drops to the no-L-KV baseline whenever the oracle ranks the critical chunk low, then the oracle, not the reloading mechanism, is the binding constraint; if accuracy stays high even under a deliberately bad oracle, then the H-KVs alone retain enough detail to support the central claim.

Watch

Extended reading notes

Core claim

Video-X2L's central claim is that the information lost by high KV compression is not equally costly across a video: most long-video tasks rely on a small number of critical chunks, so the compression budget should be allocated per task instead of globally. In the pre-filling stage, the model produces both L-KVs and H-KVs for every chunk (for example, 2× and 32× compression), and during decoding it re-loads L-KVs only for the top-k chunks ranked by a relevance oracle such as LanguageBind, using H-KVs elsewhere. The paper reports that on MLVU, LongVideoBench, and VNBench this hybrid cache beats uniform 2× compression while cutting KV cache size by 86.1% and achieving a 4.5x decoding speedup at the default 2×/32× setting. The approach is training-free, plug-compatible with any KV-compressible MLLM, and its advantage grows at higher compression ratios and longer videos.

Load-bearing premise

The load-bearing premise is that the relevance oracle correctly ranks the genuinely critical chunks at or near the top; when it fails, as the paper notes for timestamped questions, those chunks are decoded from heavily compressed H-KVs and the fine detail is lost.

Editorial extensions

If this is right

  • Any existing KV-compressible MLLM can adopt Video-X2L without retraining; the paper shows gains with Video-XL-7B as the backbone and with several relevance oracles including LanguageBind, InternVideo2, SigLIP, and attention scores.
  • At the default 2×/32× setting, Video-X2L reduces KV cache by 86.1%, shortens time-to-first-token from 139.7 ms to 52.1 ms on MLVU, and still beats uniform 2× compression on MLVU, LongVideoBench, and VNBench.
  • On detail-oriented tasks, including single- and multi-detail MLVU categories and VNBench needle tasks, the selective reloading is where the gains concentrate; holistic tasks stay comparable because the first and last chunks are always kept at low compression.
  • In the extra-long Needle-In-A-Haystack setting, Video-X2L supports up to 2,048 frames on one A100 GPU and keeps 100% needle accuracy below 1,000 frames, degrading more slowly than Video-XL beyond that length.
  • A light post-training on 4K video QA samples widens the gap over the backbone on MLVU and VideoMME, indicating the mechanism is compatible with further adaptation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural stress test the paper does not run is to make the relevance oracle adversarially wrong: if a question-relevant chunk is always ranked below the top-k, Video-X2L should behave like the H-KV-only baseline, which would confirm the oracle as the bottleneck and define the boundary of the method's promise.
  • The same two-level cache idea transfers to other long multimodal inputs, such as multi-page documents or high-resolution image grids, where most queries depend on localized regions and a cheap retriever could pick which pages or patches get fine-grained KVs.
  • Because the paper tunes top-k per task, replacing the fixed k with a confidence threshold or learned policy over the oracle's score distribution could remove a per-benchmark hyperparameter and make the method fully adaptive.
  • The paper's savings are measured at decode time; the pre-filling stage still runs two compression passes. Deriving H-KVs from the already-computed L-KVs, or making the second pass cheaper, is a direct extension that would push the efficiency gain to the whole pipeline.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Video-X^2L, a training-free extension of KV-compressible video MLLMs. During pre-filling, the video is encoded twice into low-compression KVs (L-KVs) and high-compression KVs (H-KVs). During decoding, a relevance oracle (LanguageBind in the main experiments) scores each video chunk against the task, and L-KVs are reloaded for the top-k chunks while H-KVs are used for the rest. Experiments on MLVU, VideoMME, LongVideoBench, VNBench, and a needle-in-a-haystack setting report that Video-X^2L outperforms Video-XL at both 2x and 32x compression while cutting decode-time cache by about 86% and speeding up decoding by about 4.5x. Ablations examine the role of the relevance oracle, the compression-ratio pair, the top-k value, and the two KV types.

Significance. If the reported gains are out-of-sample, the paper would make a useful practical contribution: a training-free, plug-in mechanism that preserves fine-grained detail exactly where a given question needs it, at a fraction of the decode-time cost of uniform low compression. The paper deserves credit for being explicit about the mechanism, for releasing code, and for ablating the relevance oracle and the two KV types (Tables 3 and 6). The central empirical claim, however, is currently weakened by test-set hyperparameter tuning, so the significance of the headline numbers is not yet established.

major comments (3)
  1. [Section 4.1, Table 1, Appendix A.2] The main results are produced after tuning top-k separately for each benchmark on the same test sets, with k ranging from 1 to 5 (Section 4.1). Appendix A.2 shows that accuracy is nonmonotonic in k and that the optimal k differs across task types. The numbers in Table 1 are therefore best-case selections over a test-set grid, and the reported '+1.7' or '+2.5' improvements over Video-XL are not clearly out-of-sample. Please report results with a fixed k (for example, k=3, which is used in most ablations), or select k on a validation split, and show that the headline conclusions are unchanged.
  2. [Section 4.4, Table 2] The default compression-ratio pair (2x low, 32x high) is the best-performing row in Table 2 on MLVU. This is another selection made after looking at test performance. The paper should either state that the pair was fixed before the main comparison, or provide a robustness analysis showing that the conclusions in Table 1 hold for a range of pairs (e.g., 2x16x, 2x32x, 2x72x) rather than only for the empirically best one.
  3. [Section 4.6, Table 3] Table 3 shows that on MLVU the Random baseline (63.7) and the Uniform heuristic (64.0) are close to the LanguageBind oracle (65.4), and on LongVideoBench Random (50.3) and Uniform (50.6) are close to LanguageBind (51.7). This weakens the claim that task-aware scoring is what drives the improvement: much of the gain could come from simply having some L-KVs in the context. Please add a discussion of this gap, report variance or significance across oracle choices, and temper the 'task-aware' causal claim accordingly.
minor comments (5)
  1. [Throughout] There are several typos and duplicated phrases, e.g., 'geneartes' in Section 1, 'oeprations' in the contribution list, and the repeated sentence 'This is because the two key innovative mechanism...' in Section 4.5. A careful proofread is needed.
  2. [Table 2 caption] The caption says 'Video-XL: m×n× uses a low compression ratio of m and a high compression ratio of n', but the settings are configurations of Video-X^2L, not Video-XL. This should be corrected to avoid confusion.
  3. [Appendix A.2, Figure 3] The text refers to 'Recall@topk' but the metric is not defined, and the figure is not included in the provided text. Please define the metric and ensure the figure is actually present.
  4. [Table 1] The abbreviation 'M-avg' is used without a definition, and 'LongVideo.' is an unclear truncation of LongVideoBench. Please expand these terms in the table caption or footnote.
  5. [Section 3.3, Eq. (3)] The notation 'K(V:i)' and 'V(V:i)' is ambiguous; it should be clarified whether these denote the KVs of all previous visual summary tokens before chunk i, and the indexing should be made precise.

Circularity Check

1 steps flagged · score 4.0 of 10

Headline advantage rests on per-benchmark top-k tuning on the test benchmarks, so the reported gains are partially fitted rather than out-of-sample.

  1. fitted input called prediction [Section 4.1 (Implementation), hyperparameter description; cf. Table 1 and Appendix A.2 Figure 3]
    "VideoX2L’s performance is optimized by tuning the top-k parameter of L-KVs selection for each task across all benchmarks, with k values ranging from 1 to 5."

    The headline benchmark numbers are produced after choosing k on the same test benchmarks used for comparison. Since Appendix A.2 shows accuracy varies non-monotonically with k, each reported score is effectively the maximum over a 5-point grid per benchmark, while the Video-XL baselines use fixed compression ratios. The claimed advantage is therefore, by construction, the best-case gain over that grid rather than the output of a fixed, untuned policy; a fixed-k evaluation would be needed to treat the result as an out-of-sample prediction.

full rationale

The core mechanism (bi-level KV compression plus selective reloading) is not derived from the target benchmark numbers, and the backbone Video-XL is re-evaluated under standardized compression ratios rather than taken from its original paper. The relevance oracle is an external pretrained model, so there is no self-definitional loop, and no load-bearing uniqueness theorem is imported from the authors' prior work. The main circularity-adjacent issue is the disclosed per-benchmark tuning of top-k on the test sets, which turns the reported comparisons into a best-of-grid result for Video-X2L while the baselines are fixed. This does not collapse the method to its inputs, but it makes the claimed 'huge advantage' partly a fitted maximum rather than a fully out-of-sample prediction. The score of 4 reflects that the central idea retains independent content, but the headline comparison is not entirely free of fitted-input inflation.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or architectural entities; its contributions are L-KVs and H-KVs, which are two compression levels of the same KV cache rather than new objects. The load-bearing free parameters are the per-benchmark top-k (Section 4.1), the compression-ratio pair grid-searched in Table 2, and the hand-set chunk width. The key domain assumptions are the sparsity of task-relevant video information, the reliability of the relevance oracle, and the coherence of mixing KVs from different compression passes.

free parameters (3)
  • top-k (number of L-KV chunks reloaded) = tuned per task/benchmark in range 1 to 5; exact values not reported
    Section 4.1: 'Video-X2L's performance is optimized by tuning the top-k parameter of L-KVs selection for each task across all benchmarks, with k values ranging from 1 to 5.' This is a per-benchmark fitted knob that directly affects all main results.
  • compression ratio pair (alpha_l, alpha_h) = 2x low, 32x high in main results
    Table 2 explores pairs (2x, 8x/16x/32x/72x) and (4x, 8x/16x/32x/72x); the default 2x/32x is the best-performing row ('The optimal setting is Video-X2L 2x 32x'). Reporting the best of the explored grid makes the headline numbers conditional on this selection.
  • video chunk width = 10 frames
    Section 4.1: 'subsequently partitioned into contiguous 10-frame chunks for subsequent processing.' This is a hand-chosen design decision that affects both compression granularity and oracle scoring.
assumptions (4)
  • domain assumption Task-relevant information in long videos is concentrated in a small number of chunks
    Section 3.2: 'LVU tasks ... exhibit non-uniform reliance on different parts of the video content. In most situations, it only needs to highlight a few crucial details while maintaining an abstract understanding of the global context.' The whole selective reloading mechanism depends on this sparsity.
  • domain assumption L-KVs and H-KVs from two separate compression passes can be merged into one context and decoded coherently
    Section 3.4 merges VA (L-KVs) and VB (H-KVs) into VA+B and runs self-attention with temporal reordering. The paper does not analyze distributional mismatch between KVs produced at different compression ratios; the NIAH and ablation results are the only support.
  • domain assumption The relevance oracle supplies accurate chunk-importance scores
    Section 3.4: 'the importance between the task t and a video chunk Xi is estimated by a relevance oracle.' Limitation B admits timestamped QA is hard for current oracles, and Section 4.4 attributes 72x-ratio degradation to incorrect oracle reloading.
  • standard math The Video-XL compression workflow (visual summary tokens, chunked self-attention) behaves correctly when reused as the backbone
    Section 3.3 adopts the compression workflow from Video-XL [29] without re-deriving it. The paper relies on the prior method's correctness for the KV compression operations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Task-Aware KV Compression For Cost-Effective Long Video Understanding." pith.science (2026). https://pith.science/paper/ZL6VPMZJ

@misc{pith2026250621184,
  author       = {Pith},
  title        = {Pith review of: Task-Aware KV Compression For Cost-Effective Long Video Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZL6VPMZJ}},
  note         = {Machine review of arXiv:2506.21184}
}
read the original abstract

Long-video understanding (LVU) remains a severe challenge for existing multimodal large language models (MLLMs), primarily due to the prohibitive computational cost. Recent approaches have explored KV compression to mitigate this issue, but they often suffer from significant information loss at high compression ratios. In this paper, we introduce Video-X^2L, which flexibly preserves critical video information for each LVU task. Video-X^2L involves two key operations. The first one is called bi-level KV compression. During the MLLM's pre-filling stage, Video-X^2L generates two types of compressed KVs: low-compression KVs (L-KVs) to capture fine-grained video details and high-compression KVs (H-KVs) to offer compact video representations. The second one is called selective KV re-loading. During the MLLM's decoding stage, Video-X^2L selectively re-loads L-KVs for the most critical video chunks while using H-KVs for other less important ones. This allows the MLLM to fully utilize task-specific information while maintaining the overall compactness. Video-X^2L is simple yet effective: it is free from additional training and directly compatible with existing KV-compressible MLLMs. We evaluate Video-X^2L with a variety of popular LVU benchmarks, including VideoMME, MLVU, LongVideoBench, and VNBench. Our experiment result shows that Video-X^2L outperforms existing KV-compression methods by a huge advantage while substantially saving the computation cost.

Figures

Figures reproduced from arXiv: 2506.21184 by the authors.

Figure 1
Figure 1. The framework of Video-X2L. First, it performs bi-level KV compression, which com￾presses KV cache into L-KVs (for fine-grained details) and H-KVs (for abstract representations). Second, it make selective use of the compressed KVs based on their importance to each task. The final answer is generated using the selectively re-loaded KVs of hybrid compression ratios. introduced as a more promising strategy, which lever… view at source ↗
Figure 2
Figure 2. Results on the Needle-In-A-Haystack evaluatio within a single A100 80GB GPU. The [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Analysis on the impact of recall@topk. To investigate the impact of the top-k selection and reloading task-relevant L-KVs on Video-X2L’s performance, we conduct comparative experiments with different values of k ranging from 1 to 6. We evaluate the model on single-detail and multi-detail tasks from MLVU. As shown in [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

49 extracted references · 3 canonical work pages

  1. [1]

    Flamingo: a visual language model for few-shot learning.NeurIPS, 2022

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning.NeurIPS, 2022

    work page 2022

  2. [2]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

    arXiv 2025

  3. [3]

    Sharegpt4video: Improving video understanding and generation with better captions.arXiv preprint arXiv:2406.04325, 2024

    Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, et al. Sharegpt4video: Improving video understanding and generation with better captions.arXiv preprint arXiv:2406.04325, 2024

    arXiv 2024

  4. [4]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites.arXiv preprint arXiv:2404.16821, 2024

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites.arXiv preprint arXiv:2404.16821, 2024

    arXiv 2024

  5. [5]

    Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms.arXiv preprint arXiv:2406.07476, 2024

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms.arXiv preprint arXiv:2406.07476, 2024

    arXiv 2024

  6. [6]

    Nvlm: Open frontier-class multimodal llms.arXiv preprint arXiv:2409.11402, 2024

    Wenliang Dai, Nayeon Lee, Boxin Wang, Zhuolin Yang, Zihan Liu, Jon Barker, Tuomas Rintamaki, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Nvlm: Open frontier-class multimodal llms.arXiv preprint arXiv:2409.11402, 2024

    arXiv 2024

  7. [7]

    Video-ccam: Enhancing video-language understanding with causal cross-attention masks for short and long videos.arXiv preprint arXiv:2408.14023, 2024

    Jiajun Fei, Dian Li, Zhidong Deng, Zekun Wang, Gang Liu, and Hui Wang. Video-ccam: Enhancing video-language understanding with causal cross-attention masks for short and long videos.arXiv preprint arXiv:2408.14023, 2024

    arXiv 2024

  8. [8]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis.arXiv preprint arXiv:2405.21075, 2024

    Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis.arXiv preprint arXiv:2405.21075, 2024

    arXiv 2024

Show all 49 references
  1. [9]

    Ma-lmm: Memory-augmented large multimodal model for long-term video understanding

    Bo He, Hengduo Li, Young Kyun Jang, Menglin Jia, Xuefei Cao, Ashish Shah, Abhinav Shrivastava, and Ser-Nam Lim. Ma-lmm: Memory-augmented large multimodal model for long-term video understanding. arXiv preprint arXiv:2404.05726, 2024

    2024 arXiv

  2. [10]

    Chat-univi: Unified visual representation empowers large language models with image and video understanding

    Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation empowers large language models with image and video understanding. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13700–13...

    2024

  3. [11]

    Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  4. [12]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.ICML, 2023

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.ICML, 2023

    2023

  5. [13]

    Videochat: Chat-centric video understanding.arXiv preprint arXiv:2305.06355, 2023

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding.arXiv preprint arXiv:2305.06355, 2023

    2023 arXiv

  6. [14]

    Mvbench: A comprehensive multi-modal video understanding benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195–22206, 2024

    2024

  7. [15]

    Llama-vid: An image is worth 2 tokens in large language models.arXiv preprint arXiv:2311.17043, 2023

    Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models.arXiv preprint arXiv:2311.17043, 2023

    2023 arXiv

  8. [16]

    Llama-vid: An image is worth 2 tokens in large language models

    Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. 2024

    2024

  9. [17]

    Video-llava: Learning united visual representation by alignment before projection.arXiv preprint arXiv:2311.10122, 2023

    Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection.arXiv preprint arXiv:2311.10122, 2023

    2023 arXiv

  10. [18]

    Llava-next: Improved reasoning, ocr, and world knowledge, January 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024

    2024

  11. [19]

    Visual instruction tuning.arXiv preprint arXiv:2304.08485, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.arXiv preprint arXiv:2304.08485, 2023. 10

    2023 arXiv

  12. [20]

    Visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023

    2023

  13. [21]

    St-llm: Large language models are effective temporal learners.arXiv:2404.00308, 2024

    Ruyang Liu, Chen Li, Haoran Tang, Yixiao Ge, Ying Shan, and Ge Li. St-llm: Large language models are effective temporal learners.arXiv:2404.00308, 2024

    2024 arXiv

  14. [22]

    Video-xl-pro: Reconstructive token compression for extremely long video understanding.arXiv preprint arXiv:2503.18478, 2025

    Xiangrui Liu, Yan Shu, Zheng Liu, Ao Li, Yang Tian, and Bo Zhao. Video-xl-pro: Reconstructive token compression for extremely long video understanding.arXiv preprint arXiv:2503.18478, 2025

    2025 arXiv

  15. [23]

    Valley: Video assistant with large language model enhanced ability.arXiv preprint arXiv:2306.07207, 2023

    Ruipu Luo, Ziwang Zhao, Min Yang, Junwei Dong, Da Li, Pengcheng Lu, Tao Wang, Linmei Hu, Minghui Qiu, and Zhongyu Wei. Valley: Video assistant with large language model enhanced ability.arXiv preprint arXiv:2306.07207, 2023

    2023 arXiv

  16. [24]

    Video-chatgpt: Towards detailed video understanding via large vision and language models.arXiv preprint arXiv:2306.05424, 2023

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models.arXiv preprint arXiv:2306.05424, 2023

    2023 arXiv

  17. [25]

    Gpt-4 technical report, 2023

    OpenAI. Gpt-4 technical report, 2023

    2023

  18. [26]

    Gpt-4o.https://openai.com/index/hello-gpt-4o/, May 2024

    OpenAI. Gpt-4o.https://openai.com/index/hello-gpt-4o/, May 2024

    2024

  19. [27]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530, 2024

    Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv prep...

    2024 arXiv

  20. [28]

    Longvu: Spatiotemporal adaptive compression for long video-language understanding.arXiv preprint arXiv:2410.17434, 2024

    Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishnan Varadarajan, Florian Bordes, et al. Longvu: Spatiotemporal adaptive compression for long video-language understanding.arXiv preprint arXiv:2410.17434, 2024

    arXiv

  21. [29]

    Video-xl: Extra-long vision language model for hour-scale video understanding.arXiv preprint arXiv:2409.14485, 2024

    Yan Shu, Zheng Liu, Peitian Zhang, Minghao Qin, Junjie Zhou, Zhengyang Liang, Tiejun Huang, and Bo Zhao. Video-xl: Extra-long vision language model for hour-scale video understanding.arXiv preprint arXiv:2409.14485, 2024

    2024 arXiv

  22. [30]

    Moviechat: From dense token to sparse memory for long video understanding.arXiv preprint arXiv:2307.16449, 2023

    Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Xun Guo, Tian Ye, Yan Lu, Jenq-Neng Hwang, et al. Moviechat: From dense token to sparse memory for long video understanding.arXiv preprint arXiv:2307.16449, 2023

    2023 arXiv

  23. [31]

    Cambrian-1: A fully open, vision-centric exploration of multimodal llms.Advances in Neural Information Processing Systems, 37:87310–87356, 2024

    Peter Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Adithya Jairam Vedagiri IYER, Sai Charitha Akula, Shusheng Yang, Jihan Yang, Manoj Middepogu, Ziteng Wang, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms.Advances in Neural Information Processin...

    2024

  24. [32]

    Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features.arXiv preprint arXiv:2502.14786, 2025

    Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and ...

    2025 arXiv

  25. [33]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  26. [34]

    Longllava: Scaling multi- modal llms to 1000 images efficiently via hybrid architecture.arXiv preprint arXiv:2409.02889, 2024

    Xidong Wang, Dingjie Song, Shunian Chen, Chen Zhang, and Benyou Wang. Longllava: Scaling multi- modal llms to 1000 images efficiently via hybrid architecture.arXiv preprint arXiv:2409.02889, 2024

    2024

  27. [35]

    Internvideo2: Scaling foundation models for multimodal video understanding

    Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Zun Wang, Yansong Shi, et al. Internvideo2: Scaling foundation models for multimodal video understanding. InEuropean Conference on Computer Vision, pages 396–416. Springer, 2024

    2024

  28. [36]

    Videorope: What makes for good video rotary position embedding?arXiv preprint arXiv:2502.05173, 2025

    Xilin Wei, Xiaoran Liu, Yuhang Zang, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Jian Tong, Haodong Duan, Qipeng Guo, Jiaqi Wang, et al. Videorope: What makes for good video rotary position embedding?arXiv preprint arXiv:2502.05173, 2025

    2025 arXiv

  29. [37]

    Longvideobench: A benchmark for long-context interleaved video-language understanding.arXiv preprint arXiv:2407.15754, 2024

    Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding.arXiv preprint arXiv:2407.15754, 2024

    2024 arXiv

  30. [38]

    Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding.arXiv preprint arXiv:2412.10302, 2024

    Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, et al. Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding.arXiv preprint arXiv:2412.10302, 2024. 11

    2024 arXiv

  31. [39]

    Infllm: Unveiling the intrinsic capacity of llms for understanding extremely long sequences with training-free memory.arXiv preprint arXiv:2402.04617, 2024

    Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, Song Han, and Maosong Sun. Infllm: Unveiling the intrinsic capacity of llms for understanding extremely long sequences with training-free memory.arXiv preprint arXiv:2402.04617, 2024

    2024 arXiv

  32. [40]

    V oco-llama: Towards vision compression with large language models.arXiv preprint arXiv:2406.12275, 2024

    Xubing Ye, Yukang Gan, Xiaoke Huang, Yixiao Ge, Ying Shan, and Yansong Tang. V oco-llama: Towards vision compression with large language models.arXiv preprint arXiv:2406.12275, 2024

    2024 arXiv

  33. [41]

    Video-llama: An instruction-tuned audio-visual language model for video understanding.arXiv preprint arXiv:2306.02858, 2023

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding.arXiv preprint arXiv:2306.02858, 2023

    2023 arXiv

  34. [42]

    Long context transfer from language to vision.arXiv preprint arXiv:2406.16852, 2024

    Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision.arXiv preprint arXiv:2406.16852, 2024

    2024 arXiv

  35. [43]

    Llava-next: A strong zero-shot video understanding model, April 2024

    Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava-next: A strong zero-shot video understanding model, April 2024

    2024

  36. [44]

    H2o: Heavy-hitter oracle for efficient generative inference of large language models.Advances in Neural Information Processing Systems, 36:34661–34710, 2023

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. H2o: Heavy-hitter oracle for efficient generative inference of large language models.Advances in Neural Information Processing System...

    2023

  37. [45]

    Needle in a video haystack: A scalable synthetic framework for benchmarking video mllms

    Zijia Zhao, Haoyu Lu, Yuqi Huo, Yifan Du, Tongtian Yue, Longteng Guo, Bingning Wang, Weipeng Chen, and Jing Liu. Needle in a video haystack: A scalable synthetic framework for benchmarking video mllms. arXiv preprint arXiv:2406.09367, 2024

    2024 arXiv

  38. [46]

    Mlvu: A comprehensive benchmark for multi-task long video understanding.arXiv preprint arXiv:2406.04264, 2024

    Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding.arXiv preprint arXiv:2406.04264, 2024

    2024 arXiv

  39. [47]

    Languagebind: Extending video-language pretraining to n-modality by language- based semantic alignment.arXiv preprint arXiv:2310.01852, 2023

    Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, HongFa Wang, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, et al. Languagebind: Extending video-language pretraining to n-modality by language- based semantic alignment.arXiv preprint arXiv:2310.01852, 2023

    arXiv

  40. [48]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models.arXiv preprint arXiv:2304.10592, 2023

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models.arXiv preprint arXiv:2304.10592, 2023

    2023 arXiv

  41. [49]

    Video-XL:2×

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479, 2025. 12 A Appendix A.1 A...

    2025 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.