Pith. sign in

REVIEW 4 major objections 5 minor 61 references

Seeing More, Saying More: Lightweight Language Experts are Dynamic Video Token Compressors

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A lightweight captioner lets video models spend tokens where the scene is rich.

desk verdict A competent efficiency paper with a genuinely interesting dynamic-compression idea, but the load-bearing claim that caption length tracks semantic density is never actually tested against a fixed-length control. read the letter →

arxiv 2509.00969 v2 pith:ITYPCLNT submitted 2025-08-31 cs.CV

Xiangchen Wang , Jinrui Zhang , Teng Wang , Haigang Zhang , Feng Zheng This is my paper
classification cs.CV
keywords videounderstandingvisualtokencompressiondynamicratiolanguagemodelascompressorvideo-languagemodelssoftcaptiontokenssemanticdensityefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LangDC replaces fixed-ratio visual token compression in video-language models with a dynamic scheme driven by a lightweight captioner. The captioner (CapPruner) outputs a short description of each video segment, and the hidden states of those caption tokens become the compressed visual representation; the number of tokens is simply the length of the generated caption, so information-rich clips automatically get more tokens and static clips get fewer. To make caption length track semantic density, the captioner is trained against descriptions distilled from a strong teacher model, stripped of filler words. On MVBench and Video-MME the method matches or outperforms the VideoGPT+ baseline while cutting FLOPs by 49%, and qualitative results show token counts ranging from about 120 to 300 depending on scene complexity. The paper's point is that compression ratios should be a function of the content, not a global hyperparameter.

What carries the argument

CapPruner: a lightweight pre-trained language model (Qwen-2.5-0.5B) with two projection heads. One head (the LM head) supervises caption generation against the teacher's density-aware descriptions and controls length through the end-of-sentence token; the other (post-projector) maps selected intermediate-layer hidden states of the caption tokens into the LLM's embedding space. The selected depth (layer 15) is chosen because intermediate layers balance semantic content and generality. CapPruner's tokens are combined with a fixed base pruner (mean pooling or LDPv2); the base pruner guarantees a minimal visual footprint while CapPruner supplies adaptive extra tokens.

What would settle it

Collect a dataset of video segments with independent semantic-density labels (e.g., number of objects, motion magnitude, or human-annotated complexity). Run LangDC and, for each segment, extract the caption length. If caption length does not correlate with the independent density labels across a broad sample, the core assumption that language length tracks visual information density is falsified. Alternatively, an ablation that fixes all captions to the same length while preserving content, and which still retains LangDC's accuracy, would show that the dynamic token count is not what drives th

Watch

Extended reading notes

Core claim

The central claim is that a lightweight language model can act as a dynamic token compressor for video: by describing each segment in language and using the hidden states of those caption tokens as the visual representation, the compression ratio is determined by the length of the description. LangDC trains this captioner with semantic density-aware supervision—teacher captions post-processed to remove connective and speculative words—so that the model learns to say more when the scene is rich and less when it is static. The resulting soft caption tokens are fed to the LLM alongside a base pruner's output. Over 16 frames divided into 4 segments, the method cuts FLOPs from 49.85T to 25.15T (4

Load-bearing premise

The entire dynamic compression mechanism assumes that the number of words in the generated caption is a reliable proxy for the information content of a video segment: richer clips produce longer captions and thus more tokens, while static clips produce shorter captions. This correlation is inherited from the teacher-caption post-processing pipeline and is only demonstrated qualitatively (Figure 5, Appendix A), not with a quantitative density measure or a controlled experiment

Editorial extensions

If this is right

  • Video models can spend compute where information actually is, instead of paying a flat per-frame token cost.
  • The same architecture can be dropped into existing LVLMs with different base pruners and different LLM sizes, and the savings grow with LLM size.
  • Because token count tracks scene richness, long-video performance improves without explicit long-video training.
  • Fixed-ratio compressors (pooling, Q-Former, LDPv2) become complementary: LangDC plus any base pruner beats the base pruner alone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The mechanism implies a testable generalization: caption length could be used as an explicit, controllable compression-budget knob, letting a user set a max token count per segment and having the model decide allocation across segments.
  • A cleaner validation of the core hypothesis would correlate caption length with an independent density metric (e.g., detection counts or motion energy); the paper currently demonstrates the correlation only qualitatively.
  • The teacher-distillation pipeline suggests that the quality and length distribution of the teacher's captions, not just the video content, shapes the compression policy; recaptioning with different teachers could yield different efficiency/accuracy trade-offs.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes LangDC, a language-aware dynamic token compressor for video large language models. Instead of applying a fixed compression ratio, LangDC uses a lightweight language model (Qwen2.5-0.5B), called CapPruner, to generate soft caption tokens from each video segment. The number of tokens is determined by the length of the generated caption, which is trained under 'semantic density-aware supervision' obtained from a teacher LVLM (LLaVA-OneVision) followed by post-processing with Qwen2.5-7B. The method is trained in three stages and evaluated on Video-MME, MVBench, and four open-ended VideoQA benchmarks. The main reported result is a 49% FLOPs reduction relative to VideoGPT+ with competitive or slightly better accuracy on several benchmarks. The paper also provides ablations of the training scheme, supervision signal, and pruner combinations, plus qualitative examples of adaptive token allocation.

Significance. If the central claim is validated, LangDC is a useful contribution: it demonstrates that a small language model can serve as a dynamic token compressor, reducing FLOPs substantially while preserving performance. The evaluation is broad, covering multiple-choice and open-ended VideoQA, and the ablations in Tables 6-8 and the appendix investigate several design choices. The code is publicly available, which is a strength. However, the paper's core novelty—that dynamic allocation based on caption length is the source of the benefits—is not cleanly isolated. The closest prior language-model-based compressors (VoCo-LLaMA, Video-XL) are discussed but not compared quantitatively, and the evidence for the caption-length/density proxy is qualitative. These gaps prevent the paper from currently supporting its main causal claim.

major comments (4)
  1. [§4.1, Tables 2-4] The paper distinguishes LangDC from VoCo-LLaMA (Ye et al., 2025) and Video-XL (Shu et al., 2025) in §4.1, but neither method appears in the quantitative comparisons of Tables 2-4. Because those are the closest prior works that also use language models for video token compression, the claimed advantage over 'previous approaches' is unsupported without direct comparison. Please add experiments with these baselines at comparable token budgets and FLOPs.
  2. [§4.1, Fig. 5, Fig. A1, Table A1] The dynamic-compression mechanism relies on the assumption that caption length tracks the semantic density of a video segment. The only evidence is qualitative (Fig. 5) and distributional (Fig. A1); there is no quantitative correlation between per-segment caption length and the oracle token counts from the motivation study in Table A1, nor an independent density measure. The caption post-processing in Fig. A2 removes connectives and words via a separate LLM, so length is partly an artifact of the teacher/summarizer. Please validate the proxy, e.g., by reporting correlation with oracle token counts or by comparing caption length against a controlled density metric.
  3. [§5.3, Table 6] The ablation in Table 6 compares CapPruner+BasePruner against BasePruner alone, but CapPruner changes both the token content (LM-generated soft tokens) and the token count. This does not isolate the benefit of dynamic allocation. A control with fixed-length CapPruner—e.g., truncating or padding each segment to the average allocated token count—is needed to determine whether the accuracy gain comes from the language-model soft tokens themselves or from the dynamic count. If fixed-length CapPruner performs equally, the 'dynamic' contribution is not load-bearing.
  4. [§5.3, Table 8] Table 8 shows that adding the semantic density-aware caption supervision signal improves accuracy by only 0.14-0.27 points over the no-caption condition. Given that this supervision is described in §4.1 as the mechanism that enables length-based dynamic allocation, the small effect is surprising and weakens the causal narrative. The paper should either explain this discrepancy or provide evidence (e.g., token-length distributions) showing that dynamic allocation is absent without this supervision.
minor comments (5)
  1. [General] All experimental results are single-run with no error bars or multiple seeds. Given the small differences in Tables 2 and 3 (e.g., 0.2-1.6 points), significance is unclear. Report variance or at least acknowledge the limitation.
  2. [Table 2] The FLOPs values (49.85T vs. 25.15T) are reported without specifying the exact input configuration. Please state the frame count, segment count, and any preprocessing used for the FLOPs measurement, so readers can reproduce the 49% reduction claim.
  3. [Fig. A1] The axis labels in Figure A1 appear garbled/illegible in the provided version; please ensure the final PDF has readable labels.
  4. [Limitations] The Limitations section mentions resource constraints and single-ratio implementation, but does not acknowledge the unvalidated caption-length/density proxy or the missing fixed-length control. These are central to the paper's core claim and should be discussed.
  5. [Minor typos] There are several typographical errors: 'seperately' in §4, 'LDPv2(Chu et al., 2024)' missing space before the citation, and 'Puatruaucean' in §3 (should be 'Pătrăucean'). Please proofread.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: benchmark results are independent; dynamic token count is an emergent captioner output, not a fitted input.

full rationale

The paper's main quantitative claims—49% FLOPs reduction and competitive accuracy on Video-MME, MVBench, MSVD-QA, etc.—are evaluated against external benchmarks and a strong baseline (VideoGPT+), so they cannot be forced by the method's own construction. The dynamic token budget is produced by a trained lightweight captioner (CapPruner): the number of soft caption tokens equals the number of generated caption tokens, which is an output of the model, not a parameter fitted to the evaluation targets. The supervision comes from an external teacher (LLaVA-OneVision and Qwen2.5-7B post-processing), not from the benchmark labels. The only self-citation (Zhang et al., 2024a, in Related Work) concerns instruction-tuning data and is not load-bearing. A legitimate weakness is that the paper treats caption length as a proxy for semantic density without a quantitative correlation to the oracle token counts of Table 1, and without a fixed-length CapPruner control in Table 6; however, this is an empirical validity gap, not a circular derivation, because the paper does not define the evaluation target (benchmark accuracy or oracle token count) as the model output. The Limitations section does not acknowledge this gap, but absence of a limitation note is not circularity.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The method introduces no new physical entities. Its two key assumptions are the correlation between caption length and semantic density, and the sufficiency of intermediate hidden states as compressed visual representations. Both are empirical and only partially validated.

free parameters (3)
  • max compressed tokens per segment = 128
    CapPruner compresses each segment to a maximum of 128 tokens; this upper bound is chosen by hand and caps the dynamic range.
  • CapPruner hidden state layer = 15 (Qwen2.5-0.5B)
    The layer from which hidden states are taken as compressed tokens; selected based on ablation (Figure A3).
  • LoRA rank = 128
    Used for LLM fine-tuning; standard choice, but still a hyperparameter.
assumptions (3)
  • domain assumption Teacher captions (LLaVA-OneVision) accurately describe key visual cues in each segment
    Used to generate supervision for CapPruner pretraining (Section 4.1); if the teacher misses important cues, the compressed tokens will too.
  • domain assumption Caption length (after post-processing) correlates with semantic density
    The whole dynamic ratio mechanism relies on this correlation; not proven quantitatively.
  • domain assumption Intermediate hidden states of a small LM retain visual semantics
    The paper uses hidden states from layer 15; it ablated this choice but does not justify the mechanism.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Seeing More, Saying More: Lightweight Language Experts are Dynamic Video Token Compressors." pith.science (2026). https://pith.science/paper/ITYPCLNT

@misc{pith2026250900969,
  author       = {Pith},
  title        = {Pith review of: Seeing More, Saying More: Lightweight Language Experts are Dynamic Video Token Compressors},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ITYPCLNT}},
  note         = {Machine review of arXiv:2509.00969}
}
read the original abstract

Recent advancements in large video-language models have revolutionized video understanding tasks. However, their efficiency is significantly constrained by processing high volumes of visual tokens. Existing token compression strategies apply a fixed compression ratio, ignoring the variability in semantic density among different video clips. Consequently, this lead to inadequate representation of information-rich clips due to insufficient tokens and unnecessary computation on static or content-poor ones. To address this, we propose LangDC, a Language-aware Dynamic Token Compressor. LangDC leverages a lightweight language model to describe video clips, converting them into soft caption tokens as visual representations. Trained with our proposed semantic density-aware supervision, LangDC aims to 1) cover key visual cues necessary for downstream task reasoning and 2) dynamically adjust compression ratios based on scene richness, reflected by descriptions length. Our design mimics how humans dynamically express what they see: complex scenes (seeing more) elicit more detailed language to convey nuances (saying more), whereas simpler scenes are described with fewer words. Experimental results show that our method reduces FLOPs by 49% compared to VideoGPT+ while maintaining competitive performance. Furthermore, qualitative results demonstrate our approach adaptively adjusts the token compression ratio based on video segment richness.

Figures

Figures reproduced from arXiv: 2509.00969 by the authors.

Figure 1
Figure 1. Comparison of LangDC and existing token compressors. (a) illustrates two video segments with distinct information densities; the bottom segment contains richer visual cues. However, existing token compression methods (b) represent both segments to the same number of tokens. In contrast, our proposed method (c) dynamically allocates tokens based on semantic density, drawing on the sequence length awareness of languag… view at source ↗
Figure 2
Figure 2. Overview of the proposed method. LangDC utilizes dual visual encoders to extract visual features, followed by dynamic compression using CapPruner. The compressed features are combined with the base pruner’s output and fed into the LLM. The training pipeline consists of three stages: Stage I involves cross-modal pretraining with video/image-caption pairs, Stage II focuses on CapPruner pretraining using an information… view at source ↗
Figure 3
Figure 3. Illustration of the dynamic compression mechanism in CapPruner. (a) Captions generated by a teacher model (a strong captioner) are used to supervise the training of CapPruner, facilitating it to allocate tokens according to scene richness. (b) By leveraging the hidden states of predicted captions as compact representation, CapPruner dynamically adjusts the compression ratio according to the timing of the “end-of-sen… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Comparison of GPU Memory and Latency. on TGIF-QA, while remaining competitive on MSRVTT-QA and ActivityNet-QA. These results confirm CapPruner’s dynamic compression im￾proves efficiency and preserves key semantic de￾tails, boosting generalization in zero-shot settings.…
Figure 5
Figure 5. Figure 5: Visualization of video QA examples alongside the corresponding number of allocated tokens. Action Antonym Object Existence State Change Episodic Reasoning 143.2 184.7 249.1 257.2 [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

61 extracted references · 33 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millicah, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, and 8 others. 2024. Flamingo: a visual language model for few-shot le...

    work page 2024

  4. [4]

    Max Bain, Arsha Nagrani, G \"u l Varol, and Andrew Zisserman. 2021. Frozen in time: A joint video and image encoder for end-to-end retrieval. In ICCV

    work page 2021

  5. [5]

    Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. 2022. https://api.semanticscholar.org/CorpusID:252968113 Token merging: Your vit but faster . ArXiv, abs/2210.09461

    arXiv 2022

  6. [6]

    Mu Cai, Jianwei Yang, Jianfeng Gao, and Yong Jae Lee. 2024. Matryoshka multimodal models. In The Thirteenth International Conference on Learning Representations

    work page 2024

  7. [7]

    Junbum Cha, Wooyoung Kang, Jonghwan Mun, and Byungseok Roh. 2024. Honeybee: Locality-enhanced projector for multimodal llm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    work page 2024

  8. [8]

    Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechu Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. 2023 a . Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.09478

    arXiv 2023

Show all 61 references
  1. [9]

    Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. 2024. https://api.semanticscholar.org/CorpusID:268358224 An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models

    2024

  2. [10]

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Zhong Muyan, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. 2023 b . https://api.semanticscholar.org/CorpusID:266521410 Intern vl: Scaling up vision foundation models a...

    2023

  3. [11]

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, and 1 others. 2023. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1--113

    2023

  4. [12]

    Xiangxiang Chu, Limeng Qiao, Xinyang Lin, Shuang Xu, Yang Yang, Yiming Hu, Fei Wei, Xinyu Zhang, Bo Zhang, Xiaolin Wei, and 1 others. 2023. Mobilevlm: A fast, reproducible and strong vision language assistant for mobile devices. arXiv preprint arXiv:2312.16886

    2023 arXiv

  5. [13]

    Xiangxiang Chu, Limeng Qiao, Xinyu Zhang, Shuang Xu, Fei Wei, Yang Yang, Xiaofei Sun, Yiming Hu, Xinyang Lin, Bo Zhang, and 1 others. 2024. Mobilevlm v2: Faster and stronger baseline for vision language model. arXiv preprint arXiv:2402.03766

    2024 arXiv

  6. [14]

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, and 1 others. 2022. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416

    2022 arXiv

  7. [15]

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, and 1 others. 2025. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. In Proceedings of the Computer ...

    2025

  8. [16]

    something something

    Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, and 1 others. 2017. The" something something" video database for learning and evaluating visual common...

    2017

  9. [17]

    Da Guo, Qingfang Zheng, Xiaojiang Peng, and Ming Liu. 2019. Face detection, alignment, quality assessment and attribute analysis with multi-task hybrid convolutional neural networks. ZTE Communications, 17(3):15--22

    2019

  10. [18]

    Yunseok Jang, Yale Song, Chris Dongjoo Kim, Youngjae Yu, Youngjin Kim, and Gunhee Kim. 2019. Video Question Answering with Spatio-Temporal Reasoning . IJCV

    2019

  11. [19]

    Peng Jin, Ryuichi Takanobu, Caiwan Zhang, Xiaochun Cao, and Li Yuan. 2024. Chat-univi: Unified visual representation empowers large language models with image and video understanding. In CVPR

    2024

  12. [20]

    Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, and 1 others. 2017. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950

    2017 arXiv

  13. [21]

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and 1 others. 2024 a . Llava-onevision: Easy visual task transfer. Transactions on Machine Learning Research

    2024

  14. [22]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. 2023 a . https://api.semanticscholar.org/CorpusID:256390509 Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models . In International Conference on Machine Learning

    2023

  15. [23]

    Kunchang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. 2023 b . Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355

    2023 arXiv

  16. [24]

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, and 1 others. 2024 b . Mvbench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition...

    2024

  17. [25]

    Wentong Li, Yuqian Yuan, Jian Liu, Dongqi Tang, Song Wang, Jianke Zhu, and Lei Zhang. 2024 c . Tokenpacker: Efficient visual projector for multimodal llm. arXiv preprint arXiv:2407.02392

    2024 arXiv

  18. [26]

    Xianhang Li, Haoqin Tu, Mude Hui, Zeyu Wang, Bingchen Zhao, Junfei Xiao, Sucheng Ren, Jieru Mei, Qing Liu, Huangjie Zheng, Yuyin Zhou, and Cihang Xie. 2024 d . https://api.semanticscholar.org/CorpusID:270391661 What if we recaption billions of web images with llama-3? ArXiv, a...

    2024 arXiv

  19. [27]

    Yanwei Li, Chengyao Wang, and Jiaya Jia. 2024 e . https://api.semanticscholar.org/CorpusID:265466723 Llama-vid: An image is worth 2 tokens in large language models . In European Conference on Computer Vision

    2024

  20. [28]

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.342 Video- LL a VA : Learning united visual representation by alignment before projection . In Proceedings of the 2024 Conference on Empirical Methods in ...

    2024 doi

  21. [29]

    Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. 2023 a . https://api.semanticscholar.org/CorpusID:265281544 Video-llava: Learning united visual representation by alignment before projection . In Conference on Empirical Methods in Natural Language Processing

    2023

  22. [30]

    Yijie Lin, Jie Zhang, Zhenyu Huang, Jia Liu, Xi Peng, and 1 others. 2023 b . Multi-granularity correspondence learning from long-term noisy videos. In The Twelfth International Conference on Learning Representations

    2023

  23. [31]

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024 a . https://llava-vl.github.io/blog/2024-01-30-llava-next/ Llava-next: Improved reasoning, ocr, and world knowledge

    2024

  24. [32]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024 b . Visual instruction tuning. Advances in neural information processing systems, 36

    2024

  25. [33]

    Ruyang Liu, Chen Li, Haoran Tang, Yixiao Ge, Ying Shan, and Ge Li. 2024 c . https://doi.org/10.1007/978-3-031-72998-0_1 St-llm: Large language models are effective temporal learners . In Computer Vision – ECCV 2024: 18th European Conference, Milan, Italy, September 29–October ...

    2024 doi

  26. [34]

    Ruipu Luo, Ziwang Zhao, Min Yang, Junwei Dong, Ming-Hui Qiu, Pengcheng Lu, Tao Wang, and Zhongyu Wei. 2023. https://api.semanticscholar.org/CorpusID:259138706 Valley: Video assistant with large language model enhanced ability . ArXiv, abs/2306.07207

    2023 arXiv

  27. [35]

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. 2024 a . Video-chatgpt: Towards detailed video understanding via large vision and language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024)

    2024

  28. [36]

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. 2024 b . https://arxiv.org/abs/2406.09418 Videogpt+: Integrating image and video encoders for enhanced video understanding . arxiv

    2024 arXiv

  29. [37]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Sys...

    2022

  30. [38]

    Koster, Junlin Zhang, Stephanie Winkler, and 5 others

    Viorica Puatruaucean, Lucas Smaira, Ankush Gupta, Adri \`a Recasens Continente, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Joseph Heyward, Mateusz Malinowski, Yezhou Yang, Carl Doersch, Tatiana Matejovicova, Yury Sulsky, Antoine Miech, Alexander Fr \'e chette, Hanna Klimc...

    2023 arXiv

  31. [39]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. https://api.semanticscholar.org/CorpusID:231591445 Learning transferable visual model...

    2021

  32. [40]

    Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. 2024. https://api.semanticscholar.org/CorpusID:265608767 Timechat: A time-sensitive multimodal large language model for long video understanding . IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pa...

    2024

  33. [41]

    Zhongwei Ren, Zhicheng Huang, Yunchao Wei, Yao Zhao, Dongmei Fu, Jiashi Feng, and Xiaojie Jin. 2023. https://api.semanticscholar.org/CorpusID:265659012 Pixellm: Pixel reasoning with large multimodal model . Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern ...

    2023

  34. [42]

    Yan Shu, Peitian Zhang, Zheng Liu, Minghao Qin, Junjie Zhou, Tiejun Huang, and Bo Zhao. 2025. Video-xl: Extra-long vision language model for hour-scale video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    2025

  35. [43]

    Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Xun Guo, Tianbo Ye, Yang Lu, Jenq-Neng Hwang, and Gaoang Wang. 2024. https://api.semanticscholar.org/CorpusID:260333927 Moviechat: From dense token to sparse memory for long video understanding . ...

    2024

  36. [44]

    Qwen Team. 2024. https://qwenlm.github.io/blog/qwen2.5/ Qwen2.5: A party of foundation models

    2024

  37. [45]

    Mariya Toneva and Leila Wehbe. 2019. https://api.semanticscholar.org/CorpusID:167217728 Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain) . In Neural Information Processing Systems

    2019

  38. [46]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023 a . https://arxiv.org/abs/2302.13971 Lla...

    2023 arXiv

  39. [47]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and 1 others. 2023 b . Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  40. [48]

    Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Zun Wang, Yansong Shi, Tianxiang Jiang, Songze Li, Jilan Xu, Hongjie Zhang, Yifei Huang, Yu Qiao, Yali Wang, and Limin Wang. 2024. https://api.semanticscholar.org/CorpusID:271432386 Inte...

    2024

  41. [49]

    Yuetian Weng, Mingfei Han, Haoyu He, Xiaojun Chang, and Bohan Zhuang. 2024. https://doi.org/10.1007/978-3-031-73414-4_26 Longvlm: Efficient long video understanding via large language models . In Computer Vision – ECCV 2024: 18th European Conference, Milan, Italy, September 29...

    2024 doi

  42. [50]

    Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. 2024. Longvideobench: A benchmark for long-context interleaved video-language understanding. Advances in Neural Information Processing Systems, 37:28828--28857

    2024

  43. [51]

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. 2021. Next-qa: Next phase of question-answering to explaining temporal actions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9777--9786

    2021

  44. [52]

    Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. 2017. Video question answering via gradually refined attention over appearance and motion. In ACM MM

    2017

  45. [53]

    Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie

    Jihan Yang, Shusheng Yang, Anjali W. Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. 2025. Thinking in space: How multimodal large language models see, remember, and recall spaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), page...

    2025

  46. [54]

    Xubing Ye, Yukang Gan, Xiaoke Huang, Yixiao Ge, and Yansong Tang. 2025. Voco-llama: Towards vision compression with large language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 29836--29846

    2025

  47. [55]

    Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenenbaum. 2019. Clevrer: Collision events for video representation and reasoning. arXiv preprint arXiv:1910.01442

    2019 arXiv

  48. [56]

    Qingshuang Yu, Jie Zhou, and Wenjuan Gong. 2019. A lightweight sentiment analysis method. ZTE Communications, 17(3):2--8

    2019

  49. [57]

    Hang Zhang, Xin Li, and Lidong Bing. 2023. https://doi.org/10.18653/v1/2023.emnlp-demo.49 Video- LL a MA : An instruction-tuned audio-visual language model for video understanding . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Syst...

    2023 doi

  50. [58]

    Jinrui Zhang, Teng Wang, Haigang Zhang, Ping Lu, and Feng Zheng. 2024 a . Reflective instruction tuning: Mitigating hallucinations in large vision-language models. In ECCV (68)

    2024

  51. [59]

    Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. 2024 b . Llama-adapter: Efficient fine-tuning of language models with zero-init attention. In ICLR

    2024

  52. [60]

    Shaolei Zhang, Qingkai Fang, Zhe Yang, and Yang Feng. 2025. https://arxiv.org/abs/2501.03895 Llava-mini: Efficient image and video large multimodal models with one vision token . Preprint, arXiv:2501.03895

    2025 arXiv

  53. [61]

    P Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. https://arxiv.org/abs/2306.05685 Judging llm-as-a-judge with mt-bench and chatbot arena . P...

    2023 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.