Pith. sign in

REVIEW 3 major objections 6 minor 92 references

DisCo: Towards Distinct and Coherent Visual Encapsulation in Video MLLMs

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A plug-and-play visual connector that pairs each video token with a distinct concept and tracks that concept across every frame yields consistent accuracy gains on video understanding benchmarks while using about 75% fewer tokens.

desk verdict A plausible plug-and-play video connector whose controlled ablations are the real contribution, but the SOTA and temporal-coherence claims need more evidence before they're taken at face value. read the letter →

arxiv 2507.10302 v1 pith:MJ72K4A2 submitted 2025-07-14 cs.CV

Jiahe Zhao , Rongkun Zheng , Yi Wang , Helin Wang , Hengshuang Zhao This is my paper
classification cs.CV
keywords videomultimodallargelanguagemodelsvisualencapsulationresamplersemanticdistinctnesstemporalcoherencecontrastivelearningtokenefficiencyquestionanswering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Video multimodal LLMs convert a video into a small set of visual tokens for the language model, and the quality of that conversion limits video understanding. This paper argues that existing connectors, whether linear projectors or resamplers, produce tokens that are semantically redundant—several tokens attend the same object while others are ignored—and temporally unstable, losing track of an object in some frames. DisCo instead supervises each token to represent one distinct concept, extracted from the caption as a noun phrase, and to keep attending to that concept in every frame. On ST-LLM and InternVideo2, DisCo consistently improves accuracy on MVBench, STAR, PerceptionTest, EgoSchema, MLVU, VideoMME, and video conversation benchmarks, and a 16-token DisCo outperforms a 64-token resampler. If correct, this points to a new design principle for visual connectors: token quality, not token count, is what carries video understanding.

What carries the argument

The machinery is the resampler's cross-attention mechanism treated as a readable, steerable signal. DisCo writes into it with two auxiliary losses: the Visual Concept Discriminator (VCD) uses bipartite matching between visual-token groups and caption-derived noun phrases, then applies contrastive and matching losses to give each token group distinct semantics; the Temporal Focus Calibrator (TFC) extracts frame-level attended features from the cross-attention maps and aligns each token's per-frame features to their centroid with the Frame-level Focus Alignment (FFA) loss. The same cross-attention maps that produce the tokens are thus used as the training signal to make tokens distinct and temporally consistent.

What would settle it

Measure attention faithfulness directly: on a video dataset with instance masks (e.g., DAVIS or a video instance-segmentation benchmark), locate the token assigned to a known object by the VCD matching, and compute the overlap of that token's per-frame cross-attention maps with the ground-truth object mask across all frames. If attention frequently falls on background or loses the object when it is occluded or absent, DisCo's temporal-coherence claim is measurably false.

Watch

Extended reading notes

Core claim

DisCo claims that the deficiencies of resampler-based video tokens—semantic indistinctness, where multiple tokens pile onto the same instance and other instances get no token, and temporal incoherence, where a token attends its instance in some frames but drops it in others—can be directly supervised away. The Visual Concept Discriminator extracts non-repetitive noun phrases from the video caption with GPT-4, partitions visual tokens into groups, matches groups to phrases by Hungarian bipartite matching, and trains with pairwise contrastive and matching losses so each token group locks onto its phrase. The Temporal Focus Calibrator takes the resampler's per-frame cross-attention maps, builds frame-wise attended features for each token, and applies a Frame-level Focus Alignment loss that pulls every frame's feature toward the token's temporal centroid and repels other tokens' features, so each token keeps its instance in view throughout the video. A set of global tokens preserves information the phrases miss. The reported result is that these two modules, added as a plug-in to existing resamplers, raise accuracy across a wide spectrum of video QA and conversation benchmarks while allowing token counts to drop by about 75% without losing performance.

Load-bearing premise

The Temporal Focus Calibrator assumes that a visual token's cross-attention map is a faithful measure of which instance the token represents; if attention is diffuse or glued to background, the frame-alignment loss will reinforce the wrong focus instead of tracking the intended object.

Editorial extensions

If this is right

  • Plug-in gains: DisCo raises accuracy on six video QA benchmarks and the VideoChatGPT conversation benchmark when added to ST-LLM and InternVideo2, with differences beyond the SFT control that uses the same data.
  • Token efficiency: a 16-token DisCo still outperforms a standard 64-token resampler on MVBench, implying an effective 75% reduction in visual tokens at similar or better accuracy.
  • Generality across captions: gains persist with both long dense captions (ShareGPT4o) and short captions (WebVid2M), though the token configuration should be adapted to the caption's instance count.
  • Both losses matter: ablations show VCD and TFC each contribute, and dropping either the visual-semantic contrastive loss or the matching loss reduces MVBench accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The frame-level attention-alignment idea could generalize to image MLLMs as an object-centric token regularizer, encouraging each token to track an entity across spatial regions rather than across time.
  • If the 75% token reduction holds at scale, inference cost of video MLLMs could shrink substantially, and model capacity could be redirected from redundant tokens to more concepts (more token groups) rather than more tokens per group.
  • The method's ceiling is set by caption quality: since concepts come from captions via GPT-4, objects never mentioned in captions will not receive dedicated tokens; testing on open-vocabulary video QA with unseen objects would stress this boundary.
  • The similarity-matrix and attention visualizations suggest a cheap diagnostic: token-to-concept similarity entropy could be used as a monitoring metric during training to detect when semantic distinctness collapses, without needing full benchmark evaluation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes DisCo, a plug-and-play visual encapsulation module for video MLLMs, with two components: a Visual Concept Discriminator (VCD), which aligns groups of visual tokens with distinct semantic concepts extracted from captions via GPT-4 and trained with contrastive/matching losses, and a Temporal Focus Calibrator (TFC), which uses a frame-level focus alignment (FFA) loss on cross-attention features to encourage each visual token to keep tracking its instance across frames. DisCo is applied to ST-LLM and InternVideo2 (including the HD variant), and experiments report consistent gains on MVBench, STAR, PerceptionTest, EgoSchema, MLVU, VideoMME and VideoChatGPT-Bench, along with token-efficiency and ablation studies.

Significance. If the reported effects hold, DisCo is a useful and unusually well-scoped contribution: it is plug-and-play, works across two base models, and its ablation table compares against an SFT baseline trained on the same corpus, which is stronger than the common practice of comparing only to the published baseline. The token-efficiency result, where a 16-token DisCo outperforms a 64-token resampler, is practically relevant. The manuscript also ships code and gives detailed hyperparameters. The main caveats are that the temporal-coherence mechanism is validated only qualitatively and that the headline state-of-the-art claims rest on uncontrolled comparisons.

major comments (3)
  1. [Sec. 3.3, Eqs. (5)-(6)] The TFC module's only training signal is the FFA loss, which treats the pooled cross-attention feature a_t_i as a faithful summary of the instance that token i is tracking. The paper does not provide quantitative evidence for this proxy. Fig. 6(b) shows one qualitative 'pot' example, and downstream accuracy (Tab. 6) is an indirect and insensitive check, because a model could gain from the auxiliary loss even if the attention maps do not actually track instances. If the attention maps are diffuse or background-dominated, FFA can reinforce a wrong focus, and the claimed temporal-coherence attribute is not established. Please add a quantitative validation of the proxy, such as measuring attention-map overlap with ground-truth instance tracks or masks on a subset, or an intervention that isolates the FFA mechanism, and report the distribution rather than a single example.
  2. [Sec. 4.1, Tables 2-3] The 'state-of-the-art' comparisons are not controlled: different rows use different visual encoders, LLMs, training corpora, token counts, and input resolutions, so the reported margins over prior systems cannot be attributed to the proposed module. The paper's own SFT baseline in Table 4 is the right control, but it is reported for only one run per condition and without standard deviations or significance tests. Several numeric differences in Tables 2 and 12 (e.g., MVBench 68.2 vs. 68.1, VideoMME 47.4 vs. 46.3) are within plausible run-to-run noise. Please add multiple seeds or a statistical statement, and present 'outperforms previous state-of-the-art' claims only with a controlled comparison where the sole change is the proposed method.
  3. [Sec. 4.2, Fig. 5 and Table 12] The token-efficiency claim needs clarification and stronger controls. Fig. 5 varies 'local visual tokens' but does not state how the number of global tokens and the VCD group count Ng change when the total drops from 64 to 16, and the x-axis label 'Step time (s/step)' measures training time, while the abstract's '75% fewer tokens' refers to input-token count. Table 12 compares DisCo (64 tokens) with TokenPacker and DeCo using different underlying models and training procedures. Please make the comparison setup explicit (same base model, same data, same token-count convention) or restrict the efficiency claim to a single controlled setting.
minor comments (6)
  1. [Sec. 3.2, Eq. (2)] The text and equations refer to 'i-th visual token' while the actual matching unit is a group of N/Ng tokens; please clarify how group-level features are pooled and whether each token within a group receives the same concept label.
  2. [Eq. (3)] The contrastive temperature tau is introduced but its value is never reported in the implementation details; please specify it.
  3. [Table 2] EgoSchema is evaluated on a 'subset' and MLVU on a 'subset'; please state the subset sizes and cite the convention used by prior work.
  4. [Appendix A and Tables 5, 11] There are several typos: 'basd' should be 'based', Table 5 uses 'Lvtc/Lvtm' where the text uses 'Lvsc/Lvsm', and the Table 11 caption uses 'λ_fsc' where Eq. (7) defines 'λ_ffa'.
  5. [Fig. 5] The x-axis is labeled 'Step time (s/step)' while the caption says the figure reports 'Performance and efficiency under different number of visual tokens'; please clarify which quantity is on each axis and report inference-time variation as well as training time.
  6. [Sec. 3.4] Stage 1 mixes 900K video dense captions with 23K image captions; please state how image-only data is used, for example whether it is batch-level mixing or a separate auxiliary objective, and whether the image data also receives the VCD/TFC losses.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: DisCo's VCD and TFC modules are training objectives supervised by external captions/concepts and evaluated on external benchmarks, so the central claims do not reduce to their inputs.

full rationale

The paper's derivation chain is a training pipeline rather than a formal derivation, and no load-bearing step equates a fitted parameter with a prediction. The VCD module uses GPT-4-extracted semantic concepts from video captions as external text supervision and trains pairwise visual-text alignment with standard VSC/VSM losses (Eqs. 3-4); 'semantic distinctness' is the training objective, and its value is supported by downstream accuracy and ablations (Tabs. 4-5), so it is not a result inferred from its own definition. The TFC module's FFA loss (Eq. 6) directly optimizes temporal consistency of cross-attention features (Eq. 5), which is an explicit design goal rather than a covert prediction; the unvalidated assumption that attention maps faithfully track instances is a correctness risk (the paper gives only one qualitative 'pot' example), not circularity, because the loss could in principle optimize the wrong target and the paper evaluates on external benchmarks. Citations to the authors' prior InternVideo2 work are used as a baseline and codebase, not as a uniqueness theorem or as the sole justification for the central claim; the SFT ablation with the same training corpus shows gains are not forced by data alone. No equation reduces to its own input, and no self-citation chain is invoked to forbid alternatives. Overall, the work is self-contained against external benchmarks and the appropriate finding is no significant circularity.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The ledger is dominated by design hyperparameters chosen on the evaluation benchmarks. The two modules introduce no new physical entities; they are implemented with standard attention, losses, and learned queries. The main assumptions are about the reliability of GPT-4 extraction and the interpretability of cross-attention maps, both of which are plausible but not independently verified.

free parameters (5)
  • Number of visual token groups Ng = 16 for InternVideo2, 12 for ST-LLM, 6 for WebVid2M adaption
    Chosen by ablation on MVBench, STAR, and EgoSchema (Table 5); balances semantic coverage against per-group completeness.
  • Tokens per group N/Ng = 2
    Selected by ablation in Table 5; Ng=64 with one token per group hurts performance.
  • Number of global tokens = 32 of 96 for InternVideo2, 8 of 32 for ST-LLM
    Hand-set in Sec. 4 to preserve unaligned global information; no systematic sweep is reported.
  • Loss weights lambda_vsc, lambda_vsm, lambda_ffa = 1.0, 1.0, 1.0
    Ablated in Table 11; the 1.0 setting gives the best MVBench/STAR/EgoSchema combination.
  • Contrastive temperature tau = not reported
    Appears in Eqs. 3 and 6 but its value or learnability is not specified, leaving an implicit free hyperparameter.
assumptions (4)
  • domain assumption GPT-4 extracted semantic instances from video captions are complete, non-repetitive, and visually grounded.
    VCD relies on these text concepts as supervision; if captions omit salient objects or GPT-4 hallucinates, aligned tokens miss content. Used in Sec. 3.2 and Appendix B.
  • domain assumption Cross-attention maps of resampler tokens are a faithful proxy for the visual instance each token represents.
    TFC uses these maps as the only signal for temporal coherence (Eq. 5); if attention is diffuse or misaligned, FFA loss cannot track instances.
  • domain assumption Pairwise contrastive losses between visual tokens and text embeddings promote semantic distinctness in the final LLM tokens.
    Assumed rather than proven; supported only by ablations. Introduced in Sec. 3.2.
  • standard math Standard math: Hungarian algorithm and contrastive learning formulations are valid.
    Used in Eq. 2 and Eqs. 3/6; no proof needed beyond prior literature.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DisCo: Towards Distinct and Coherent Visual Encapsulation in Video MLLMs." pith.science (2026). https://pith.science/paper/MJ72K4A2

@misc{pith2026250710302,
  author       = {Pith},
  title        = {Pith review of: DisCo: Towards Distinct and Coherent Visual Encapsulation in Video MLLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MJ72K4A2}},
  note         = {Machine review of arXiv:2507.10302}
}
read the original abstract

In video Multimodal Large Language Models (video MLLMs), the visual encapsulation process plays a pivotal role in converting video contents into representative tokens for LLM input. While linear projectors are widely employed for encapsulation, they introduce semantic indistinctness and temporal incoherence when applied to videos. Conversely, the structure of resamplers shows promise in tackling these challenges, but an effective solution remains unexplored. Drawing inspiration from resampler structures, we introduce DisCo, a novel visual encapsulation method designed to yield semantically distinct and temporally coherent visual tokens for video MLLMs. DisCo integrates two key components: (1) A Visual Concept Discriminator (VCD) module, assigning unique semantics for visual tokens by associating them in pair with discriminative concepts in the video. (2) A Temporal Focus Calibrator (TFC) module, ensuring consistent temporal focus of visual tokens to video elements across every video frame. Through extensive experiments on multiple video MLLM frameworks, we demonstrate that DisCo remarkably outperforms previous state-of-the-art methods across a variety of video understanding benchmarks, while also achieving higher token efficiency thanks to the reduction of semantic indistinctness. The code: https://github.com/ZJHTerry18/DisCo.

Figures

Figures reproduced from arXiv: 2507.10302 by the authors.

Figure 1
Figure 1. Illustrations of different visual encapsulation methods [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The overall structure of DisCo. DisCo is highlighted by (i) a Visual Concept Discriminator (VCD) module, which conducts a [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The structure of the TFC module. In TFC, frame-wise [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Qualitative examples of video understanding. Utilizing DisCo, video MLLMs achieve (a)(b) better correctness, (c)(d) stronger [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Performance and efficiency under different number of [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: (a) Similarity matrix between visual tokens and text instances. Resamplers exhibit severe semantic redundancy across tokens, [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: The prompt we used to guide GPT-4 to perform the [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Examples of the semantic instance extraction process. [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

92 extracted references · 36 canonical work pages

  1. [1]

    arXiv:2303.08774 (2023) 4, 6, 9

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.: Gpt-4 technical report. arXiv:2303.08774 (2023) 4, 6, 9

    arXiv 2023

  2. [2]

    In: NeurIPS (2022) 1, 2

    Alayrac, J.B., Donahue, J., Luc, P., Miech, A., Barr, I., Has- son, Y ., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al.: Flamingo: a visual language model for few-shot learn- ing. In: NeurIPS (2022) 1, 2

    2022

  3. [3]

    Ablations on the weights different components in the total training loss of DisCo

    Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., Zhou, J.: Qwen-vl: A versatile vision-language Table 11. Ablations on the weights different components in the total training loss of DisCo. λvsc, λvsm, λf f aindicates weights for the losses in Eq.7. λvsc λvsm λf f a MVBench STAR EgoSchema 0.5 1.0 1.0 66.9 75.5 70.4 2.0 1.0 1.0 68...

    arXiv 2023

  4. [4]

    In: ICCV (2021) 2, 6, 9

    Bain, M., Nagrani, A., Varol, G., Zisserman, A.: Frozen in time: A joint video and image encoder for end-to-end re- trieval. In: ICCV (2021) 2, 6, 9

    2021

  5. [5]

    In: NeurIPS (2020) 2

    Brown, T.B.: Language models are few-shot learners. In: NeurIPS (2020) 2

    2020

  6. [6]

    In: ECCV (2020) 4

    Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with trans- formers. In: ECCV (2020) 4

    2020

  7. [7]

    In: CVPR (2024) 1, 2

    Cha, J., Kang, W., Mun, J., Roh, B.: Honeybee: Locality- enhanced projector for multimodal llm. In: CVPR (2024) 1, 2

    2024

  8. [8]

    arXiv:2410.02768 (2024)

    Chen, J., Ma, K., Huang, H., Shen, J., Fang, H., Zang, X., Ban, C., He, Z., Sun, H., Kang, Y .: Bovila: Bootstrapping video-language alignment via llm-based self-questioning and answering. arXiv:2410.02768 (2024)

    arXiv 2024

Show all 92 references
  1. [9]

    arXiv:2310.09478 (2023) 3

    Chen, J., Zhu, D., Shen, X., Li, X., Liu, Z., Zhang, P., Kr- ishnamoorthi, R., Chandra, V ., Xiong, Y ., Elhoseiny, M.: Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. arXiv:2310.09478 (2023) 3

    2023 arXiv

  2. [10]

    arXiv:2311.12793 (2023) 1

    Chen, L., Li, J., Dong, X., Zhang, P., He, C., Wang, J., Zhao, F., Lin, D.: Sharegpt4v: Improving large multi-modal mod- els with better captions. arXiv:2311.12793 (2023) 1

    2023 arXiv

  3. [11]

    arXiv:2406.04325 (2024) 1, 5

    Chen, L., Wei, X., Li, J., Dong, X., Zhang, P., Zang, Y ., Chen, Z., Duan, H., Lin, B., Tang, Z., et al.: Sharegpt4video: Improving video understanding and generation with better captions. arXiv:2406.04325 (2024) 1, 5

    2024 arXiv

  4. [12]

    I see you, Batman!

    Chen, X., Huang, L., Liu, Y ., Shen, Y ., Zhao, D., Zhao, H.: Anydoor: Zero-shot object-level image customization. In: CVPR (2024) 1 10 Video Caption A person is wearing a vibrant pink scarf wrapped around the neck, with one side draping longer than the other over a long-sleev...

    2024

  5. [13]

    arXiv:2312.06722 (2023) 2

    Chen, Y ., Ge, Y ., Ge, Y ., Ding, M., Li, B., Wang, R., Xu, R., Shan, Y ., Liu, X.: Egoplan-bench: Benchmarking egocentric embodied planning with multimodal large language models. arXiv:2312.06722 (2023) 2

    2023 arXiv

  6. [14]

    In: CVPR (2024) 3

    Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q., Zhu, X., Lu, L., et al.: Internvl: Scal- ing up vision foundation models and aligning for generic visual-linguistic tasks. In: CVPR (2024) 3

    2024

  7. [15]

    arXiv:2406.07476 (2024) 5

    Cheng, Z., Leng, S., Zhang, H., Xin, Y ., Li, X., Chen, G., Zhu, Y ., Zhang, W., Luo, Z., Zhao, D., et al.: Videollama 2: Advancing spatial-temporal modeling and audio understand- ing in video-llms. arXiv:2406.07476 (2024) 5

    2024 arXiv

  8. [16]

    Chiang, W.L., Li, Z., Lin, Z., Sheng, Y ., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y ., Gonzalez, J.E., et al.: Vi- cuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality (2023) 6

    2023

  9. [17]

    arXiv:2402.03766 (2024) 2

    Chu, X., Qiao, L., Zhang, X., Xu, S., Wei, F., Yang, Y ., Sun, X., Hu, Y ., Lin, X., Zhang, B., et al.: Mobilevlm v2: Faster and stronger baseline for vision language model. arXiv:2402.03766 (2024) 2

    2024 arXiv

  10. [18]

    arXiv:1810.04805 (2018) 2

    Devlin, J.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805 (2018) 2

    2018 arXiv

  11. [19]

    arXiv:2010.11929 (2020) 3

    Dosovitskiy, A.: An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv:2010.11929 (2020) 3

    2020 arXiv

  12. [20]

    arXiv:2303.03378 (2023) 1

    Driess, D., Xia, F., Sajjadi, M.S., Lynch, C., Chowdhery, A., Ichter, B., Wa hid, A., Tompson, J., Vuong, Q., Yu, T., et al.: Palm-e: An embodied multimodal language model. arXiv:2303.03378 (2023) 1

    2023 arXiv

  13. [21]

    arXiv:2407.21783 (2024) 2

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al.: The llama 3 herd of models. arXiv:2407.21783 (2024) 2

    2024 arXiv

  14. [22]

    Fan, Y ., Ma, X., Wu, R., Du, Y ., Li, J., Gao, Z., Li, Q.: Videoagent: A memory-augmented multimodal agent for video understanding (2024)

    2024

  15. [23]

    In: CVPR (2023) 6

    Fang, Y ., Wang, W., Xie, B., Sun, Q., Wu, L., Wang, X., Huang, T., Wang, X., Cao, Y .: Eva: Exploring the limits of masked visual representation learning at scale. In: CVPR (2023) 6

    2023

  16. [24]

    arXiv:2405.21075 (2024) 6

    Fu, C., Dai, Y ., Luo, Y ., Li, L., Ren, S., Zhang, R., Wang, Z., Zhou, C., Shen, Y ., Zhang, M., et al.: Video-mme: The first- ever comprehensive evaluation benchmark of multi-modal llms in video analysis. arXiv:2405.21075 (2024) 6

    2024 arXiv

  17. [25]

    arXiv:2307.08041 (2023) 1

    Ge, Y ., Ge, Y ., Zeng, Z., Wang, X., Shan, Y .: Planting a seed of vision in large language model. arXiv:2307.08041 (2023) 1

    2023 arXiv

  18. [26]

    In: ICCV (2017) 6

    Goyal, R., Ebrahimi Kahou, S., Michalski, V ., Materzynska, J., Westphal, S., Kim, H., Haenel, V ., Fruend, I., Yianilos, P., Mueller-Freitag, M., et al.: The” something something” video database for learning and evaluating visual common sense. In: ICCV (2017) 6

    2017

  19. [27]

    In: ICLR (2022) 6

    Hu, E.J., Shen, Y ., Wallis, P., Allen-Zhu, Z., Li, Y ., Wang, S., Wang, L., Chen, W.: Lora: Low-rank adaptation of large language models. In: ICLR (2022) 6

    2022

  20. [28]

    In: CVPR (2019) 2

    Hudson, D.A., Manning, C.D.: Gqa: A new dataset for real- world visual reasoning and compositional question answer- ing. In: CVPR (2019) 2

    2019

  21. [29]

    arXiv:2310.06825 (2023) 6

    Jiang, A.Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D.S., Casas, D.d.l., Bressand, F., Lengyel, G., Lam- ple, G., Saulnier, L., et al.: Mistral 7b. arXiv:2310.06825 (2023) 6

    2023 arXiv

  22. [30]

    In: CVPR (2024) 6

    Jin, P., Takanobu, R., Zhang, W., Cao, X., Yuan, L.: Chat-univi: Unified visual representation empowers large language models with image and video understanding. In: CVPR (2024) 6

    2024

  23. [31]

    arXiv:1705.06950 (2017) 6

    Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijayanarasimhan, S., Viola, F., Green, T., Back, T., Nat- 11 sev, P., et al.: The kinetics human action video dataset. arXiv:1705.06950 (2017) 6

    2017 arXiv

  24. [32]

    In: ECCV (2024) 1, 2

    Korbar, B., Xian, Y ., Tonioni, A., Zisserman, A., Tombari, F.: Text-conditioned resampler for long form video under- standing. In: ECCV (2024) 1, 2

    2024

  25. [33]

    arXiv:2306.05425 (2023) 2

    Li, B., Zhang, Y ., Chen, L., Wang, J., Pu, F., Yang, J., Li, C., Liu, Z.: Mimic-it: Multi-modal in-context instruction tuning. arXiv:2306.05425 (2023) 2

    2023 arXiv

  26. [34]

    arXiv:2305.03726 (2023) 5

    Li, B., Zhang, Y ., Chen, L., Wang, J., Yang, J., Liu, Z.: Ot- ter: a multi-modal model with in-context instruction tuning. arXiv:2305.03726 (2023) 5

    2023 arXiv

  27. [35]

    arXiv:2408.03326 (2024) 2

    Li, B., Zhang, Y ., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Li, Y ., Liu, Z., Li, C.: Llava-onevision: Easy visual task transfer. arXiv:2408.03326 (2024) 2

    2024 arXiv

  28. [36]

    In: ICML (2023) 1, 2, 3

    Li, J., Li, D., Savarese, S., Hoi, S.: Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In: ICML (2023) 1, 2, 3

    2023

  29. [37]

    In: ICML (2022) 4, 6

    Li, J., Li, D., Xiong, C., Hoi, S.: Blip: Bootstrapping language-image pre-training for unified vision-language un- derstanding and generation. In: ICML (2022) 4, 6

    2022

  30. [38]

    arXiv:2305.06355 (2023) 2

    Li, K., He, Y ., Wang, Y ., Li, Y ., Wang, W., Luo, P., Wang, Y ., Wang, L., Qiao, Y .: Videochat: Chat-centric video under- standing. arXiv:2305.06355 (2023) 2

    2023 arXiv

  31. [39]

    In: CVPR (2024) 1, 2, 5, 6

    Li, K., Wang, Y ., He, Y ., Li, Y ., Wang, Y ., Liu, Y ., Wang, Z., Xu, J., Chen, G., Luo, P., et al.: Mvbench: A comprehen- sive multi-modal video understanding benchmark. In: CVPR (2024) 1, 2, 5, 6

    2024

  32. [40]

    arXiv:2405.13911 (2024) 1

    Li, W., Fan, H., Wong, Y ., Kankanhalli, M., Yang, Y .: Topa: Extend large language models for video understanding via text-only pre-alignment. arXiv:2405.13911 (2024) 1

    2024 arXiv

  33. [41]

    arXiv:2407.02392 (2024) 1, 2, 3, 10

    Li, W., Yuan, Y ., Liu, J., Tang, D., Wang, S., Qin, J., Zhu, J., Zhang, L.: Tokenpacker: Efficient visual projector for multi- modal llm. arXiv:2407.02392 (2024) 1, 2, 3, 10

    2024 arXiv

  34. [42]

    In: ECCV (2024) 5, 6

    Li, Y ., Wang, C., Jia, J.: Llama-vid: An image is worth 2 tokens in large language models. In: ECCV (2024) 5, 6

    2024

  35. [43]

    arXiv:2311.10122 (2023) 1, 3, 5

    Lin, B., Ye, Y ., Zhu, B., Cui, J., Ning, M., Jin, P., Yuan, L.: Video-llava: Learning united visual representation by align- ment before projection. arXiv:2311.10122 (2023) 1, 3, 5

    2023 arXiv

  36. [44]

    In: ECCV (2014) 2

    Lin, T.Y ., Maire, M., Belongie, S., Hays, J., Perona, P., Ra- manan, D., Doll ´ar, P., Zitnick, C.L.: Microsoft coco: Com- mon objects in context. In: ECCV (2014) 2

    2014

  37. [45]

    In: CVPR (2024) 1

    Liu, H., Li, C., Li, Y ., Lee, Y .J.: Improved baselines with visual instruction tuning. In: CVPR (2024) 1

    2024

  38. [46]

    In: NeurIPS (2023) 1, 2, 3, 6

    Liu, H., Li, C., Wu, Q., Lee, Y .J.: Visual instruction tuning. In: NeurIPS (2023) 1, 2, 3, 6

    2023

  39. [47]

    In: ECCV (2024) 5, 6, 9

    Liu, R., Li, C., Tang, H., Ge, Y ., Shan, Y ., Li, G.: St-llm: Large language models are effective temporal learners. In: ECCV (2024) 5, 6, 9

    2024

  40. [48]

    In: NeurIPS (2020) 3

    Locatello, F., Weissenborn, D., Unterthiner, T., Mahen- dran, A., Heigold, G., Uszkoreit, J., Dosovitskiy, A., Kipf, T.: Object-centric learning with slot attention. In: NeurIPS (2020) 3

    2020

  41. [49]

    In: ICML (2023) 2

    Longpre, S., Hou, L., Vu, T., Webson, A., Chung, H.W., Tay, Y ., Zhou, D., Le, Q.V ., Zoph, B., Wei, J., et al.: The flan col- lection: Designing data and methods for effective instruction tuning. In: ICML (2023) 2

    2023

  42. [50]

    arXiv:2406.09418 (2024) 1

    Maaz, M., Rasheed, H., Khan, S., Khan, F.: Videogpt+: In- tegrating image and video encoders for enhanced video un- derstanding. arXiv:2406.09418 (2024) 1

    2024 arXiv

  43. [51]

    arXiv:2306.05424 (2023) 2, 6

    Maaz, M., Rasheed, H., Khan, S., Khan, F.S.: Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv:2306.05424 (2023) 2, 6

    2023 arXiv

  44. [52]

    In: NeurIPS (2023) 6

    Mangalam, K., Akshulakov, R., Malik, J.: Egoschema: A di- agnostic benchmark for very long-form video language un- derstanding. In: NeurIPS (2023) 6

    2023

  45. [53]

    In: NeurIPS (2024) 1

    Mu, Y ., Zhang, Q., Hu, M., Wang, W., Ding, M., Jin, J., Wang, B., Dai, J., Qiao, Y ., Luo, P.: Embodiedgpt: Vision- language pre-training via embodied chain of thought. In: NeurIPS (2024) 1

    2024

  46. [54]

    In: NeurIPS (2024) 6

    Patraucean, V ., Smaira, L., Gupta, A., Recasens, A., Mar- keeva, L., Banarse, D., Koppula, S., Malinowski, M., Yang, Y ., Doersch, C., et al.: Perception test: A diagnostic bench- mark for multimodal video models. In: NeurIPS (2024) 6

    2024

  47. [55]

    In: ICCV (2015) 2

    Plummer, B.A., Wang, L., Cervantes, C.M., Caicedo, J.C., Hockenmaier, J., Lazebnik, S.: Flickr30k entities: Collect- ing region-to-phrase correspondences for richer image-to- sentence models. In: ICCV (2015) 2

    2015

  48. [56]

    arXiv:2405.16009 (2024) 1

    Qian, R., Dong, X., Zhang, P., Zang, Y ., Ding, S., Lin, D., Wang, J.: Streaming long video understanding with large language models. arXiv:2405.16009 (2024) 1

    2024 arXiv

  49. [57]

    In: ICML (2021) 4

    Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural lan- guage supervision. In: ICML (2021) 4

    2021

  50. [58]

    arXiv:2307.09288 (2023)

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y ., Bashlykov, N., Batra, S., Bhargava, P., Bhos- ale, S., et al.: Llama 2: Open foundation and fine-tuned chat models. arXiv:2307.09288 (2023)

    2023 arXiv

  51. [59]

    In: NeurIPS (2017) 3

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. In: NeurIPS (2017) 3

    2017

  52. [60]

    arXiv:2409.12191 (2024) 2

    Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., et al.: Qwen2-vl: Enhanc- ing vision-language model’s perception of the world at any resolution. arXiv:2409.12191 (2024) 2

    2024 arXiv

  53. [61]

    In: NeurIPS (2024) 3

    Wang, W., Chen, Z., Chen, X., Wu, J., Zhu, X., Zeng, G., Luo, P., Lu, T., Zhou, J., Qiao, Y ., et al.: Visionllm: Large language model is also an open-ended decoder for vision- centric tasks. In: NeurIPS (2024) 3

    2024

  54. [62]

    arXiv:2409.02889 (2024) 5

    Wang, X., Song, D., Chen, S., Zhang, C., Wang, B.: Longllava: Scaling multi-modal llms to 1000 images effi- ciently via a hybrid architecture. arXiv:2409.02889 (2024) 5

    2024

  55. [63]

    In: ECCV (2024) 1, 2, 5, 6, 9

    Wang, Y ., Li, K., Li, X., Yu, J., He, Y ., Chen, G., Pei, B., Zheng, R., Xu, J., Wang, Z., et al.: Internvideo2: Scaling video foundation models for multimodal video understand- ing. In: ECCV (2024) 1, 2, 5, 6, 9

    2024

  56. [64]

    In: NeurIPS (2021) 2, 6

    Wu, B., Yu, S., Chen, Z., Tenenbaum, J.B., Gan, C.: Star: A benchmark for situated reasoning in real-world videos. In: NeurIPS (2021) 2, 6

    2021

  57. [65]

    In: CVPR (2021) 6 12

    Xiao, J., Shang, X., Yao, A., Chua, T.S.: Next-qa: Next phase of question-answering to explaining temporal actions. In: CVPR (2021) 6 12

    2021

  58. [66]

    arXiv:2402.13088 (2024) 3

    Xu, J., Lan, C., Xie, W., Chen, X., Lu, Y .: Slot-vlm: Slowfast slots for video-language modeling. arXiv:2402.13088 (2024) 3

    2024 arXiv

  59. [67]

    In: CVPR (2016) 2

    Xu, J., Mei, T., Yao, T., Rui, Y .: Msr-vtt: A large video de- scription dataset for bridging video and language. In: CVPR (2016) 2

    2016

  60. [68]

    arXiv:2404.16994 (2024) 1, 3

    Xu, L., Zhao, Y ., Zhou, D., Lin, Z., Ng, S.K., Feng, J.: Pllava: Parameter-free llava extension from images to videos for video dense captioning. arXiv:2404.16994 (2024) 1, 3

    2024 arXiv

  61. [69]

    RAL (2024) 1

    Xu, Z., Zhang, Y ., Xie, E., Zhao, Z., Guo, Y ., Wong, K.Y .K., Li, Z., Zhao, H.: Drivegpt4: Interpretable end-to-end au- tonomous driving via large language model. RAL (2024) 1

    2024

  62. [70]

    arXiv:2405.20985 (2024) 3, 10

    Yao, L., Li, L., Ren, S., Wang, L., Liu, Y ., Sun, X., Hou, L.: Deco: Decoupling token compression from se- mantic abstraction in multimodal large language models. arXiv:2405.20985 (2024) 3, 10

    2024 arXiv

  63. [71]

    arXiv:2304.14178 (2023)

    Ye, Q., Xu, H., Xu, G., Ye, J., Yan, M., Zhou, Y ., Wang, J., Hu, A., Shi, P., Shi, Y ., et al.: mplug-owl: Modulariza- tion empowers large language models with multimodality. arXiv:2304.14178 (2023)

    2023 arXiv

  64. [72]

    In: CVPR (2024) 1, 3

    Ye, Q., Xu, H., Ye, J., Yan, M., Hu, A., Liu, H., Qian, Q., Zhang, J., Huang, F.: mplug-owl2: Revolutionizing multi- modal large language model with modality collaboration. In: CVPR (2024) 1, 3

    2024

  65. [73]

    In: ICLR (2020) 6

    Yi, K., Gan, C., Li, Y ., Kohli, P., Wu, J., Torralba, A., Tenen- baum, J.B.: Clevrer: Collision events for video representa- tion and reasoning. In: ICLR (2020) 6

    2020

  66. [74]

    In: ICLR (2024) 2

    You, H., Zhang, H., Gan, Z., Du, X., Zhang, B., Wang, Z., Cao, L., Chang, S.F., Yang, Y .: Ferret: Refer and ground anything anywhere at any granularity. In: ICLR (2024) 2

    2024

  67. [75]

    In: ECCV (2016) 2

    Yu, L., Poirson, P., Yang, S., Berg, A.C., Berg, T.L.: Model- ing context in referring expressions. In: ECCV (2016) 2

    2016

  68. [76]

    In: CVPR (2023) 2

    Yu, Z., Zheng, L., Zhao, Z., Wu, F., Fan, J., Ren, K., Yu, J.: Anetqa: A large-scale benchmark for fine-grained composi- tional reasoning over untrimmed videos. In: CVPR (2023) 2

    2023

  69. [77]

    arXiv:2306.02858 (2023) 2, 3, 5, 6

    Zhang, H., Li, X., Bing, L.: Video-llama: An instruction- tuned audio-visual language model for video understanding. arXiv:2306.02858 (2023) 2, 3, 5, 6

    2023 arXiv

  70. [78]

    arXiv:2407.03320 (2024) 2

    Zhang, P., Dong, X., Zang, Y ., Cao, Y ., Qian, R., Chen, L., Guo, Q., Duan, H., Wang, B., Ouyang, L., et al.: Internlm- xcomposer-2.5: A versatile large vision language model sup- porting long-contextual input and output. arXiv:2407.03320 (2024) 2

    2024 arXiv

  71. [79]

    arXiv:2303.16199 (2023)

    Zhang, R., Han, J., Liu, C., Gao, P., Zhou, A., Hu, X., Yan, S., Lu, P., Li, H., Qiao, Y .: Llama-adapter: Effi- cient fine-tuning of language models with zero-init attention. arXiv:2303.16199 (2023)

    2023 arXiv

  72. [80]

    arXiv:2404.01258 (2024) 6, 9

    Zhang, R., Gui, L., Sun, Z., Feng, Y ., Xu, K., Zhang, Y ., Fu, D., Li, C., Hauptmann, A., Bisk, Y ., et al.: Direct pref- erence optimization of video large multimodal models from language model reward. arXiv:2404.01258 (2024) 6, 9

    2024 arXiv

  73. [81]

    In: ICLR (2025) 5, 6

    Zhang, S., Fang, Q., Yang, Z., Feng, Y .: Llava-mini: Ef- ficient image and video large multimodal models with one vision token. In: ICLR (2025) 5, 6

    2025

  74. [82]

    Zhang, Y ., Li, B., Liu, H., Lee, Y .J., Gui, L., Fu, D., Feng, J., Liu, Z., Li, C.: Llava-next: A strong zero-shot video under- standing model (2024) 5

    2024

  75. [83]

    arXiv:2410.02713 (2024)

    Zhang, Y ., Wu, J., Li, W., Li, B., Ma, Z., Liu, Z., Li, C.: Video instruction tuning with synthetic data. arXiv:2410.02713 (2024)

    2024 arXiv

  76. [84]

    arXiv:2309.07915 (2023) 2

    Zhao, H., Cai, Z., Si, S., Ma, X., An, K., Chen, L., Liu, Z., Wang, S., Han, W., Chang, B.: Mmicl: Empowering vision-language model with multi-modal in-context learn- ing. arXiv:2309.07915 (2023) 2

    2023 arXiv

  77. [85]

    Zhao, Q., Wang, S., Zhang, C., Fu, C., Do, M.Q., Agarwal, N., Lee, K., Sun, C.: Antgpt: Can large language models help long-term action anticipation from videos? In: ICLR (2024) 2

    2024

  78. [86]

    In: NeurIPS (2023) 2

    Zheng, L., Chiang, W.L., Sheng, Y ., Zhuang, S., Wu, Z., Zhuang, Y ., Lin, Z., Li, Z., Li, D., Xing, E., et al.: Judg- ing llm-as-a-judge with mt-bench and chatbot arena. In: NeurIPS (2023) 2

    2023

  79. [87]

    In: NeurIPS (2024) 1

    Zheng, R., Qi, L., Chen, X., Wang, Y ., Wang, K., Qiao, Y ., Zhao, H.: Syncvis: Synchronized video instance segmenta- tion. In: NeurIPS (2024) 1

    2024

  80. [88]

    arXiv preprint arXiv:2407.14500 (2024) 2

    Zheng, R., Qi, L., Chen, X., Wang, Y ., Wang, K., Qiao, Y ., Zhao, H.: Villa: Video reasoning segmentation with large language model. arXiv preprint arXiv:2407.14500 (2024) 2

    2024 arXiv

  81. [89]

    arXiv:2406.04264 (2024) 6

    Zhou, J., Shu, Y ., Zhao, B., Wu, B., Xiao, S., Yang, X., Xiong, Y ., Zhang, B., Huang, T., Liu, Z.: Mlvu: A compre- hensive benchmark for multi-task long video understanding. arXiv:2406.04264 (2024) 6

    2024 arXiv

  82. [90]

    In: AAAI (2018) 2

    Zhou, L., Xu, C., Corso, J.: Towards automatic learning of procedures from web instructional videos. In: AAAI (2018) 2

    2018

  83. [91]

    arXiv:2304.10592 (2023) 1

    Zhu, D., Chen, J., Shen, X., Li, X., Elhoseiny, M.: Minigpt- 4: Enhancing vision-language understanding with advanced large language models. arXiv:2304.10592 (2023) 1

    2023 arXiv

  84. [92]

    arXiv:2312.09251 (2023) 2 13

    Zhu, J., Ding, X., Ge, Y ., Ge, Y ., Zhao, S., Zhao, H., Wang, X., Shan, Y .: Vl-gpt: A generative pre-trained trans- former for vision and language understanding and genera- tion. arXiv:2312.09251 (2023) 2 13

    2023 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.