Pith. sign in

REVIEW 4 major objections 5 minor 73 references

Top-Down Compression: Revisit Efficient Vision Token Projection for Visual Instruction Tuning

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LLaVA-Meteor cuts visual tokens by up to 95% without losing accuracy

desk verdict A credible efficiency-focused compression paper with consistent gains over LLaVA-UHD; the instruction-aware selection claim is indirect, but the method itself is solid and worth refereeing. read the letter →

arxiv 2505.11945 v2 pith:DKEMUPZY submitted 2025-05-17 cs.CV

Bonan li , Zicheng Zhang , Songhua Liu , Weihao Yu , Xinchao Wang This is my paper · ORCID
classification cs.CV
keywords visualinstructiontuningtokencompressionvision-languagemodelstatespaceselectioninstruction-awarehigh-resolutionimageefficientmultimodalLLM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a new projection design, LLaVA-Meteor, breaks the usual accuracy-versus-efficiency trade-off in visual instruction tuning by compressing visual tokens before they enter a large language model. It first runs a cheap global fusion over all tokens using a selective state space operator and a local-to-single scan, then selects a small set of the most important tokens based on both visual saliency and instruction-related cues. Across 12 benchmarks, LLaVA-Meteor matches or exceeds the LLaVA-UHD baseline while using 75–95% fewer tokens: at 144 tokens per sub-image it averages 62.4 versus 60.4, and even at 32 tokens it stays within about one point.

What carries the argument

The load-bearing object is the Top-Down Compression paradigm, composed of Flash Global Fusion (FGF) and Visual-Native Selection (VNS). FGF uses a bidirectional selective state space operator with a local-to-single scanning strategy that summarizes each token's 3×3 spatial neighborhood before global propagation, and it inserts one learnable instruction token in the center of each sub-image's sequence to absorb instruction-related cues. VNS then computes a visual importance score from normalized class-token attention and a native importance score from softmax-normalized dot-product similarity to the instruction token, aggregates them as λ·visual + (1-λ)·native, and selects the Top-K tokens by that fused score.

What would settle it

Take the same model and compare the Top-K token sets selected on one image under two different questions; if the native expert carries instruction information, the selected sets should diverge toward regions relevant to each question. If the sets are nearly identical across questions, the instruction token is not instruction-aware and the gains come from the fusion module or extra parameters.

Watch

Extended reading notes

Core claim

The central claim is that a two-stage Top-Down Compression – global fusion before local selection – lets a vision-language model keep essentially all the accuracy of a dense token input at a fraction of the compute. The fusion stage enriches every token with holistic context and distills instruction-relevant cues into a shared learnable instruction token; the selection stage then scores each token twice, once from the frozen vision encoder's class-token attention and once from similarity to the instruction token, and merges the scores with λ = 0.8. Because selection is applied to enriched tokens rather than raw features, the retained subset carries both global semantic context and task-aware emphasis, which the experiments show is enough to beat the uncompressed baseline at the same token budget and remain competitive at extreme compression.

Load-bearing premise

The claim that instruction-aware selection improves accuracy rests on the assumption that the learnable instruction token, trained only through the vision-language loss and never given the explicit question, actually learns instruction-relevant cues rather than acting as a generic image summary.

Editorial extensions

If this is right

  • At the same 144-token budget as LLaVA-UHD, LLaVA-Meteor improves the average across 12 benchmarks by 2.0 points, so token reduction need not cost accuracy.
  • Dropping to 64 tokens loses only about 1.4 points relative to the 144-token version, and 32 tokens stays within 1.1 points of LLaVA-UHD, enabling roughly 95% compression for resource-limited deployment.
  • The projector uses 37.08 million trainable parameters versus 137.84 million in the LLaVA-UHD projector, and throughput in tokens per second rises as the number of retained tokens falls.
  • The dual-expert selection helps most on reasoning and attribute-discrimination tasks, while the visual expert alone is nearly sufficient for dense prediction tasks.
  • Because the fusion and selection steps are separable, the same Top-Down Compression recipe could be dropped into other high-resolution vision-language frameworks that currently use heavier fusion or saliency-only pruning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The native expert's contribution likely depends on the diversity of instruction patterns in the training mixture: on counting and color questions the visual expert should dominate, while OCR and attribute questions should shift λ toward the native expert; this is a testable tuning axis the paper leaves implicit.
  • Because the instruction token never sees the explicit question text, its 'instruction preference' is really a learned prior over common user attention patterns; a natural extension is to condition the token on the actual instruction embedding to make selection genuinely query-dependent.
  • The local-to-single scan is a cheap spatial-locality plug-in that other token-pruning pipelines could adopt, and the paper itself notes the neighborhood window can grow to 5×5 or 7×7 on higher-resolution feature maps for multi-scale context.
  • If the native expert truly encodes instruction-relevant cues, the same fusion-then-select pipeline should transfer to video by sharing the instruction token across frames and selecting spatiotemporally important tokens.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes LLaVA-Meteor, a vision-language model built on LLaVA-UHD, in which a Top-Down Compression pipeline replaces the standard projector. The pipeline has two stages: Flash Global Fusion (FGF), a lightweight SSM-based module with a local-to-single scanning pattern and a shared learnable instruction token (INS), and Visual-Native Selection (VNS), which scores tokens using both a CLS-attention-based visual expert and an INS-similarity-based native expert, then keeps the Top-K tokens per sub-image for the LLM. Experiments across 12 benchmarks report that reducing visual tokens by 75--95% yields results comparable to or better than LLaVA-UHD, including a +2.0 average gain at 144 tokens and only a 1.1 average drop at 32 tokens. The paper also reports a lightweight projector (37M parameters) and higher token throughput relative to LLaVA-UHD.

Significance. If the reported results hold, the proposed architecture is practically relevant: it offers a nontrivial accuracy/efficiency trade-off, strong benchmark coverage, and a simple fusion/selection decomposition that could be reused by other efficient VLM pipelines. The local-to-single scanning idea is a straightforward and plausible improvement over plain Mamba scanning. However, the paper's central novelty is the claim that the native expert makes selection instruction-aware. That claim is not supported by the training signal: the discrete Top-K step prevents gradients from flowing to the native scores, and the instruction token is shared across images and prompt-independent. The reported gains could therefore arise from a generic learned query rather than from instruction-conditioned selection. The lack of released code, model weights, or multi-seed statistics further weakens confidence in the headline comparisons.

major comments (4)
  1. [Section 3.3, Eqs. (4)--(6)] The Top-K selection is non-differentiable, so the native scores NS^i and the aggregation AS^i never receive a direct gradient from the language-model loss. The instruction token INS is trained only through its influence on the fused features F^i inside FGF, and it is shared across all images and not conditioned on the text prompt (as the paper acknowledges in Section 1). Consequently, the claim that the native expert "captures instruction preferences" and that Visual-Native Selection is "instruction-aware" is not justified by the training procedure. The +1.3 average gain of Visual-Native over Visual-only in Table 3 could come from the extra INS parameters, from the FGF pathway, or from a generic learned query rather than from instruction-conditioned token selection. Please provide direct evidence of instruction dependence: e.g., show that selected token subsets change with different questions on the same image, or compare against a version with a randomly initialized and frozen INS, or replace the top-K with a differentiable relaxation that provides supervision to the scores.
  2. [Section 4.4 and Table 4] The text accompanying Table 4 describes Visual-Native Selection as "training-free," but Section 3.2 defines INS as a "shared learnable instruction token" optimized in both pretraining and instruction tuning, and the native expert score is computed from this token. This is a direct contradiction. If the selection scores are never updated by any loss, the authors should state this explicitly and explain how the native expert is supposed to acquire instruction knowledge; if they are updated indirectly through FGF, then calling the module "training-free" is misleading. The efficiency comparison and the parameter-count argument depend on this distinction, so it needs to be resolved.
  3. [Tables 1--4 and Section 4.3] The central empirical claims rest on single runs without released code, model weights, or evaluation scripts. Several of the key deltas are small: for example, in Table 1 many individual benchmarks differ by only 0.1--0.7 points between LLaVA-Meteor and LLaVA-UHD, and the +1.3 gain in Table 3 is driven by a few benchmarks (MMB, MMVet, POPE, SEED) while other benchmarks are nearly unchanged. Without seed variance or error bars, the reader cannot tell whether the headline +2.0 average is a real effect or noise. Efficiency is reported only as TPS and projector parameters; actual inference latency and memory usage for the LLM are not reported, although those are the quantities the token reduction is supposed to improve. Please report multi-seed statistics, release the artifacts, and add end-to-end latency/memory measurements.
  4. [Abstract and Section 4.3] The claim of 95% token compression with "comparable or superior" performance is overstated for the 32-token configuration. In Table 1, the 32-token row is on average 1.1 points lower than LLaVA-UHD, and it drops by 2.7 points on TextVQA and 4.2 points on DocVQA, both of which are text-oriented benchmarks where compression is most consequential. This is not "comparable" in the strong sense implied by the abstract. The authors should either soften the claim to describe the accuracy-efficiency trade-off explicitly or provide additional evidence, such as task-specific analysis, that the losses are acceptable.
minor comments (5)
  1. [Section 3.2] The word "ouputs" appears in the sentence after Eq. (2); it should be "outputs."
  2. [Section 4.3] The phrase "144 tokens for on sub-images" appears to contain a typo; it should probably be "144 tokens for one sub-image."
  3. [Section 3.3, Eq. (3)] The summation notation in Eq. (3) is corrupted ("PHuWu" appears where a summation symbol is intended); please fix the formula so that the normalization is unambiguous.
  4. [Tables 1 and 4] The token counts in Table 1 ("~256," "~114," "~56") and Table 4 ("144," "64," "32") are confusing at first glance because the former are average totals across sub-images while the latter are per-sub-image counts. Please clarify this in the table captions and refer to the same quantity consistently in the text.
  5. [Section 3.3] The progressive weighting scheme for the native expert (linearly increasing its contribution from 0 to 1-lambda over early training) is mentioned but never ablated or analyzed. Since this is a training-dynamics choice, a small ablation or a sentence explaining its effect would strengthen the paper.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's central claims are empirical benchmark results with fixed hyperparameters, not fitted quantities renamed as predictions.

full rationale

The paper makes no first-principles prediction that could reduce to its inputs by construction. Its headline claims are benchmark measurements (Tables 1-4) obtained with a fixed token budget and a fixed default lambda=0.8, which is described as a prior choice rather than a per-dataset fitted value. The Visual-Native Selection mechanism is an architectural design, and the claim that the native expert is instruction-aware is an empirical hypothesis, not a derivation: the paper explicitly concedes that the instruction token is not conditioned on explicit prompts and is only weakly supervised, which raises construct-validity concerns but not circularity. The only self-citation is the MM-Vet benchmark reference [67], co-authored by members of this paper, but MM-Vet is an externally frozen evaluation benchmark and is not used as a load-bearing theoretical premise. There is no imported uniqueness theorem, no ansatz smuggled in via citation, and no renaming of a known result that replaces derivation. Accordingly, the appropriate finding is no significant circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 1 invented entities

The central claim depends on three hand-set hyperparameters (lambda, Top-K, window size) and on the assumption that CLIP class attention and a weakly supervised instruction token capture complementary importance signals. The instruction token is the only genuinely new entity, and it has no independent evidence beyond the paper's own ablations. The lack of code and the single-backbone evaluation make it hard to separate the contribution of the fusion module, the selection module, and the extra trained parameters.

free parameters (3)
  • lambda (expert weight) = 0.8
    Section 3.3, Equation 5. Set by hand as the default after claiming it yields favorable performance across datasets. The paper itself suggests dataset-specific tuning may improve results, so the reported scores depend on this choice.
  • Top-K token budget per sub-image = 144, 64, 32 (per sub-image)
    Section 4.3 and Table 1. The paper reports three operating points, but no analysis showing how the optimal K depends on image resolution, dataset, or aspect ratio. The claimed 75 to 95 percent reduction is defined relative to these chosen budgets.
  • Local window size for local-to-single scanning = 3x3
    Section 3.2. The paper recommends larger windows for high-resolution feature maps but does not tune or evaluate this, so the default window size is an unvalidated choice.
assumptions (4)
  • domain assumption CLIP class-token attention from the frozen vision encoder is a valid proxy for general visual saliency.
    Section 3.3, Visual Expert Scoring. The entire visual expert depends on this, and the paper provides only qualitative visualizations as support.
  • domain assumption A single learnable instruction token, not conditioned on the actual user prompt, can learn instruction-relevant cues from weak supervision.
    Section 1 and Section 3.2. The authors explicitly state the instruction token is not conditioned on explicit prompts and relies on common user attention patterns being reinforced in large-scale data. This is the load-bearing assumption behind the native expert.
  • domain assumption Mamba selective scan SSMs, designed for 1D sequences, can provide sufficient global visual context when combined with the local-to-single windowing.
    Section 3.2. The paper assumes the scanning order plus local windows compensates for the lack of 2D inductive bias, backed only by the ablation in Table 2.
  • domain assumption Evaluating on 12 benchmarks with a single base model (LLaVA-UHD with Vicuna-13B) is sufficient to demonstrate general superiority or parity under token compression.
    Section 4. The comparisons are made against LLaVA-UHD as the primary baseline, with other models included as context. Different token budgets and model sizes could change the conclusion.
invented entities (1)
  • Instruction token (INS)
    purpose: A learnable token inserted into each sub-image sequence to summarize instruction-relevant cues and serve as the native expert in selection.
    It has no falsifiable handle outside this paper. The claim that it encodes common user attention patterns like phone numbers or faces is only supported by qualitative selections in Figure 4.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Top-Down Compression: Revisit Efficient Vision Token Projection for Visual Instruction Tuning." pith.science (2026). https://pith.science/paper/DKEMUPZY

@misc{pith2026250511945,
  author       = {Pith},
  title        = {Pith review of: Top-Down Compression: Revisit Efficient Vision Token Projection for Visual Instruction Tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DKEMUPZY}},
  note         = {Machine review of arXiv:2505.11945}
}
read the original abstract

Visual instruction tuning aims to enable large language models to comprehend the visual world, with a pivotal challenge lying in establishing an effective vision-to-language projection. However, existing methods often grapple with the intractable trade-off between accuracy and efficiency. In this paper, we present LLaVA-Meteor, a novel approach designed to break this deadlock, equipped with a novel Top-Down Compression paradigm that strategically compresses visual tokens without compromising core information. Specifically, we construct a trainable Flash Global Fusion module based on efficient selective state space operators, which aligns the feature space while enabling each token to perceive holistic visual context and instruction preference at low cost. Furthermore, a local-to-single scanning manner is employed to effectively capture local dependencies, thereby enhancing the model's capability in vision modeling. To alleviate computational overhead, we explore a Visual-Native Selection mechanism that independently assesses token significance by both the visual and native experts, followed by aggregation to retain the most critical subset. Extensive experiments show that our approach reduces visual tokens by 75--95% while achieving comparable or superior performance across 12 benchmarks, significantly improving efficiency.

Figures

Figures reproduced from arXiv: 2505.11945 by the authors.

Figure 1
Figure 1. Illustration of visual token compression strategies. (a) Naive and learnable fusion methods [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of the proposed LLaVA-Meteor, which adopts a two-stage Top-Down Compres [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Qualitative comparison of LLaVA-UHD and our LLaVA-Meteor. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Qualitative comparisons for different expert selection strategies. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

73 extracted references · 37 canonical work pages

  1. [1]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. In NeurIPS, pages 23716–23736, 2022

    work page 2022

  2. [2]

    Vita: An efficient video-to-text algorithm using vlm for rag-based video analysis system

    Md Adnan Arefeen, Biplob Debnath, Md Yusuf Sarwar Uddin, and Srimat Chakradhar. Vita: An efficient video-to-text algorithm using vlm for rag-based video analysis system. In CVPR, pages 2266–2274, 2024

    work page 2024

  3. [3]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 1(2):3, 2023

    arXiv 2023

  4. [4]

    Honeybee: Locality- enhanced projector for multimodal llm

    Junbum Cha, Wooyoung Kang, Jonghwan Mun, and Byungseok Roh. Honeybee: Locality- enhanced projector for multimodal llm. In CVPR, pages 13817–13827, 2024

    work page 2024

  5. [5]

    An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models

    Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. In ECCV, pages 19–35, 2024

    work page 2024

  6. [6]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. Science China Information Sciences , 67(12):220101, 2024

    2024

  7. [7]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In CVPR, pages 24185–24198, 2024

    2024

  8. [8]

    Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality

    Wei-Lin Chiang, Zhuohan Li, Ziqing Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023), 2(3):6, 2023

    2023

Show all 73 references
  1. [9]

    Don’t look twice: Faster video transformers with run-length tokenization

    Rohan Choudhury, Guanglei Zhu, Sihan Liu, Koichiro Niinuma, Kris Kitani, and László Jeni. Don’t look twice: Faster video transformers with run-length tokenization. In NeurIPS, pages 28127–28149, 2024

    2024

  2. [10]

    Mobilevlm v2: Faster and stronger baseline for vision language model

    Xiangxiang Chu, Limeng Qiao, Xinyu Zhang, Shuang Xu, Fei Wei, Yang Yang, Xiaofei Sun, Yiming Hu, Xinyang Lin, Bo Zhang, et al. Mobilevlm v2: Faster and stronger baseline for vision language model. arXiv preprint arXiv:2402.03766, 2024

    2024 arXiv

  3. [11]

    Internlm-xcomposer2-4khd: A pioneering large vision-language model handling resolutions from 336 pixels to 4k hd

    Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Songyang Zhang, Haodong Duan, Wenwei Zhang, Yining Li, et al. Internlm-xcomposer2-4khd: A pioneering large vision-language model handling resolutions from 336 pixels to 4k hd. In NeurIPS, pages 42566–42592, 2024

    2024

  4. [12]

    Tc-llava: Rethinking the transfer from image to video understanding with temporal considerations

    Mingze Gao, Jingyu Liu, Mingda Li, Jiangtao Xie, Qingbin Liu, Bo Zhao, Xi Chen, and Hui Xiong. Tc-llava: Rethinking the transfer from image to video understanding with temporal considerations. arXiv preprint arXiv:2409.03206, 2024. 10

    2024 arXiv

  5. [13]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In CVPR, pages 6904–6913, 2017

    2017

  6. [14]

    Mamba: Linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023

    2023 arXiv

  7. [15]

    Efficiently modeling long sequences with structured state spaces

    Albert Gu, Karan Goel, and Christopher Re. Efficiently modeling long sequences with structured state spaces. In ICLR, 2022

    2022

  8. [16]

    Llava-uhd: an lmm perceiving any aspect ratio and high- resolution images

    Zonghao Guo, Ruyi Xu, Yuan Yao, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu, and Gao Huang. Llava-uhd: an lmm perceiving any aspect ratio and high- resolution images. In ECCV, pages 390–406, 2024

    2024

  9. [17]

    Vizwiz grand challenge: Answering visual questions from blind people

    Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In CVPR, pages 3608–3617, 2018

    2018

  10. [18]

    Bliva: A simple multimodal llm for better handling of text-rich visual questions

    Wenbo Hu, Yifan Xu, Yi Li, Weiyue Li, Zeyuan Chen, and Zhuowen Tu. Bliva: A simple multimodal llm for better handling of text-rich visual questions. In AAAI, pages 2256–2264, 2024

    2024

  11. [19]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, pages 6700–6709, 2019

    2019

  12. [20]

    Token compensator: Altering inference cost of vision transformer without re-tuning

    Shibo Jie, Yehui Tang, Jianyuan Guo, Zhi-Hong Deng, Kai Han, and Yunhe Wang. Token compensator: Altering inference cost of vision transformer without re-tuning. In ECCV, pages 76–94, 2024

    2024

  13. [21]

    Logicad: Explainable anomaly detection via vlm-based text feature extraction

    Er Jin, Qihui Feng, Yongli Mou, Gerhard Lakemeyer, Stefan Decker, Oliver Simons, and Johannes Stegmaier. Logicad: Explainable anomaly detection via vlm-based text feature extraction. In AAAI, pages 4129–4137, 2025

    2025

  14. [22]

    Llms meet vlms: Boost open vocabulary object detection with fine-grained descriptors

    Sheng Jin, Xueying Jiang, Jiaxing Huang, Lewei Lu, and Shijian Lu. Llms meet vlms: Boost open vocabulary object detection with fine-grained descriptors. In ICLR, 2024

    2024

  15. [23]

    Mm-reasoner: A multi- modal knowledge-aware framework for knowledge-based visual question answering

    Mahmoud Khademi, Ziyi Yang, Felipe Frujeri, and Chenguang Zhu. Mm-reasoner: A multi- modal knowledge-aware framework for knowledge-based visual question answering. InEMNLP, pages 6571–6581, 2023

    2023

  16. [24]

    Vlm-pl: Advanced pseudo labeling approach for class incremental object detection via vision-language model

    Junsu Kim, Yunhoe Ku, Jihyeon Kim, Junuk Cha, and Seungryul Baek. Vlm-pl: Advanced pseudo labeling approach for class incremental object detection via vision-language model. In CVPR, pages 4170–4181, 2024

    2024

  17. [25]

    Lookupvit: Compressing visual information to a limited number of tokens

    Rajat Koner, Gagan Jain, Prateek Jain, V olker Tresp, and Sujoy Paul. Lookupvit: Compressing visual information to a limited number of tokens. In ECCV, pages 322–337, 2024

    2024

  18. [26]

    Visual genome: Connecting language and vision using crowdsourced dense image annotations

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer ...

    2017

  19. [27]

    Ez-hoi: Vlm adaptation via guided prompt learning for zero-shot hoi detection

    Qinqian Lei, Bo Wang, and Robby Tan. Ez-hoi: Vlm adaptation via guided prompt learning for zero-shot hoi detection. In NeurIPS, pages 55831–55857, 2024

    2024

  20. [28]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  21. [29]

    Seed- bench: Benchmarking multimodal llms with generative comprehension

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed- bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023. 11

    2023 arXiv

  22. [30]

    Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. In ICML, pages 19730–19742, 2023

    2023

  23. [31]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML, pages 12888– 12900, 2022

    2022

  24. [32]

    Inference optimal vlms need only one visual token but larger models

    Kevin Y Li, Sachin Goyal, Joao D Semedo, and J Zico Kolter. Inference optimal vlms need only one visual token but larger models. arXiv preprint arXiv:2411.03312, 2024

    2024 arXiv

  25. [33]

    Tokenpacker: Efficient visual projector for multimodal llm

    Wentong Li, Yuqian Yuan, Jian Liu, Dongqi Tang, Song Wang, Jie Qin, Jianke Zhu, and Lei Zhang. Tokenpacker: Efficient visual projector for multimodal llm. arXiv preprint arXiv:2407.02392, 2024

    2024 arXiv

  26. [34]

    Llama-vid: An image is worth 2 tokens in large language models

    Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. In ECCV, pages 323–340, 2024

    2024

  27. [35]

    Mini-gemini: Mining the potential of multi-modality vision language models

    Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403.18814, 2024

    2024 arXiv

  28. [36]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In EMNLP, 2023

    2023

  29. [37]

    Monkey: Image resolution and text label are important things for large multi-modal models

    Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Monkey: Image resolution and text label are important things for large multi-modal models. In CVPR, pages 26763–26773, 2024

    2024

  30. [38]

    Vila: On pre-training for visual language models

    Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. In CVPR, pages 26689–26699, 2024

    2024

  31. [39]

    Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models

    Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, et al. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. arXiv preprint arXiv:2311.07575, 2023

    2023 arXiv

  32. [40]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In CVPR, pages 26296–26306, 2024

    2024

  33. [41]

    Llavanext: Improved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llavanext: Improved reasoning, ocr, and world knowledge, 2024

    2024

  34. [42]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, pages 34892–34916, 2023

    2023

  35. [43]

    Multi-stage vision token dropping: Towards efficient multimodal large language model

    Ting Liu, Liangtao Shi, Richang Hong, Yue Hu, Quanjun Yin, and Linfeng Zhang. Multi-stage vision token dropping: Towards efficient multimodal large language model. arXiv preprint arXiv:2411.10803, 2024

    2024 arXiv

  36. [44]

    Mmbench: Is your multi-modal model an all-around player? In ECCV, pages 216–233, 2024

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In ECCV, pages 216–233, 2024

    2024

  37. [45]

    Deepseek-vl: towards real-world vision-language under- standing

    Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, et al. Deepseek-vl: towards real-world vision-language under- standing. arXiv preprint arXiv:2403.05525, 2024

    2024 arXiv

  38. [46]

    Questioning, answering, and captioning for zero-shot detailed image caption

    Duc-Tuan Luu, Viet-Tuan Le, and Duc Minh V o. Questioning, answering, and captioning for zero-shot detailed image caption. In Proceedings of the Asian Conference on Computer Vision , pages 242–259, 2024

    2024

  39. [47]

    Does vlm classification benefit from llm description semantics? In AAAI, pages 5973–5981, 2025

    Pingchuan Ma, Lennart Rietdorf, Dmytro Kotovenko, Vincent Tao Hu, and Björn Ommer. Does vlm classification benefit from llm description semantics? In AAAI, pages 5973–5981, 2025. 12

    2025

  40. [48]

    Chartqa: A benchmark for question answering about charts with visual and logical reasoning.arXiv preprint arXiv:2203.10244, 2022

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning.arXiv preprint arXiv:2203.10244, 2022

    2022 arXiv

  41. [49]

    Infographicvqa

    Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawa- har. Infographicvqa. In WACV, pages 1697–1706, 2022

    2022

  42. [50]

    Docvqa: A dataset for vqa on document images

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In WACV, pages 2200–2209, 2021

    2021

  43. [51]

    Ocr-vqa: Visual question answering by reading text in images

    Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In ICDAR, pages 947–952, 2019

    2019

  44. [52]

    X-instructblip: A framework for aligning x-modal instruction-aware representations to llms and emergent cross-modal reasoning

    Artemis Panagopoulou, Le Xue, Ning Yu, Junnan Li, Dongxu Li, Shafiq Joty, Ran Xu, Silvio Savarese, Caiming Xiong, and Juan Carlos Niebles. X-instructblip: A framework for aligning x-modal instruction-aware representations to llms and emergent cross-modal reasoning. arXiv prepr...

    2023 arXiv

  45. [53]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pages 8748–8763, 2021

    2021

  46. [54]

    Llava-prumerge: Adaptive token reduction for efficient large multimodal models

    Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. Llava-prumerge: Adaptive token reduction for efficient large multimodal models. arXiv preprint arXiv:2403.15388, 2024

    2024

  47. [55]

    Textcaps: a dataset for image captioning with reading comprehension

    Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. Textcaps: a dataset for image captioning with reading comprehension. In ECCV, pages 742–758, 2020

    2020

  48. [56]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In CVPR, pages 8317–8326, 2019

    2019

  49. [57]

    Less is more: A simple yet effective token reduction method for efficient multi-modal llms

    Dingjie Song, Wenjun Wang, Shunian Chen, Xidong Wang, Michael X Guan, and Benyou Wang. Less is more: A simple yet effective token reduction method for efficient multi-modal llms. In COLING, pages 7614–7623, 2025

    2025

  50. [58]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024

    2024 arXiv

  51. [59]

    Fastvlm: Efficient vision encoding for vision language models

    Pavan Kumar Anasosalu Vasu, Fartash Faghri, Chun-Liang Li, Cem Koc, Nate True, Albert Antony, Gokul Santhanam, James Gabriel, Peter Grasch, Oncel Tuzel, et al. Fastvlm: Efficient vision encoding for vision language models. arXiv preprint arXiv:2412.13303, 2024

    2024 arXiv

  52. [60]

    Marvelovd: Marrying object recognition and vision-language models for robust open-vocabulary object detection

    Kuo Wang, Lechao Cheng, Weikai Chen, Pingping Zhang, Liang Lin, Fan Zhou, and Guanbin Li. Marvelovd: Marrying object recognition and vision-language models for robust open-vocabulary object detection. In ECCV, pages 106–122, 2024

    2024

  53. [61]

    Fashionvqa: A domain-specific visual question answering system

    Min Wang, Ata Mahjoubfar, and Anupama Joshi. Fashionvqa: A domain-specific visual question answering system. In CVPR, pages 3514–3519, 2023

    2023

  54. [62]

    Cogvlm: Visual expert for pretrained language models

    Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Song XiXuan, et al. Cogvlm: Visual expert for pretrained language models. Advances in Neural Information Processing Systems , 37:121475–121499, 2024

    2024

  55. [63]

    Rl-vlm-f: reinforcement learning from vision language foundation model feedback

    Yufei Wang, Zhanyi Sun, Jesse Zhang, Zhou Xian, Erdem Biyik, David Held, and Zackory Erickson. Rl-vlm-f: reinforcement learning from vision language foundation model feedback. In ICML, pages 51484–51501, 2024

    2024

  56. [64]

    Vary: Scaling up the vision vocabulary for large vision- language model

    Haoran Wei, Lingyu Kong, Jinyue Chen, Liang Zhao, Zheng Ge, Jinrong Yang, Jianjian Sun, Chunrui Han, and Xiangyu Zhang. Vary: Scaling up the vision vocabulary for large vision- language model. In ECCV, pages 408–424, 2024. 13

    2024

  57. [65]

    Pvc: Progressive visual token compression for unified image and video processing in large vision-language models

    Chenyu Yang, Xuan Dong, Xizhou Zhu, Weijie Su, Jiahao Wang, Hao Tian, Zhe Chen, Wenhai Wang, Lewei Lu, and Jifeng Dai. Pvc: Progressive visual token compression for unified image and video processing in large vision-language models. arXiv preprint arXiv:2412.09613, 2024

    2024 arXiv

  58. [66]

    Visionzip: Longer is better but not necessary in vision language models

    Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. Visionzip: Longer is better but not necessary in vision language models. arXiv preprint arXiv:2412.04467, 2024

    2024

  59. [67]

    Mm-vet: Evaluating large multimodal models for integrated capabilities

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490, 2023

    2023 arXiv

  60. [68]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In CVPR, pages 9556–9567, 2024

    2024

  61. [69]

    Good at captioning bad at counting: Benchmarking gpt-4v on earth observation data

    Chenhui Zhang and Sherrie Wang. Good at captioning bad at counting: Benchmarking gpt-4v on earth observation data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7839–7849, 2024

    2024

  62. [70]

    [cls] attention is all you need for training-free visual token pruning: Make vlm inference faster

    Qizhe Zhang, Aosong Cheng, Ming Lu, Zhiyong Zhuo, Minqi Wang, Jiajun Cao, Shaobo Guo, Qi She, and Shanghang Zhang. [cls] attention is all you need for training-free visual token pruning: Make vlm inference faster. arXiv preprint arXiv:2412.01818, 2024

    2024 arXiv

  63. [71]

    Llava-mini: Efficient image and video large multimodal models with one vision token

    Shaolei Zhang, Qingkai Fang, Zhe Yang, and Yang Feng. Llava-mini: Efficient image and video large multimodal models with one vision token. In ICLR, 2025

    2025

  64. [72]

    Minigpt-4: Enhanc- ing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhanc- ing vision-language understanding with advanced large language models. In ICLR, 2024

    2024

  65. [73]

    Focusllava: A coarse-to-fine approach for efficient and effective visual token compression

    Yuke Zhu, Chi Xie, Shuang Liang, Bo Zheng, and Sheng Guo. Focusllava: A coarse-to-fine approach for efficient and effective visual token compression. arXiv preprint arXiv:2411.14228, 2024. 14

    2024 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.