Pith. sign in

REVIEW 4 major objections 5 minor 5 cited by

FocusLLaVA: A Coarse-to-Fine Approach for Efficient and Effective Visual Token Compression

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that removing visual redundancy via a coarse-to-fine, vision-then-text guided token sampler makes a high-resolution multimodal LLM both faster and more accurate.

desk verdict A well-motivated coarse-to-fine compression method with a fair internal comparison, but the accuracy gains are small and uncertified, so the 'compression improves accuracy' claim needs more evidence before I'd trust it. read the letter →

arxiv 2411.14228 v1 pith:C2EGESCA submitted 2024-11-21 cs.CV

Yuke Zhu , Chi Xie , Shuang Liang , Bo Zheng , Sheng Guo This is my paper · ORCID
classification cs.CV
keywords visualtokencompressionmultimodallargelanguagemodelshigh-resolutionimageunderstandingvision-guidedsamplingtext-guidedattentionselectioncoarse-to-fineefficientinferenceinstruction-awarepruning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FocusLLaVA argues that the visual tokens produced from high-resolution images contain a large amount of redundancy, and that removing this redundancy with learned, instruction-aware selection can make a multimodal language model faster and more accurate at the same time. The method compresses in two stages: a vision-guided sampler first chooses a downsampling scale for each local image region based only on image content, and a text-guided sampler later keeps the visual tokens that the user's question actually attends to. With about 39 percent of the original visual tokens, FocusLLaVA outperforms its full-token LLaVA-NeXT baseline on 9 of 10 benchmarks while increasing throughput from 2.85 to 4.01 samples per second. If the claim holds, it means high-resolution visual understanding does not have to be bought at the price of quadratic token growth.

What carries the argument

The two load-bearing components are the vision-guided sampler and the text-guided sampler. The vision-guided sampler partitions the global visual feature map into 4x4 local blocks, max-pools each block at three scales (4x4, 2x2, 1x1), and uses a learned selector that scores each block against the global image features to choose one scale per region; a switch-transformer-style balance loss prevents the selector from always choosing the same branch. The text-guided sampler, inserted at layer 8 of the LLM, computes per-token importance $S_i$ as the average over text tokens of the max over attention heads of the text-to-visual attention scores, then keeps the smallest number of top tokens whose cumulative normalized importance exceeds $\gamma$; during training it randomly samples a layer and a $\gamma$ value to make the model robust to token dropping. The coarse-to-fine structure matters because visual-only compression is applied early in the projector, where it is cheap, while text-guided selection is deferred to a middle LLM layer where the language signal has become reliable.

What would settle it

Run FocusLLaVA and its exact baseline several times with different random seeds on the same 1M-sample training data; if the average accuracy gap on GQA, MME, ScienceQA, and TextVQA falls inside the run-to-run variance, then the claim that compression improves performance is not confirmed.

Watch

Extended reading notes

Core claim

The paper's central claim is that removing visual redundancy can improve both efficiency and performance, contradicting the common trade-off view. Concretely, the authors show that a coarse-to-fine pipeline—vision-guided region-level scale selection in the projector followed by text-guided, attention-based token selection inside the LLM—keeps only about 39 percent of the visual tokens and still raises scores on nearly all evaluated benchmarks relative to an identically trained baseline. The vision-guided sampler treats each 4x4 local feature block as a unit and selects among 1x1, 2x2, and 4x4 max-pooled representations using a learned correlation with the global image; the text-guided sampler computes an importance score from multi-head attention between text and visual tokens and drops tokens whose cumulative normalized importance falls below a threshold $\gamma$. The authors interpret the result as evidence that low-information regions and instruction-irrelevant tokens are not needed for correct answers, and that a model trained with stochastic token dropping can learn to answer without them.

Load-bearing premise

The load-bearing premise is that the self-implemented LLaVA-NeXT baseline is identical to FocusLLaVA except for the compression modules, so the small benchmark differences (for example +0.3 on GQA and +0.2 on MMBench-CN) are caused by those modules rather than by training noise or implementation details, and the paper reports a single run without multiple seeds.

Editorial extensions

If this is right

  • High-resolution multimodal models can be made faster without a performance penalty if redundancy is removed both before and inside the LLM.
  • A learned per-region downsampling choice beats handcrafted token-importance metrics: replacing the selector with a fixed similarity heuristic drops scores as token budgets shrink.
  • The balance loss is necessary: without it the scale selector collapses onto one branch and performance falls well below baseline.
  • Stochastic training with a random layer and random threshold converts text-guided token dropping from a performance loss into a small gain on several benchmarks.
  • More downsampling scales help fine-grained reading tasks such as TextVQA, suggesting the framework benefits from richer scale choices.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the two-stage separation holds up, visual redundancy and instruction relevance are partly independent axes, so each sampler could be tuned or retrained separately without retraining the whole model.
  • The text-guided sampler's dependence on the question suggests the same architecture could serve as a dynamic per-query token budget, spending more tokens when the question demands fine detail and fewer for global questions.
  • A natural testable extension is to apply the same coarse-to-fine recipe to video or document inputs, where temporal and layout redundancy is even larger than in single images.
  • The reliance on attention-map importance implies the method's ceiling is tied to how well the LLM's self-attention locates answers; if attention is noisy or hallucinated, the selected tokens may be wrong even though they are few.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes FocusLLaVA, a visual-token compression method for high-resolution multimodal LLMs built on LLaVA-NeXT. It combines a vision-guided sampler, which performs region-level multi-scale downsampling before the LLM, with a text-guided sampler, which prunes visual tokens inside an intermediate LLM layer based on attention relevance to the instruction. The authors report that keeping about 39% of visual tokens improves the model over their LLaVA-NeXT baseline on 9 of 10 benchmarks and raises throughput from 2.85 to 4.01 samples per second. The paper includes implementation details, ablations of the two modules, sensitivity studies on window size, balance-loss weight, threshold, and layer placement, and qualitative analyses of selected regions.

Significance. If the empirical claims withstand scrutiny, the paper makes a useful contribution to efficient multimodal inference: it addresses visual redundancy with a learned coarse-to-fine scheme rather than a hand-crafted heuristic, and the disentangled placement of the two samplers is clearly motivated. The ablations are informative and the qualitative visualizations support the intuition that the vision-guided sampler focuses on information-dense regions while the text-guided sampler focuses on instruction-relevant regions. The paper also compares against a heuristic token-dropping baseline in the supplementary, which helps position the learned selector. However, the headline result depends on small benchmark deltas against a self-implemented baseline, with no error bars, no released code or checkpoints, and no full speed-measurement protocol, so the reproducibility and statistical strength of the central claim are not yet established.

major comments (4)
  1. [Section 4.3, Table 1] The central claim that removing visual redundancy improves both efficiency and accuracy rests on Table 1, where FocusLLaVA exceeds the self-implemented LLaVA-NeXT baseline by margins such as +0.3 on GQA, +0.2 on MMBench-CN, +0.4 on ScienceQA, and -6 on MME-Cognition. For instruction-tuned 8B MLLMs these differences are within typical run-to-run variance, yet no repeated seeds, error bars, or significance tests are reported. Please provide multiple runs with mean and standard deviation, or paired comparisons where possible; if that is infeasible, the performance claim should be weakened to matched performance at reduced token count and higher throughput.
  2. [Sections 4.2 and 4.3, Table 2] The 'strict alignment' between FocusLLaVA and its baseline is not independently verifiable. The fine-tuning data replaces the unreleased LLaVA-NeXT mixture with ALLaVA-Instruct-VFLAN-4V and other open-source sets, and Section 4.2 states that the specific data configuration is available in the supplementary, but the supplementary contains no such table. In addition, the speed comparison (2.85 vs 4.01 samples/s) is reported without input resolution, batch size, generation length, decoding method, hardware details, number of runs, or whether the measurement includes prefill and decoding phases. Please release the exact data recipe, code and model checkpoints, and a complete speed-measurement protocol so that the efficiency claim can be reproduced.
  3. [Section 4.4, Tables 2-6 and Figure 2] Several key hyperparameters are selected by varying them on the same benchmark suite that is later used for the headline results in Table 1: the window size w (Table 4), balance-loss weight alpha (Table 5), text-guided threshold and random-layer schedule (Table 6), and the insertion layer (Figure 2). Because the reported improvements over the baseline are small, the observed advantages could be artifacts of tuning on the evaluation benchmarks rather than evidence of a general property of the compression modules. Please state whether any held-out split or separate validation set was used for hyperparameter selection, or show that the conclusions are stable across a range of settings.
  4. [Section 3.2, Eq. (2)] The training-time formula for the vision-guided sampler multiplies the selected downsampled token by Top1(Softmax(Z)), while inference uses the raw selected token without this probability scaling. This train/test mismatch is not discussed. Additionally, because the value passed forward is DS[argmax(Z)](Xr) scaled by a scalar probability, gradients reach the selector only through the probability of the chosen scale and never through the unselected downsampled features. Please clarify whether this is intended as a straight-through estimator, and report whether using hard selection with the balance loss alone, or removing the probability multiplication, changes the results.
minor comments (5)
  1. [Section 4.4, Figure 2] The axis labels in Figure 2 are garbled: the x-axis appears to be labeled with both benchmark score values and samples/sec values, making the accuracy-versus-speed trade-off impossible to read. Please split the two curves into separate panels with clear axis labels.
  2. [Section 4.4, Table 2 caption] The formula m - n + i*n/32 for converting remaining visual tokens should define m, n, and i explicitly, and should state that this is an average over the 32 LLM layers rather than the instantaneous token count.
  3. [Section 3.2, Eqs. (3)-(5)] The summation limits are typeset inconsistently (for example, 'nX0' and 'N-1X0'), and the indicator function in Eq. (4) should be written with standard indicator notation; please also clarify the relation between n in Eq. (3) and the number of scales S.
  4. [Section 4.4] There are minor language issues: 'harmfull' should be 'harmful', '31th' should be '31st', and the sentence describing the balance loss as ensuring 'each expert is selected uniformly' should be qualified, since the auxiliary loss encourages balanced fractional usage rather than exact uniform selection per sample.
  5. [References] Reference [4] has a malformed author string ('Sagnak Ta, and sırlar'); the author list should be corrected. Also, references [11] and [30] appear to point to the same Open-LLaVA-NeXT repository with different numbering and should be unified or disambiguated.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation: the performance and efficiency claims are empirical benchmark comparisons, not fitted quantities renamed as predictions.

full rationale

This paper makes an empirical claim that two learned visual-token-compression modules improve both accuracy and speed relative to a self-implemented LLaVA-NeXT baseline. There is no first-principles derivation whose output is defined by its input, and no equation in the paper constructs the reported benchmark improvements from a fitted parameter. The mild self-referential element is that several hyperparameters (balance-loss weight, text-guided threshold gamma, sampler layer, window size) were selected using the same evaluation benchmarks that later appear in the main comparison; this is ordinary hyperparameter tuning and selection-on-benchmark, not a fitted-input-renamed-as-prediction, because the paper does not present those settings as predictions and the central comparison remains a matched baseline versus method. There is also no load-bearing self-citation chain: the references to Open-LLaVA-NeXT and LLaVA-NeXT are external baselines and implementations, not uniqueness theorems or prior results by the present authors. Concerns about small single-run deltas, the absence of multiple seeds, and the unverifiable training-data alignment are correctness or evidence weaknesses, not circularity. The score of 1 reflects only the mild benchmark-selection flavor rather than any reduction-by-construction.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim rests on several domain assumptions: the attention-based importance signal at layer 8 is a reliable selector, the LLaVA-NeXT reproduction is a perfectly matched baseline, the stochastic training transfers to fixed inference settings, and max-pooling preserves task-relevant details. The free parameters are hand-chosen and ablated on the same evaluation benchmarks, so they do not provide independent confirmation of the method.

free parameters (5)
  • gamma (text-guided sampler threshold) = 0.85
    Chosen by hand; controls how many visual tokens are kept by cumulative attention importance (Eq. 9); tuned on the evaluation benchmarks.
  • alpha (balance loss weight) = 0.1
    Selected via ablation on TextVQA, MME, ScienceQA, and GQA (Table 5); weights the switch-transformer style auxiliary loss.
  • Window size w for local feature block = 4
    Ablated on the evaluation benchmarks (Table 4); default 4x4.
  • Scale set for multi-scale downsampling = 3 branches (4x4, 2x2, 1x1)
    Default; a 7-branch variant is tested in Table 3 and selected after observing TextVQA gains.
  • Text-guided sampler insertion layer = 8
    Chosen from Figure 2 based on performance and speed tradeoff on evaluation benchmarks; training uses random layers 8 to 24.
assumptions (4)
  • domain assumption The attention-based importance score at layer 8 (max over heads, mean over text tokens) reliably identifies visual tokens needed for the final answer.
    Section 3.3, Eqs. (6)-(8). If attention is unfocused or noisy, top-k selection can drop critical tokens, hurting accuracy.
  • domain assumption The self-implemented LLaVA-NeXT is a faithful baseline: identical LLM, visual encoder, training data, and settings, so measured deltas are attributable to the proposed modules.
    Section 4.3. The deltas are small and no seeds are reported, so violations would undermine the claim.
  • domain assumption Stochastic training (random layer and threshold) makes the model robust so that inference with fixed layer 8 and threshold 0.85 does not degrade accuracy.
    Section 3.3, 'we can randomly select a decoder layer and an importance threshold gamma'. The transfer from random to fixed settings is not formally justified, only shown in Table 6.
  • domain assumption Max-pooling downsampling of 4x4 feature blocks to 2x2 or 1x1 preserves task-relevant information; information density is a valid proxy for retention.
    Section 3.2, 'use a set of Max-Pooling operations'. The vision-only ablation shows slight TextVQA loss, indicating some information loss.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FocusLLaVA: A Coarse-to-Fine Approach for Efficient and Effective Visual Token Compression." pith.science (2026). https://pith.science/paper/C2EGESCA

@misc{pith2026241114228,
  author       = {Pith},
  title        = {Pith review of: FocusLLaVA: A Coarse-to-Fine Approach for Efficient and Effective Visual Token Compression},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/C2EGESCA}},
  note         = {Machine review of arXiv:2411.14228}
}
read the original abstract

Recent advances on Multi-modal Large Language Models have demonstrated that high-resolution image input is crucial for model capabilities, especially for fine-grained tasks. However, high-resolution images lead to a quadratic increase in the number of visual tokens input into LLMs, resulting in significant computational costs. Current work develop visual token compression methods to achieve efficiency improvements, often at the expense of performance. We argue that removing visual redundancy can simultaneously improve both efficiency and performance. We build a coarse-to-fine visual token compression method, with a vision-guided sampler for compressing redundant regions with low information density, and a text-guided sampler for selecting visual tokens that are strongly correlated with the user instructions.With these two modules, the proposed FocusLLaVA achieves improvements in both efficiency and performance. We validate the effectiveness of our approach on a wide range of evaluation datasets.

Figures

Figures reproduced from arXiv: 2411.14228 by the authors.

Figure 1
Figure 1. (a) Overall structure of FocusLLaVA. The two core modules are vision-guided sampler and text-guided sampler. The features from each sub-images are first concatenated into a whole and then partitioned by regions, each forming a local feature block. It is then processed by vision-guided sampler. (b) The structure of vision-guided sampler. It takes a feature block and global image’s features as inputs and output the pr… view at source ↗
Figure 2
Figure 2. Performance and speed with textual guidance in dif￾ferent layers [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Statistics of multi-scale sampling. For each sample, the [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Heatmap of selected areas from vision-guided sampler and text-guided sampler. For each image set, from left to right: the original image, the heatmap of vision-guided sampler, the heatmap of text-guided sampler. that the vision-guided sampler tends to focus on areas of…
Figure 5
Figure 5. Figure 5: The importance map from different layers of LLM. There are 32 layers in total. We select the importance map every two layers. The maps are arranged in reading order. Evolution of textual guidance. We explore the differences of importance map defined by Eq. (8) across v…
Figure 6
Figure 6. Figure 6: Selected areas from vision-guided sampler and text￾guided sampler. Q: What state Is this car from? A: Califolia Q: Whe was this photo taken? A: 02/14/2012 [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: to demonstrate the different characteristics of visual guidance and textual guidance. Besides, we also visualize the different areas selected across different instructions. It is shown in [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Difference of selected areas across different queries. irrelevant areas in the image. Subsequently, it shifts its at￾tention to the entire body of the player, including the limbs and the clothing. The model then begins to concentrate on [PITH_FULL_IMAGE:figures/full_f…
Figure 9
Figure 9. Figure 9: Difference of selected areas across different queries [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: Full details of the evolution of textual guidance across LLM layers. the number on the clothing. However, after initially focus￾ing on the numbers, the model does not maintain a consis￾tent focus; instead, it intermittently returns its attention to the player and the …

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Learning to Predict Middle-Layer Attention in MLLMs for Visual Token Prunin

    cs.AI 2026-08 conditional novelty 6.0 of 10

    MAP predicts sample-specific middle-layer attention with a lightweight model and prunes visual tokens before the LLM, keeping 97.5% of performance at 5.56% of tokens.

  2. METEOR: Multi-Encoder Collaborative Token Pruning for Efficient Vision Language Models

    cs.CV 2025-07 conditional novelty 6.0 of 10

    METEOR is a three-stage token pruning framework that reduces visual tokens in multi-encoder MLLMs by 76% with only a 0.3% average accuracy drop.

  3. ET-Prune: Evidence-Aware Dynamic Budgeting for Visual Token Pruning in Text-Rich MLLMs

    cs.CV 2026-08 conditional novelty 5.0 of 10

    A question-aware, per-image budget decides which visual tokens to keep, letting OCR-focused multimodal models hold accuracy at about half the tokens.

  4. Top-Down Compression: Revisit Efficient Vision Token Projection for Visual Instruction Tuning

    cs.CV 2025-05 conditional novelty 5.0 of 10

    A visual instruction tuning model that combines global fusion and dual-expert token selection to cut visual tokens by 75 to 95 percent with comparable or better benchmark scores.

  5. LLaVA-Zip: Adaptive Visual Token Compression with Intrinsic Image Information

    cs.CV 2024-12 conditional novelty 3.0 of 10

    DFMR uses the mean within-patch standard deviation of CLIP features to pick a pooling factor of 1, 2, or 3 per image, and training LLaVA-1.5 with this dynamic compression improves average benchmark scores over a rando...

Reference graph

Works this paper leans on

59 extracted references · 27 canonical work pages · cited by 5 Pith papers

  1. [1]

    Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sa- hand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Kar ´en Simonyan

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Men- sch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L. Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sa- hand Sharifzadeh, Mikolaj B...

    work page 2022

  2. [2]

    Hired: Attention-guided token dropping for efficient inference of high-resolution vision-language models in resource- constrained environments

    Kazi Hasan Ibn Arif, JinYi Yoon, Dimitrios S Nikolopou- los, Hans Vandierendonck, Deepu John, and Bo Ji. Hired: Attention-guided token dropping for efficient inference of high-resolution vision-language models in resource- constrained environments. arXiv preprint arXiv:2408.10945,

    arXiv

  3. [3]

    Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond. CoRR,

    work page

  4. [4]

    Introducing our multimodal models, 2023

    Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, , Sagnak ˘ Ta, and sırlar. Introducing our multimodal models, 2023. 2

    work page 2023

  5. [5]

    Ma- tryoshka multimodal models

    Mu Cai, Jianwei Yang, Jianfeng Gao, and Yong Jae Lee. Ma- tryoshka multimodal models. CoRR, abs/2405.17430, 2024. 1, 2

    arXiv 2024

  6. [6]

    Madtp: Multi- modal alignment-guided dynamic token pruning for accel- erating vision-language transformer

    Jianjian Cao, Peng Ye, Shengze Li, Chong Yu, Yan- song Tang, Jiwen Lu, and Tao Chen. Madtp: Multi- modal alignment-guided dynamic token pruning for accel- erating vision-language transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15710–15719, 2024. 2

    2024

  7. [7]

    Allava: Harness- ing gpt4v-synthesized data for A lite vision-language model

    Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Juny- ing Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jian- quan Li, Xiang Wan, and Benyou Wang. Allava: Harness- ing gpt4v-synthesized data for A lite vision-language model. CoRR, abs/2402.11684, 2024. 2

    arXiv 2024

  8. [8]

    Allava: Harness- ing gpt4v-synthesized data for a lite vision-language model,

    Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Juny- ing Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jian- quan Li, Xiang Wan, and Benyou Wang. Allava: Harness- ing gpt4v-synthesized data for a lite vision-language model,

    work page

Show all 59 references
  1. [9]

    Geoqa: A geomet- ric question answering benchmark towards multimodal nu- merical reasoning

    Jiaqi Chen, Jianheng Tang, Jinghui Qin, Xiaodan Liang, Lingbo Liu, Eric Xing, and Liang Lin. Geoqa: A geomet- ric question answering benchmark towards multimodal nu- merical reasoning. In Findings of the Association for Com- putational Linguistics: ACL-IJCNLP 2021 , pages 513–523,

    2021

  2. [10]

    Dragonfly: Multi-resolution zoom supercharges large visual-language model

    Kezhen Chen, Rahul Thapa, Rahul Chalamala, Ben Athi- waratkun, Shuaiwen Leon Song, and James Zou. Dragonfly: Multi-resolution zoom supercharges large visual-language model. arXiv preprint arXiv:2406.00977, 2024. 2

    2024 arXiv

  3. [11]

    Open-llava-next: An open- source implementation of llava-next series for facilitating the large multi-modal model community

    Lin Chen and Long Xing. Open-llava-next: An open- source implementation of llava-next series for facilitating the large multi-modal model community. https://github. com/xiaoachen98/Open-LLaVA-NeXT, 2024. 5

    2024

  4. [12]

    Sharegpt4v: Improving large multi-modal models with better captions

    Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793, 2023. 5

    2023 arXiv

  5. [13]

    An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models

    Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. CoRR, abs/2403.06764, 2024. 2

    2024 arXiv

  6. [14]

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: Towards general- purpose vision-language models with instruction tuning. In Advances in Neural Information Processing Systems 36...

    2023

  7. [15]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition ...

    2021

  8. [16]

    Switch transformers: Scaling to trillion parameter models with sim- ple and efficient sparsity

    William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with sim- ple and efficient sparsity. Journal of Machine Learning Re- search, 23(120):1–39, 2022. 4

    2022

  9. [17]

    MME: A comprehensive evaluation benchmark for multimodal large language models

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Ji. MME: A comprehensive evaluation benchmark for multimodal large language models. CoRR, abs/2306.13394, 2023. 5, 6

    2023 arXiv

  10. [18]

    Matryoshka query transformer for large vision-language models

    Wenbo Hu, Zi-Yi Dou, Liunian Harold Li, Amita Ka- math, Nanyun Peng, and Kai-Wei Chang. Matryoshka query transformer for large vision-language models. CoRR, abs/2405.19315, 2024. 2

    2024 arXiv

  11. [19]

    Bliva: A simple multimodal llm for better handling of text-rich visual questions

    Wenbo Hu, Yifan Xu, Yi Li, Weiyue Li, Zeyuan Chen, and Zhuowen Tu. Bliva: A simple multimodal llm for better handling of text-rich visual questions. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 2256– 2264, 2024. 2

    2024

  12. [20]

    Hudson and Christopher D

    Drew A. Hudson and Christopher D. Manning. GQA: A new dataset for real-world visual reasoning and compositional question answering. In IEEE Conference on Computer Vi- sion and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 6700–6709. Computer Visi...

    2019

  13. [21]

    Dvqa: Understanding data visualizations via ques- tion answering

    Kushal Kafle, Scott Cohen, Brian Price, and Christopher Kanan. Dvqa: Understanding data visualizations via ques- tion answering. In CVPR, 2018. 5

    2018

  14. [22]

    A diagram is worth a dozen images

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Min Joon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV, pages 235–251....

    2016

  15. [23]

    Ocr-free document understanding transformer

    Geewook Kim, Teakgyu Hong, Moonbin Yim, JeongYeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sang- doo Yun, Dongyoon Han, and Seunghyun Park. Ocr-free document understanding transformer. In Computer Vision - ECCV 2022 - 17th European Conference, Tel Aviv, Israel, October...

    2022

  16. [24]

    Otterhd: A high-resolution multi- modality model

    Bo Li, Peiyuan Zhang, Jingkang Yang, Yuanhan Zhang, Fanyi Pu, and Ziwei Liu. Otterhd: A high-resolution multi- modality model. CoRR, abs/2311.04219, 2023. 2, 6

    2023 arXiv

  17. [25]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA , pages 19730...

    2023

  18. [26]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Pro- cessing, EMNLP 2023, Singapore, December 6-10, 2023...

    2023

  19. [27]

    Mini-gemini: Mining the potential of multi-modality vision language models

    Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. CoRR, abs/2403.18814, 2024. 2, 6

    2024 arXiv

  20. [28]

    Mon- key: Image resolution and text label are important things for large multi-modal models

    Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Mon- key: Image resolution and text label are important things for large multi-modal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog...

    2024

  21. [29]

    Moe-llava: Mixture of experts for large vision-language models

    Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Junwu Zhang, Munan Ning, and Li Yuan. Moe-llava: Mixture of experts for large vision-language models. CoRR, abs/2401.15947, 2024. 2

    2024 arXiv

  22. [30]

    Open-llava-next: An open- source implementation of llava-next series for facilitating the large multi-modal model community

    Chen Lin and Xing Long. Open-llava-next: An open- source implementation of llava-next series for facilitating the large multi-modal model community. https://github. com/xiaoachen98/Open-LLaVA-NeXT , 2024. 2, 3, 5

    2024

  23. [31]

    SPHINX: the joint mix- ing of weights, tasks, and visual embeddings for multi-modal large language models

    Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, Jiaming Han, Siyuan Huang, Yichi Zhang, Xuming He, Hongsheng Li, and Yu Qiao. SPHINX: the joint mix- ing of weights, tasks, and visual embeddings for multi-modal lar...

    2023 arXiv

  24. [32]

    Boosting multimodal large language models with visual to- kens withdrawal for rapid inference

    Zhihang Lin, Mingbao Lin, Luxi Lin, and Rongrong Ji. Boosting multimodal large language models with visual to- kens withdrawal for rapid inference. CoRR, abs/2405.05803,

    arXiv

  25. [33]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in Neural Informa- tion Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023. 1, 2, 5

    2023

  26. [34]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26296–26306, 2024. 1, 2, 6

    2024

  27. [35]

    Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024. 2, 5, 6

    2024

  28. [36]

    Mmbench: Is your multi-modal model an all-around player? CoRR, abs/2307.06281, 2023

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player? CoRR, abs/2307.06281, 2023. 5, 6

    2023 arXiv

  29. [37]

    Textmonkey: An ocr-free large multimodal model for understanding document.CoRR, abs/2403.04473, 2024

    Yuliang Liu, Biao Yang, Qiang Liu, Zhang Li, Zhiyin Ma, Shuo Zhang, and Xiang Bai. Textmonkey: An ocr-free large multimodal model for understanding document.CoRR, abs/2403.04473, 2024. 2

    2024 arXiv

  30. [38]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In Ad- vances in Neural Information Processing Systems 35: An...

    2022

  31. [39]

    Feast your eyes: Mixture-of- resolution adaptation for multimodal large language models

    Gen Luo, Yiyi Zhou, Yuxin Zhang, Xiawu Zheng, Xi- aoshuai Sun, and Rongrong Ji. Feast your eyes: Mixture-of- resolution adaptation for multimodal large language models. arXiv preprint arXiv:2403.03003, 2024. 2

    2024 arXiv

  32. [40]

    Joty, and Enamul Hoque

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq R. Joty, and Enamul Hoque. Chartqa: A benchmark for question an- swering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022 , p...

    2022

  33. [41]

    Minesh Mathew, Dimosthenis Karatzas, and C. V . Jawa- har. Docvqa: A dataset for VQA on document images. In IEEE Winter Conference on Applications of Computer Vi- sion, WACV 2021, Waikoloa, HI, USA, January 3-8, 2021 , pages 2199–2208. IEEE, 2021. 5

    2021

  34. [42]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of th...

    2021

  35. [43]

    Llava-prumerge: Adaptive token reduction for efficient large multimodal models

    Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. Llava-prumerge: Adaptive token reduction for efficient large multimodal models. CoRR, abs/2403.15388, 2024. 2, 6

    2024

  36. [44]

    Towards VQA models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards VQA models that can read. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 8317–8326...

    2019

  37. [45]

    Less is more: A sim- ple yet effective token reduction method for efficient multi- modal llms

    Dingjie Song, Wenjun Wang, Shunian Chen, Xidong Wang, Michael Guan, and Benyou Wang. Less is more: A sim- ple yet effective token reduction method for efficient multi- modal llms. arXiv preprint arXiv:2409.10994, 2024. 1, 2, 6

    2024 arXiv

  38. [46]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. 2

    2024 arXiv

  39. [47]

    Vary: Scaling up the vision vocabulary for large vision-language models

    Haoran Wei, Lingyu Kong, Jinyue Chen, Liang Zhao, Zheng Ge, Jinrong Yang, Jianjian Sun, Chunrui Han, and Xiangyu Zhang. Vary: Scaling up the vision vocabulary for large vision-language models. CoRR, abs/2312.06109, 2023. 2

    2023 arXiv

  40. [48]

    Llava-uhd: an lmm perceiving any aspect ratio and high-resolution images

    Ruyi Xu, Yuan Yao, Zonghao Guo, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu, Maosong Sun, and Gao Huang. Llava-uhd: an lmm perceiving any aspect ratio and high-resolution images. arXiv preprint arXiv:2403.11703, 2024. 2

    2024 arXiv

  41. [49]

    mplug-owl3: Towards long image-sequence understanding in multi-modal large language models

    Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-owl3: Towards long image-sequence understanding in multi-modal large language models. arXiv preprint arXiv:2408.04840, 2024. 6

    2024 arXiv

  42. [50]

    Mm-vet: Evaluating large multimodal models for integrated capabilities

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. In Forty-first International Conference on Ma- chine Learning, 2024. 5

    2024

  43. [51]

    Texthawk: Exploring efficient fine- grained perception of multimodal large language models

    Ya-Qi Yu, Minghui Liao, Jihao Wu, Yongxin Liao, Xiaoyu Zheng, and Wei Zeng. Texthawk: Exploring efficient fine- grained perception of multimodal large language models. arXiv preprint arXiv:2404.09204, 2024. 1, 2

    2024 arXiv

  44. [52]

    Tex- thawk2: A large vision-language model excels in bilingual ocr and grounding with 16x fewer tokens

    Ya-Qi Yu, Minghui Liao, Jiwen Zhang, and Jihao Wu. Tex- thawk2: A large vision-language model excels in bilingual ocr and grounding with 16x fewer tokens. arXiv preprint arXiv:2410.05261, 2024. 2

    2024 arXiv

  45. [53]

    Token-level correlation-guided com- pression for efficient multimodal document understanding

    Renshan Zhang, Yibo Lyu, Rui Shao, Gongwei Chen, Weili Guan, and Liqiang Nie. Token-level correlation-guided com- pression for efficient multimodal document understanding. arXiv preprint arXiv:2407.14439, 2024. 2

    2024 arXiv

  46. [54]

    Beyond llava-hd: Diving into high-resolution large multimodal models

    Yi-Fan Zhang, Qingsong Wen, Chaoyou Fu, Xue Wang, Zhang Zhang, Liang Wang, and Rong Jin. Beyond llava-hd: Diving into high-resolution large multimodal models. arXiv preprint arXiv:2406.08487, 2024. 1, 2, 6 FocusLLaV A: A Coarse-to-Fine Approach for Efficient and Effective Visu...

    2024 arXiv

  47. [56]

    This is used to illustrate that a learned metric rather than hand-crafted will solve the prob- lem of performance reduction

    Comparison with Heuristic Visual Token Dropping In this part, we compare the proposed method with the man- ually designed metrics. This is used to illustrate that a learned metric rather than hand-crafted will solve the prob- lem of performance reduction. To this end, we modif...

  48. [57]

    6 and Fig

    More Examples In this part, more examples are visualized in Fig. 6 and Fig. 7 to demonstrate the different characteristics of visual guidance and textual guidance. Besides, we also visualize the different areas selected across different instructions. It is shown in Fig. 8 and Fig. 9

  49. [58]

    Details of the Evolution of Textual Guidance In this part, we further analyze the evolution of textual guid- ance as the number of layers increases. In Fig. 10, the ques- tion posed is ”What number is on the player’s jersey?”. Our analysis reveals that the textual guidance doe...

    2002

  50. [59]

    During our training process, under the constraint of balance loss, the model is required to select three different visual scales with as equal probabil- ity as possible

    Imbalance Loss In this section, we further investigate the impact of balance loss on model performance. During our training process, under the constraint of balance loss, the model is required to select three different visual scales with as equal probabil- ity as possible. We ...

  51. [2279]

    Association for Computational Linguistics, 2022. 5

    2022

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.