Pith. sign in

REVIEW 4 major objections 5 minor 44 references

Instruction-Guided Fusion of Multi-Layer Visual Features in Large Vision-Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that dynamically weighting vision-encoder layers by the text instruction improves large vision-language models, beating a 7B baseline and task-agnostic fusion on 8 of 10 benchmarks without adding visual tokens.

desk verdict Solid empirical layer-wise analysis plus a plausible fusion module, but the instruction-guidance claim is not isolated by the ablation. read the letter →

arxiv 2501.08443 v3 pith:7AWBXT4T submitted 2024-12-26 cs.CV cs.LG

Xu Li , Yi Zheng , Haotian Chen , Xiaolei Chen , Yuxuan Liang , Chenghang Lai , Bin Li , Xiangyang Xue This is my paper · ORCID
classification cs.CVcs.LG
keywords largevision-languagemodelsinstruction-guidedfusionhierarchicalvisualfeaturesmulti-layerfeatureinstructiontuningfine-grainedperceptionweightallocationCLIP-ViT
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that large vision-language models leave performance on the table by using only the final layers of the vision encoder, and that the usual fix of mixing all layer features uniformly is also wrong because different tasks need different visual levels. It claims that a lightweight module can read the text instruction and use it to assign fusion weights to groups of encoder layers, emphasizing low-level details for fine-grained perception and mid-to-high-level details for semantic tasks. Integrated into a standard 7B vision-language model, this instruction-guided aggregator is reported to improve accuracy on most of ten benchmarks compared with the baseline and with existing task-agnostic fusion methods, without increasing the number of visual tokens. The paper also presents a systematic study of 18 benchmarks showing that layer contributions are complementary and task-dependent.

What carries the argument

The instruction-guided vision aggregator is the load-bearing module. It divides the 24 CLIP-ViT layers into four groups (low, low-to-mid, mid-to-high, high), averages the class and patch tokens inside each group, embeds the text instruction with a frozen sentence-embedding model, and feeds the embedding as a query through four cross-attention transformer blocks whose softmax head outputs one weight per group. The weights recombine the pooled patch features by weighted sum, and the result is concatenated with the penultimate layer's features so no visual-token count is added. An entropy-based auxiliary loss with coefficient 0.02 keeps the weight allocator from collapsing onto a single group.

What would settle it

A reader could settle the claim by taking the trained weight allocator and swapping the instruction embedding for a constant, randomly shuffled, or zero vector while freezing everything else; if benchmark scores barely move, the guidance from text is not carrying the effect, and the gains are attributable to the extra parameters and multi-layer inputs. A complementary check would compare the learned per-task weight patterns against fixed task-specific weights learned from the training set.

Watch

Extended reading notes

Core claim

The central claim is that task-guided weighting of hierarchical visual features, rather than any fixed recipe, is what unlocks the complementary information in a vision encoder. The authors show that single-layer choices each excel in different task categories, that pairing the penultimate layer with another layer helps, and that static weight patterns underperform a dynamic scheme. They then construct the instruction-guided vision aggregator, which pools the encoder's 24 layers into four visual groups, embeds the user instruction with a frozen sentence-embedding model, and uses a cross-attention weight allocator to produce normalized group weights; the group-pooled patch features are summed with those weights and concatenated with the penultimate layer features. On 10 benchmarks the method scores best on 8 of them against the LLaVA-1.5 baseline and two task-agnostic fusion methods, and an analysis of the learned weights shows mid-to-high groups carrying semantic tasks while low groups gain weight in fine-grained benchmarks.

Load-bearing premise

The argument assumes the frozen sentence-embedding model produces an instruction representation informative enough for the weight allocator to choose genuinely task-appropriate layer weights; if that signal were weak, the reported gains could come from the extra parameters and multi-layer inputs rather than from instruction guidance.

Editorial extensions

If this is right

  • A vision-language model can improve fine-grained and reasoning performance without paying the cost of extra visual tokens, since the aggregator recombines existing layer features.
  • Task categories carry recognizable weight signatures, such as mid-to-high dominance for Chart and OCR and a higher low-group weight for fine-grained vision, which could inform which encoder layers to keep or fine-tune.
  • The same instruction-guided aggregation principle should transfer to other vision encoders and larger language models, making the module a drop-in addition to existing pipelines.
  • Frozen instruction embeddings suffice for routing, so the added trainable machinery is small relative to the language model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test the paper does not run is whether the aggregator's weights are more predictive of the task than the instruction's exact wording; using the same question paraphrased differently should produce similar weight patterns if the module is capturing task semantics.
  • The weight-balancing loss suggests the raw training signal pushes the allocator toward one group; without it the model may rely on mid-to-high features even when low-level detail is needed, so the entropy term is arguably doing part of the task adaptation.
  • The method could be plugged into high-resolution or multi-scale vision pipelines, where layer-group pooling plus instruction routing may reduce the cost of processing many patches.
  • The reliance on clear instructions implies the approach may degrade on vague or multi-intent queries, and a robust version would likely need to fold visual content into the routing decision as well.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper studies the use of hierarchical visual features from a CLIP-ViT encoder in the LLaVA-1.5 framework. It first presents a systematic analysis across 18 benchmarks in 6 task categories, training models with features from different single layers or pairwise layer combinations. The findings indicate task-dependent complementarity and suboptimality of static fusion. The authors then propose an instruction-guided vision aggregator that divides the 24 encoder layers into four groups, average-pools within each group, computes a weighted sum using weights produced by a cross-attention allocator conditioned on an MPNet sentence embedding of the text instruction, and concatenates the result with the penultimate-layer features. They evaluate the integrated model on 10 benchmarks against LLaVA-1.5, DenseConnector, MMFuser, and several 7B-scale LVLMs, reporting best scores on 8 of 10 benchmarks, and analyze the learned weight distributions per task category.

Significance. If the findings hold, the paper provides a useful layer-wise analysis for LVLMs and a lightweight, token-efficient way to adapt visual feature fusion to task instructions. The study is comprehensive in its benchmark coverage, and the method is clearly described and reproducible in principle. However, the current evidence does not yet establish the central claim that the instruction signal is causally responsible for the gains: the key ablation removes the entire aggregator rather than just the instruction conditioning, and several reported numbers are internally inconsistent. These issues need to be resolved before the contribution can be fully assessed.

major comments (4)
  1. [Section 5.5, Table 6] The 'w/o aggregator' ablation replaces the entire instruction-guided vision aggregator with average pooling, simultaneously removing the sentence embedding and the weight allocator's parameters. This comparison cannot attribute the gain to the text instruction; the improvement could come purely from the allocator's added capacity or from conditioning on the visual group CLS features. Please add a control that keeps the aggregator architecture but removes the instruction information (e.g., a constant or random embedding) and a visual-only variant to isolate the role of the instruction.
  2. [Section 3.3, Table 3; Section 5.5, Table 6] The tables contain unexplained anomalous values. In Table 3, the 'Decrease-then-increase' row reports MME-p = 472.1, while all other configurations are around 1470; this is likely a typo that needs correction. In Table 6, the full method scores 48.0 on RealWorldQA, whereas the three ablated variants score 56.4, 56.0, and 55.8; the text does not mention this large drop, and it contradicts the narrative of consistent improvements. Please correct or explain these values.
  3. [Sections 3.1-3.3 and 5.2-5.3] The architecture and hyperparameters (number of groups, group boundaries, concatenation with the penultimate layer, entropy coefficient lambda=0.02) were selected using the same 18-benchmark evaluation set that substantially overlaps with the 10 benchmarks in the final comparison. This selection-on-evaluation protocol means the reported gains are inflated relative to a fair held-out evaluation. Please evaluate the method on a separate set of benchmarks or provide a clear protocol that avoids using the test benchmarks for model selection.
  4. [Tables 4-6] All results are reported as single runs without variance or significance information. Many differences are small (e.g., GQA 63.1 vs 63.8, MMB-en 66.9 vs 67.5), so the claim of being best on 8 out of 10 benchmarks is not statistically supported. Please report multiple runs or at least specify the number of seeds and provide error bars.
minor comments (5)
  1. [Keywords, Section 1] There are typos: 'Insturction' in the keywords, 'thier' and 'ecoder' in Section 1, and inconsistent spacing in 'L VLMs' throughout. Please proofread.
  2. [Eq. (8)] The entropy regularization term uses sum w_k log w_k, which is the negative entropy. The sign convention should be stated explicitly so that readers understand that minimizing this term with positive lambda encourages balanced weights.
  3. [Table 5] Many entries are marked with '-'. Please clarify whether these are not evaluated, not reported, or not applicable, and add footnote definitions.
  4. [Section 5.6, Table 7] The claim that the low-level group receives a 'notable' weight increase in Fine-Grained tasks is supported by only a small numerical difference (0.20 overall vs 0.17-0.18 in other categories). Consider toning down the language or reporting per-benchmark variability.
  5. [Conclusion] The conclusion states the method relies on clear and high-quality instructions, but no experiment tests sensitivity to instruction quality. Adding a small study with corrupted or ambiguous instructions would strengthen the discussion.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the claimed gains are empirical and the learned aggregator is not derived from the evaluation benchmarks by construction.

full rationale

The paper's derivation chain is empirically grounded rather than circular. The preliminary observations (Section 3) independently train single-layer, two-layer, and fixed-weight models and report their benchmark results; these findings motivate the architecture but do not define its outputs. The instruction-guided vision aggregator (Section 4) is a learned module whose weights are optimized with the LM loss and an entropy regularizer, not computed from the benchmark labels or from the static weight distributions tested in Table 3. The headline result that the method wins on 8 of 10 benchmarks (Section 5.3) comes from a separately trained model evaluated on those benchmarks, so it is not statistically forced by construction. The ablation in Table 6 replaces the entire aggregator with average pooling, which conflates removal of the instruction embedding with removal of the trainable allocator; however, that is an experimental-design and attribution limitation, not a self-definitional or construction-equivalence circularity. There is no load-bearing self-citation, no imported uniqueness theorem, and no renamed known result. The paper's own stated limitation regarding reliance on clear instructions is an input-quality caveat, not a circular step. Accordingly, no specific circular step meets the evidentiary bar required by the analysis rules.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No new physical or empirical entities are introduced; the aggregator is a learned module. The main burden is on hyperparameter choices and the assumption that instruction embeddings carry task-discriminative signal.

free parameters (4)
  • entropy regularization coefficient lambda = 0.02
    Hand-set in Eq. 8 to balance weight allocation; affects how task-specific the learned weights are.
  • number of visual groups K = 4 (layers 1-6, 7-12, 13-18, 19-24)
    Chosen by hand in Section 5.1; determines granularity of fusion.
  • weight allocator size = 4 transformer blocks, hidden 1024, 4 heads
    Architecture hyperparameters from Section 5.1, no ablations reported.
  • penultimate layer for concatenation = layer 23
    Selected because LLaVA-L23 had best overall score in Table 1, a decision made on the same benchmarks used for final evaluation.
assumptions (4)
  • domain assumption Different layers of CLIP-ViT capture complementary information (low-level detail to high-level semantics).
    Drawn from cited works [12,13] and assumed before the analysis.
  • ad hoc to paper Average pooling across layers within a group preserves representative information.
    Used in Eqs. 2 and 4; pooling is a simplification that could discard useful features.
  • domain assumption MPNet sentence embeddings capture task-relevant instruction semantics.
    The weight allocator relies on this embedding to assign weights; no ablation tests the effect of the instruction signal.
  • domain assumption Frozen vision encoder and sentence encoder, with only the aggregator, adapter, and LLM trained, is sufficient.
    Follows the LLaVA-1.5 training pipeline.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Instruction-Guided Fusion of Multi-Layer Visual Features in Large Vision-Language Models." pith.science (2026). https://pith.science/paper/7AWBXT4T

@misc{pith2026250108443,
  author       = {Pith},
  title        = {Pith review of: Instruction-Guided Fusion of Multi-Layer Visual Features in Large Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7AWBXT4T}},
  note         = {Machine review of arXiv:2501.08443}
}
read the original abstract

Large Vision-Language Models (LVLMs) have achieved remarkable success in a wide range of multimodal tasks by integrating pre-trained vision encoders and large language models. However, current LVLMs primarily rely on visual features extracted from the final layers of the vision encoder, overlooking the complementary information available in shallower layers. While recent approaches have explored the use of multilayer visual features in LVLMs, they tend to be task-agnostic and fail to examine the dependencies of hierarchical visual features on specific tasks. To address these gaps, we systematically investigate the contributions of visual features from different encoder layers using 18 benchmarks spanning 6 task categories. Our findings reveal that multilayer features provide complementary strengths with varying task dependencies, and uniform fusion leads to suboptimal performance. Building on these insights, we propose the instruction-guided vision aggregator, a module that dynamically integrates multi-layer visual features based on textual instructions, without increasing the number of visual tokens. Extensive evaluations demonstrate the superior performance of our method. Additionally, an in-depth analysis of the aggregator's behavior highlights the dominance of mid-to-high-level features in semantic-rich tasks and the critical role of low-level features in fine-grained perception.

Figures

Figures reproduced from arXiv: 2501.08443 by the authors.

Figure 1
Figure 1. Attention distribution of LVLMs across image patches when generating the answer token. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Performance comparison of our method against the baseline model (LLaVA-v1.5-7B [1]) [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Performance comparison of LVLMs trained using different single-layer visual features [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: (a) Overview of the proposed framework. (b) Detailed architecture of the weight allocator [PITH_FULL_IMAGE:figures/full_fig_p016_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

44 extracted references · 17 canonical work pages

  1. [1]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 26296–26306, 2024

    2024

  2. [2]

    Instructblip: Towards general-purpose vision-language models with instruction tuning

    Wenliang Dai, Junnan Li, DONGXU LI, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems , volume 36, pages ...

    work page 2023

  3. [3]

    Minigpt-v2: large language model as a unified interface for vision- language multi-task learning

    Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model as a unified interface for vision- language multi-task learning. arXiv preprint arXiv:2310.09478 , 2023

    arXiv

  4. [4]

    Llava-med: Training a large language-and-vision assistant for biomedicine in one day

    Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems , 36, 2024

    work page 2024

  5. [5]

    Talk2bev: Language-enhanced bird’s- 29 eye view maps for autonomous driving

    Tushar Choudhary, Vikrant Dewangan, Shivam Chandhok, Shubham Priyadar- shan, Anushka Jain, Arun K Singh, Siddharth Srivastava, Krishna Murthy Jatavallabhula, and K Madhava Krishna. Talk2bev: Language-enhanced bird’s- 29 eye view maps for autonomous driving. In 2024 IEEE International Conference on Robotics and Automation (ICRA) , pages 16345–16352. IEEE, 2024

    work page 2024

  6. [6]

    What can VLMs do for zero-shot embodied task planning? In ICML 2024 Workshop on LLMs and Cognition , 2024

    Xian Fu, Min Zhang, Jianye HAO, Peilong Han, Hao Zhang, Lei Shi, and Hongyao Tang. What can VLMs do for zero-shot embodied task planning? In ICML 2024 Workshop on LLMs and Cognition , 2024

    work page 2024

  7. [7]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020

    arXiv 2010

  8. [8]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748–8763. PMLR, 2021

    2021

Show all 44 references
  1. [9]

    Sig- moid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sig- moid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 11975–11986, 2023

    2023

  2. [10]

    A survey on hallucination in large vision-language models

    Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. A survey on hallucination in large vision-language models. arXiv preprint arXiv:2402.00253 , 2024

    2024 arXiv

  3. [11]

    Eyes wide shut? exploring the visual shortcomings of multimodal llms

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9568–9578, 2024. 30

    2024

  4. [12]

    Teaching matters: Investigating the role of supervision in vision transformers

    Matthew Walmer, Saksham Suri, Kamal Gupta, and Abhinav Shrivastava. Teaching matters: Investigating the role of supervision in vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7486–7496, 2023

    2023

  5. [13]

    What do vision trans- formers learn? a visual exploration

    Amin Ghiasi, Hamid Kazemi, Eitan Borgnia, Steven Reich, Manli Shu, Micah Goldblum, Andrew Gordon Wilson, and Tom Goldstein. What do vision trans- formers learn? a visual exploration. arXiv preprint arXiv:2212.06727 , 2022

    2022 arXiv

  6. [14]

    Dense connector for mllms

    Huanjin Yao, Wenhao Wu, Taojiannan Yang, YuXin Song, Mengxi Zhang, Haocheng Feng, Yifan Sun, Zhiheng Li, Wanli Ouyang, and Jingdong Wang. Dense connector for mllms. arXiv preprint arXiv:2405.13800 , 2024

    2024 arXiv

  7. [15]

    Mmfuser: Multimodal multi-layer feature fuser for fine- grained vision-language understanding

    Yue Cao, Yangzhou Liu, Zhe Chen, Guangchen Shi, Wenhai Wang, Danhuai Zhao, and Tong Lu. Mmfuser: Multimodal multi-layer feature fuser for fine- grained vision-language understanding. arXiv preprint arXiv:2410.11829 , 2024

    2024 arXiv

  8. [16]

    Mome: Mixture of multimodal experts for generalist multimodal large language models

    Leyang Shen, Gongwei Chen, Rui Shao, Weili Guan, and Liqiang Nie. Mome: Mixture of multimodal experts for generalist multimodal large language models. arXiv preprint arXiv:2407.12709 , 2024

    2024 arXiv

  9. [17]

    Mova: Adapting mixture of vision experts to multimodal context

    Zhuofan Zong, Bingqi Ma, Dazhong Shen, Guanglu Song, Hao Shao, Dongzhi Jiang, Hongsheng Li, and Yu Liu. Mova: Adapting mixture of vision experts to multimodal context. arXiv preprint arXiv:2404.13046 , 2024

    2024 arXiv

  10. [18]

    Cambrian-1: A fully open, vision-centric exploration of multimodal llms

    Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, 31 et al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. arXiv preprint arXiv:2406.16860 , 2024

    2024 arXiv

  11. [19]

    V?: Guided visual search as a core mechanism in multimodal llms

    Penghao Wu and Saining Xie. V?: Guided visual search as a core mechanism in multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13084–13094, 2024

    2024

  12. [20]

    Divide, conquer and combine: A training-free framework for high-resolution image perception in multimodal large language models

    Wenbin Wang, Liang Ding, Minyan Zeng, Xiabin Zhou, Li Shen, Yong Luo, and Dacheng Tao. Divide, conquer and combine: A training-free framework for high-resolution image perception in multimodal large language models. arXiv preprint arXiv:2408.15556, 2024

    2024 arXiv

  13. [21]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355, 2023

    2023 arXiv

  14. [22]

    Mme: A comprehensive evaluation benchmark for multimodal large language models

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394, 2023

    2023 arXiv

  15. [23]

    Gqa: A new dataset for real- world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real- world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 6700–6709, 2019

    2019

  16. [24]

    Seed-bench: Benchmarking multimodal large language models

    Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, 32 and Ying Shan. Seed-bench: Benchmarking multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13299–13308, 2024

    2024

  17. [25]

    Mmbench: Is your multi-modal model an all-around player? In European Conference on Computer Vision, pages 216–233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European Conference on Computer Vision, pages 216–233. Springer, 2025

    2025

  18. [26]

    Math- vista: Evaluating mathematical reasoning of foundation models in visual con- texts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Ha- jishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Math- vista: Evaluating mathematical reasoning of foundation models in visual con- texts. arXiv preprint arXiv:2310.02255 , 2023

    arXiv

  19. [27]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on C...

    2024

  20. [28]

    A diagram is worth a dozen images

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Ha- jishirzi, and Ali Farhadi. A diagram is worth a dozen images. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14 , pages 235–25...

    2016

  21. [29]

    Learn to explain: Mul- timodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun 33 Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Mul- timodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems , 35:2...

    2022

  22. [30]

    Chartqa: A benchmark for question answering about charts with visual and logical reasoning

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.10244 , 2022

    2022 arXiv

  23. [31]

    Docvqa: A dataset for vqa on document images

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision , pages 2200–2209, 2021

    2021

  24. [32]

    On the hidden mystery of ocr in large multimodal models

    Yuliang Liu, Zhang Li, Biao Yang, Chunyuan Li, Xucheng Yin, Cheng-lin Liu, Lianwen Jin, and Xiang Bai. On the hidden mystery of ocr in large multimodal models. arXiv preprint arXiv:2305.07895 , 2023

    2023 arXiv

  25. [33]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019

    2019

  26. [34]

    Grok-1.5 vision preview

    xAI. Grok-1.5 vision preview. Technical report, 2024

    2024

  27. [35]

    Gaussian error linear units (gelus)

    Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016

    2016 arXiv

  28. [36]

    Mpnet: Masked 34 and permuted pre-training for language understanding

    Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. Mpnet: Masked 34 and permuted pre-training for language understanding. Advances in neural in- formation processing systems, 33:16857–16867, 2020

    2020

  29. [37]

    Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023) , ...

    2023

  30. [38]

    Decoupled weight decay regularization

    I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  31. [39]

    Vizwiz grand challenge: Answering visual questions from blind people

    Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grau- man, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3608...

    2018

  32. [40]

    Mm-vet: Evaluating large multimodal models for integrated capabilities

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490 , 2023

    2023 arXiv

  33. [41]

    mplug-owl2: Revolutionizing multi-modal large lan- guage model with modality collaboration

    Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. mplug-owl2: Revolutionizing multi-modal large lan- guage model with modality collaboration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition ...

    2024

  34. [42]

    Llama-adapter v2: Parameter- efficient visual instruction model

    Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei 35 Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al. Llama-adapter v2: Parameter- efficient visual instruction model. arXiv preprint arXiv:2304.15010 , 2023

    2023 arXiv

  35. [43]

    Obelics: An open web-scale fil- tered dataset of interleaved image-text documents

    Hugo Lauren¸ con, Lucile Saulnier, Leo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh. Obelics: An open web-scale fil- tered dataset of interleaved image-text documents. In A...

    2023

  36. [44]

    Qwen-vl: A versatile vision- language model for understanding, localization, text reading, and beyond

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision- language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 1(2):3, 2023. 36

    2023 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.