Pith. sign in

REVIEW 3 major objections 5 minor 51 references

This paper argues that vision-language models can be substantially improved by replacing the single final-layer visual hookup with a dynamic, gated many-to-many bridge between vision-encoder layers and language-decoder layers.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-08-03 10:12 UTC pith:JEXJR53K

load-bearing objection A plausible fusion module with a genuinely new design, but the headline result leans on a LLaVA-1.5 baseline trained on different data, and the 'significant' claim is not supported by the controlled numbers. the 3 major comments →

arxiv 2601.10710 v2 pith:JEXJR53K submitted 2026-01-15 cs.CV

From One-to-One to Many-to-Many: Dynamic Cross-Layer Injection for Deep Vision-Language Fusion

Cheng Chen , Yuyu Guo , Pengpeng Zeng , Jingkuan Song , Peng Di , Hang Yu , Lianli Gao This is my paper
classification cs.CV
keywords vision-language modelscross-layer injectionhierarchical visual featuresadaptive gating fusionlow-rank adaptationmany-to-many fusionOCR and groundingfine-grained visual reasoning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Standard vision-language models feed the language model only the final-layer output of the vision encoder—a single, static point of contact. This paper argues that this one-to-one connection is a bottleneck, and proposes Cross-Layer Injection (CLI) to replace it with a many-to-many bridge: features are sampled from many layers of the vision transformer and injected, through a learned gate, into many layers of the language decoder. The key claim is that the gate—not the extra features themselves—is what unlocks the improvement, because it lets the decoder decide, at each decoding step, which level of visual detail is relevant. The authors validate the claim on two established open vision-language model families and eighteen benchmarks, reporting consistent gains especially on OCR, visual grounding, and open-ended visual chat.

Core claim

On the paper's own terms, the discovery is that a dynamic many-to-many connection between vision and language hierarchies outperforms both the conventional final-layer-only projection and static deep-fusion alternatives. The framework samples intermediate token maps from a set of vision-encoder layers and, at multiple decoder layers, merges them into the language model's hidden state under the control of a context-dependent gate. Because the gate weights are computed from the current hidden state and the incoming visual summary, the model can 'zoom in' on fine-grained details from early vision layers while simultaneously drawing on global semantics from late layers. The authors report that a

What carries the argument

The machinery is a pair of lightweight modules. Adaptive Multi-Projection (AMP) uses a low-rank adapter added to the shared pre-trained projector for each sampled vision layer, so features from different depths land in a common language-embedding space without training a separate full projector per layer. Adaptive Gating Fusion (AGF) is the load-bearing piece: at each injection point, two learnable query vectors attend separately to the incoming visual tokens and to the LLM's current hidden state; the two distilled summaries are concatenated, passed through a linear layer and a sigmoid, and produce a per-visual-token gate value W in [0,1]. The hidden state is updated only on visual positions

Load-bearing premise

The load-bearing premise is that the vision encoder's early layers really do store fine local detail and its late layers really do store global semantics, and that the language decoder's shallow layers are the syntax layers and its deep layers the reasoning layers, so that injecting 'every fourth visual layer into every fourth language layer' is the right wiring; if layers do not have these roles, the reported gains could come simply from giving the model more parameters and

What would settle it

Train the same gated framework twice on identical data: once with the standard natural-order layer mapping (every fourth visual layer to every fourth decoder layer) and once with the visual layers assigned to decoder slots in a fixed random permutation, so the model still gets the same number of parameters and the same gating machinery but no 'criss-cross' semantics. If the shuffled version matches the natural-order version on the 18 benchmarks, the hierarchy story is not load-bearing; if the natural order wins by the reported margins, the alignment assumption is supported.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X LinkedIn Reddit HN

If this is right

  • If the central claim is right, then the standard 'final layer to first layer' VLM design is leaving measurable performance on the table, and any VLM can be upgraded by adding gated multi-layer injection rather than scaling the model.
  • The ablation results imply that future fusion designs should put their effort into selective, context-aware gating; simply adding more layers of visual features without a gate can hurt performance, as the paper's brute-force additive comparison shows.
  • Because the framework is described as plug-and-play and was tested on two different base architectures, the same injection pattern could become a drop-in component for other vision-language models.
  • The reported 'criss-cross' gating pattern, if it holds, gives a concrete, testable picture of how multimodal reasoning uses visual memory: shallow language layers ground nouns with semantic prototypes while deep language layers re-examine fine visual evidence.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The biggest open question the paper leaves implicit is whether the hierarchy alignment story is the real cause; I would interpret the gating ablation as evidence that a learned selection mechanism is more important than the specific layer hierarchy, suggesting that many-to-many injection is best viewed as a flexible visual-memory mechanism rather than a fixed architectural principle.
  • The stride-four sampling schedule is a hand-chosen hyperparameter; one could imagine learning which vision layers to sample or even using continuous features, which might further improve the fine-grained tasks where the paper currently needs more training data.
  • The mechanism is not inherently visual: any hierarchical encoder-decoder pair (audio, video, or structured sensor data) could use the same gated cross-layer injection pattern, and the same gate logic could be reused.
  • The paper's data-scaling analysis suggests that gains on fine-grained perception tasks grow with more instruction data while reasoning gains appear early; a natural extension is to check whether the framework's advantage widens with larger training sets and longer training runs.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Cross-Layer Injection (CLI), a fusion module for vision-language models that replaces the conventional single ViT-final-layer-to-LLM-input projection with a many-to-many scheme: visual features are sampled from multiple ViT layers, aligned by a LoRA-based Adaptive Multi-Projection (AMP), and injected into multiple LLM decoder layers through Adaptive Gating Fusion (AGF). The method is integrated into LLaVA-OneVision (0.5B/7B) and LLaVA-1.5-7B, and evaluated on 18 benchmarks plus OCR/grounding tasks. The central claim is that dynamic, context-dependent injection of hierarchical visual features yields consistent and significant gains over the single-layer baseline and over static fusion alternatives.

Significance. The architecture is timely and addresses a real limitation of dominant VLM designs, namely the under-use of intermediate ViT representations. A parameter-efficient, controlled demonstration of this idea would be a useful contribution to the deep-fusion literature. The paper also reports comparisons against re-trained baselines and alternative fusion methods, which is the right experimental template. However, as written, the empirical support for the headline claims is substantially weaker than the abstract and conclusion state, and one of the two main architecture-validity experiments is not controlled for training data. The central idea remains plausible, but the current evidence does not establish 'consistent and significant' improvements across 18 benchmarks.

major comments (3)
  1. [§4.1 / App. A.2 / Tables 3 and 8] The LLaVA-1.5 comparison is not controlled. The baseline numbers in Table 3 (AI2D 66.3, ChartQA 38.9, DocVQA 32.2, LLaVA-W 62.9, GQA 57.6, etc.) match the official LLaVA-1.5-7B checkpoint, while the CLI model is fine-tuned on the ~1.4M-sample LLaVA-OneVision mixture described in App. B. The paper says it 'utilized the identical training dataset and procedure as in our LLaVA-OneVision experiments,' which is a different dataset from the official LLaVA-1.5 training data. Thus the +7.5 and +8.6 partial-sum gains in Table 3 could reflect the larger/different training set rather than CLI. The 'architecture-agnostic' claim requires a same-data re-trained LLaVA-1.5 baseline, as was done for the OneVision baselines.
  2. [§4.2 / Tables 1 and 2] The claim of 'significant performance improvements' is not supported by the controlled OneVision results. In the 7B row of Table 2, CLI is worse than the baseline on MathVerse (-1.7), MMBench (-0.3), MMVet (-0.3), SeedBench (-0.4), and ScienceQA (-0.8); in the 0.5B row of Table 1, InfoVQA (-0.4), LLaVA-W (-0.6), and GQA (0.0) are also flat or negative. Only aggregate sums (+9.7 and +3.2) are positive, and no variance estimates, confidence intervals, or significance tests are reported. With single runs and mixed per-benchmark deltas, the abstract's 'significant improvements' overstates the evidence. The paper should report repeated-seed means/variance or otherwise quantify uncertainty, and the claims should be restricted to the benchmarks with reliable positive effects.
  3. [§3.2, Eqs. (6)–(9), Fig. 1(b)] The mechanism as written does not implement the claimed per-layer selective gating. Eq. (8) produces a single scalar W in [0,1], and Eq. (9) applies the same scalar to the visual-token hidden states. There is no attention over the set of visual layers {V_hat_k} that would let the LLM 'query the entire visual hierarchy' and select the most relevant layer for the current decoding context. The claim that Fig. 1(b) shows 'deeper LLM decoder layers dynamically query features from the full spectrum of vision encoder layers' is therefore not grounded in the equations unless W is a vector with per-layer or per-token entries, and even then a scalar gain cannot select among layers. Please clarify whether the gate is per-layer, per-token, or global, and describe how the heatmap in Fig. 1(b) is computed from the model.
minor comments (5)
  1. [App. D / Table 8] Appendix D states that SLI causes 'severe losses on benchmarks like MMMU (−10.2%) and SEED-Bench (−10.3%),' but Table 8 shows SLI MMMU at +0.8 and SeedBench at -10.3. The MMMU number appears to be a typo or refers to a different setting.
  2. [Eq. (9)] The dimensions of the mask, W, and V_hat are not specified. It is unclear whether W is a scalar, a per-token vector, or a per-layer vector, which is directly relevant to understanding the gating behavior.
  3. [References / [40]] The SEED-Bench citation appears to point to a 2025 ACL paper on 'seed science,' which is not the standard SEED-Bench image benchmark. Please cite the correct SEED-Bench source.
  4. [Table 5] The table entry 'Resolution 384× {{1×1}, . . . ,{6×6}}' and 'Max #Tokens 729×10' is malformed and does not clearly communicate the resolution or token budget used.
  5. [Fig. 1(b) / Fig. 4] The qualitative heatmap and bar charts would benefit from a description of how the gating weights are aggregated and whether any variance across runs is shown; as presented, they are illustrative only.

Circularity Check

0 steps flagged

No circular reasoning identified; the benchmark claims rest on external held-out evaluations after training.

full rationale

The paper's derivation chain is: define an architecture (AMP + AGF, Eqs. 4-9), train it on a fixed instruction-following dataset, and measure zero-shot performance on held-out benchmarks using LMMs-Eval (Sec. 4.1 and Appendix C). No benchmark score is used as a training signal, and no reported 'prediction' is a renamed fit: the LoRA projectors (A_k, B_k) and gate weights (Eqs. 5-8) are learned from the training data, while the test results are external evaluations, not quantities reconstructed from those parameters. The claimed improvements are therefore empirically testable rather than true by construction. The nearest potential concerns are not circularity: the hierarchical-layer premise (early ViT = texture, late ViT = semantics, cited to [7-9]) is an imported assumption that is not directly tested, and the LLaVA-1.5 comparison in Table 3 / Appendix D may mix training data (the 1.4M OneVision mixture vs. the official LLaVA-1.5 checkpoint), which is an experimental-control risk. Neither reduces a prediction to an input of the paper's own equations, and there are no load-bearing self-citations. Figure 1(b)'s heatmap is a visualization of the learned gates, not a fitted target used to claim 'significant improvements.'

Axiom & Free-Parameter Ledger

7 free parameters · 5 axioms · 0 invented entities

The paper's central claim depends mainly on the hierarchical-role assumption for ViT/LLM layers, on the chosen hyperparameters (LoRA rank/alpha, injection stride), and on the evaluation framework. It introduces no new physical entities; 'many-to-many bridge' is a metaphor for the wiring between layers.

free parameters (7)
  • LoRA rank = 128
    Set for AMP in all experiments (Appendix A.1); not swept.
  • LoRA alpha = 128
    Set with rank; not swept.
  • Vision-layer sampling stride = every 4th of 28 (SigLIP) / 24 (CLIP) layers
    Controls which hierarchical features are available; chosen by authors, not ablated.
  • LLM injection stride = every 4th decoder layer
    Controls injection density; ablation Fig. 7 shows '8/7' underperforms '12/14', so the choice is consequential.
  • Attention probes qv, qh = learned
    Learned query vectors in AGF (Eq. 6-7), trained on the fine-tuning set.
  • Gate linear layer + sigmoid = learned
    Produces scalar gate W (Eq. 8).
  • Fine-tuning dataset mixture percentages = Table 7
    Sampling proportions for the ~1.4M subset; affects all model comparisons.
axioms (5)
  • domain assumption Early ViT layers capture local texture/structure; late layers capture semantic concepts.
    Motivates multi-layer sampling in Sec 1/3.2; cited to [9], not verified here.
  • domain assumption LLM decoder layers have depth-dependent roles (shallow: syntax; deep: reasoning).
    Used to justify injecting at many decoder layers; cited to [7,8], not verified.
  • domain assumption The public LLaVA-OneVision Stage-1.5 checkpoint is a clean, contamination-free base.
    Sec 4.1; used to avoid data contamination; unverifiable from the paper.
  • domain assumption LMMs-Eval scores across models are directly comparable.
    All comparisons and sums rely on this; different eval settings can shift numbers.
  • ad hoc to paper A scalar gate W in [0,1] and additive update (Eq. 9) preserve LLM behavior outside visual positions.
    AGF design; no stability analysis is provided.

reviewed 2026-08-03 · how reviews work

0 comments
Cite this review

Pith. "Pith review of From One-to-One to Many-to-Many: Dynamic Cross-Layer Injection for Deep Vision-Language Fusion." pith.science (2026). https://pith.science/paper/JEXJR53K

@misc{pith2026260110710,
  author       = {Pith},
  title        = {Pith review of: From One-to-One to Many-to-Many: Dynamic Cross-Layer Injection for Deep Vision-Language Fusion},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JEXJR53K}},
  note         = {Machine review of arXiv:2601.10710}
}
Share X LinkedIn Reddit HN
read the original abstract

Vision-Language Models (VLMs) create a severe visual feature bottleneck by using a crude, asymmetric connection that links only the output of the vision encoder to the input of the large language model (LLM). This static architecture fundamentally limits the ability of LLMs to achieve comprehensive alignment with hierarchical visual knowledge, compromising their capacity to accurately integrate local details with global semantics into coherent reasoning. To resolve this, we introduce Cross-Layer Injection (CLI), a novel and lightweight framework that forges a dynamic many-to-many bridge between the two modalities. CLI consists of two synergistic, parameter-efficient components: an Adaptive Multi-Projection (AMP) module that harmonizes features from diverse vision layers, and an Adaptive Gating Fusion (AGF) mechanism that empowers the LLM to selectively inject the most relevant visual information based on its real-time decoding context. We validate the effectiveness and versatility of CLI by integrating it into LLaVA-OneVision and LLaVA-1.5. Extensive experiments on 18 diverse benchmarks demonstrate significant performance improvements, establishing CLI as a scalable paradigm that unlocks deeper multimodal understanding by granting LLMs on-demand access to the full visual hierarchy.

Figures

Figures reproduced from arXiv: 2601.10710 by Cheng Chen, Hang Yu, Jingkuan Song, Lianli Gao, Peng Di, Pengpeng Zeng, Yuyu Guo.

Figure 1
Figure 1. Figure 1: (a) An illustrative failure case where our baseline model, [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: (a) The Conventional VLM Pipeline. A vision encoder [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: An overview of the proposed Cross-Layer Injection (CLI) Framework. The left panel illustrates the overall “many-to [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Performance gains from CLI on fine-grained visual reasoning. On both OCR and a comprehensive suite of visual ground [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Qualitative comparison of CLI on the LLaVA-OV-7B model across diverse benchmarks. Outputs from Baseline Projector are [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Ablation study on the effect of training data volume. Performance of our proposed framework (CLI) and the baseline model [PITH_FULL_IMAGE:figures/full_fig_p018_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Ablation study on the density of cross-layer injection points. The performance of different injection strategies—from high [PITH_FULL_IMAGE:figures/full_fig_p019_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Qualitative comparison of CLI and the baseline on diverse visual reasoning tasks. [PITH_FULL_IMAGE:figures/full_fig_p020_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

51 extracted references · 12 linked inside Pith

  1. [1]

    Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sa- hand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Kar ´en Simonyan

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Men- sch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L. Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sa- hand Sharifzadeh, Mikolaj B...

    2022

  2. [2]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Bar- bara Engelhardt, Sivan Sabato, and Jonathan Scarlett, edi- tors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023...

    2023

  3. [3]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023,

    2023

  4. [4]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, W A,USA, June 16-22, 2024, pages 26286–26296. IEEE, 2024. 1, 2, 5, 13, 16

    2024

  5. [5]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021,...

    2021

  6. [6]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pages 11941– 11952. IEEE, 2023. 1, 5, 13

    2023

  7. [7]

    Probing llms for joint encoding of linguistic categories

    Giulio Starace, Konstantinos Papakostas, Rochelle Choenni, Apostolos Panagiotopoulos, Matteo Rosati, Alina Leidinger, and Ekaterina Shutova. Probing llms for joint encoding of linguistic categories. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pages 7158–7179. Association for Computational Lingu...

    2023

  8. [8]

    Demystifying the roles of LLM layers in retrieval, knowledge, and reasoning

    Xinyuan Song, Keyu Wang, Pengxiang Li, Lu Yin, and Shi- 9 wei Liu. Demystifying the roles of LLM layers in retrieval, knowledge, and reasoning. CoRR, abs/2510.02091, 2025. 2

    arXiv 2025

  9. [9]

    Deep vit features as dense visual descriptors

    Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel. Deep vit features as dense visual descriptors. CoRR, abs/2112.05814, 2021. 1, 2

    Pith/arXiv arXiv 2021

  10. [10]

    Deep- stack: Deeply stacking visual tokens is surprisingly sim- ple and effective for lmms

    Lingchen Meng, Jianwei Yang, Rui Tian, Xiyang Dai, Zuxuan Wu, Jianfeng Gao, and Yu-Gang Jiang. Deep- stack: Deeply stacking visual tokens is surprisingly sim- ple and effective for lmms. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10...

    2024

  11. [11]

    EVLM: an efficient vision-language model for visual understanding

    Kaibing Chen, Dong Shen, Hanwen Zhong, Huasong Zhong, Kui Xia, Di Xu, Wei Yuan, Yifei Hu, Bin Wen, Tianke Zhang, Changyi Liu, Dewen Fan, Huihui Xiao, Jiahong Wu, Fan Yang, Size Li, and Di Zhang. EVLM: an efficient vision-language model for visual understanding. CoRR, abs/2407.14177, 2024. 2, 3

    Pith/arXiv arXiv 2024

  12. [12]

    Qwen3-vl.https : / / github

    Qwen Team. Qwen3-vl.https : / / github . com / QwenLM/Qwen3-VL, 2025. 2, 3, 5, 6, 7, 17

    2025

  13. [13]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language mod- els. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29,

    2022

  14. [14]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, Sand- hini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz...

    2020

  15. [15]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, Aur´elien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971, 2023. 2

    Pith/arXiv arXiv 2023

  16. [16]

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: Towards general- purpose vision-language models with instruction tuning. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023,...

    2023

  17. [17]

    Minigpt-v2: large language model as a unified interface for vision-language multi-task learning

    Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elho- seiny. Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. CoRR, abs/2310.09478, 2023. 3

    Pith/arXiv arXiv 2023

  18. [18]

    Minigpt-4: Enhancing vision- language understanding with advanced large language mod- els

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- language understanding with advanced large language mod- els. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11,

    2024

  19. [19]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. CoRR, abs/2308.12966, 2023. 3

    Pith/arXiv arXiv 2023

  20. [20]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhang- wei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, Lixin Gu, Xuehui Wang, Qingyun Li, Yimin Ren, Zixuan Chen, Jiapeng Luo, Jiahao Wang, Tan Jiang, Bo Wang, Conghui He, Botian Shi, Xingcheng Zhang, Han Lv, Yi Wang, Wenqi Shao, Pei Chu, Zhongying Tu, Tong He, Zhiyong Wu, Huipeng Deng, J...

    Pith/arXiv arXiv 2024

  21. [21]

    FUSION: fully in- tegration of vision-language representations for deep cross- modal understanding

    Zheng Liu, Mengjie Liu, Jingzhou Chen, Jingwei Xu, Bin Cui, Conghui He, and Wentao Zhang. FUSION: fully in- tegration of vision-language representations for deep cross- modal understanding. CoRR, abs/2504.09925, 2025. 3

    Pith/arXiv arXiv 2025

  22. [22]

    mplug-owl3: Towards long image-sequence understanding in multi-modal large language models

    Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-owl3: Towards long image-sequence understanding in multi-modal large language models. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net,

    2025

  23. [23]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Zi- wei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. Trans. Mach. Learn. Res., 2025, 2025. 5, 13, 14

    2025

  24. [24]

    Qwen2 technical re- port

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jian- wei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, M...

    Pith/arXiv arXiv 2024

  25. [25]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhang- hao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yong- hao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. 5, 13 10

    2023

  26. [26]

    Lmms- eval: Reality check on the evaluation of large multimodal models

    Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Lmms- eval: Reality check on the evaluation of large multimodal models. In Findings of the Association for Computational Linguistics: NAACL 2025, Albuquerque, New Mexico, USA, April 29 - May 4, 2025, pages 8...

    2025

  27. [27]

    A diagram is worth a dozen images

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Min Joon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV, volume 9908 of Lecture Notes in Computer Science, pages 235–251. Springer, 2016. 5, 15

    2016

  28. [28]

    Joty, and Enamul Hoque

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq R. Joty, and Enamul Hoque. Chartqa: A benchmark for question an- swering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 2263–

    2022

  29. [29]

    Minesh Mathew, Dimosthenis Karatzas, and C. V . Jawa- har. Docvqa: A dataset for VQA on document images. In IEEE Winter Conference on Applications of Computer Vision, W ACV2021, Waikoloa,HI, USA, January 3-8, 2021, pages 2199–2208. IEEE, 2021. 5, 15

    2021

  30. [30]

    Minesh Mathew, Viraj Bagal, Rub `en Tito, Dimosthenis Karatzas, Ernest Valveny, and C. V . Jawahar. Infograph- icvqa. In IEEE/CVF Winter Conference on Applications of Computer Vision, W ACV2022, Waikoloa,HI, USA, January 3-8, 2022, pages 2582–2591. IEEE, 2022. 5, 15

    2022

  31. [31]

    A survey on multimodal large language models

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. CoRR, abs/2306.13549, 2023. 5, 15

    Pith/arXiv arXiv 2023

  32. [32]

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player? In Computer Vision - ECCV 2024 - 18th European Conference, Milan, Italy, September 29-October 4, 2024, Proceedings, Part VI, volume 15064 of Lecture Notes...

    2024

  33. [33]

    Mm-vet: Evaluating large multimodal models for inte- grated capabilities

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for inte- grated capabilities. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27,

    2024

  34. [34]

    Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, Peng Gao, and Hongsheng Li. MATH- VERSE: does your multi-modal LLM truly see the diagrams in visual math problems? In Computer Vision - ECCV 2024 - 18th European Conference, Milan, Italy, September 29-October 4, 2024, Proceedings, Part...

    2024

  35. [35]

    Mathvista: Evaluating math rea- soning in visual contexts with gpt-4v, bard, and other large multimodal models

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating math rea- soning in visual contexts with gpt-4v, bard, and other large multimodal models. CoRR, abs/2310.02255, 2023. 5, 15

    Pith/arXiv arXiv 2023

  36. [36]

    MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI

    Xiang Yue, Yuansheng Ni, Tianyu Zheng, Kai Zhang, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weim- ing Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for e...

    2024

  37. [37]

    Hudson and Christopher D

    Drew A. Hudson and Christopher D. Manning. GQA: A new dataset for real-world visual reasoning and composi- tional question answering. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 6700–6709. Computer Vision Foundation / IEEE, 2019. 5, 15

    2019

  38. [38]

    OpenReview.net, 2024. 5, 15

    2024

  39. [39]

    Learn to explain: Multimodal rea- soning via thought chains for science question answer- ing

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai- Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal rea- soning via thought chains for science question answer- ing. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, ...

    2022

  40. [40]

    Seedbench: A multi- task benchmark for evaluating large language models in seed science

    Jie Ying, Zihong Chen, Zhefan Wang, Wanli Jiang, Chenyang Wang, Zhonghang Yuan, Haoyang Su, Huanjun Kong, Fan Yang, and Nanqing Dong. Seedbench: A multi- task benchmark for evaluating large language models in seed science. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (V olume1: Long Papers), ACL 2025, Vienna, ...

    2025

  41. [41]

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and Feng Zhao. Are we on the right way for eval- uating large vision-language models? In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Can...

    2024

  42. [42]

    Evaluating object hallucina- tion in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucina- tion in large vision-language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 11 6-10, 2023, pages 292–305. Association for Computational Linguistics, 2023. 5, 16

    2023

  43. [43]

    OK-VQA: A visual question answer- ing benchmark requiring external knowledge

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. OK-VQA: A visual question answer- ing benchmark requiring external knowledge. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 3195–3204. Computer Vision Foundation / IEEE,

    2019

  44. [44]

    Internlm-xcomposer-2.5: A versatile large vision language model supporting long-contextual input and output

    Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang, Linke Ouyang, Songyang Zhang, Wenwei Zhang, Yining Li, Yang Gao, Peng Sun, Xinyue Zhang, Wei Li, Jingwen Li, Wenhai Wang, Hang Yan, Conghui He, Xingcheng Zhang, Kai Chen, Jifeng Dai, Yu Qiao, Dahua Lin, and Jiaqi Wang. Internlm-xcomposer-2.5: A versati...

    Pith/arXiv arXiv 2024

  45. [45]

    Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks. CoRR, abs/2312.14238, 2023. 5, 6, 7

    Pith/arXiv arXiv 2023

  46. [46]

    Stage-1.5: High-Quality Knowledge Learning

    Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mo- hammad Shoeybi, and Song Han. VILA: on pre-training for visual language models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, W A,USA, June 16-22, 2024, pages 26679–26689. IEEE, 2024. 5, 6, 7 12 From One-to-One to Many-to-Many: Dynamic Cross-Layer Injection for Dee...

    2024

  47. [48]

    Grok-1.5 vision preview: Connecting the dig- ital and physical worlds with our first multimodal model

    X.AI Corp. Grok-1.5 vision preview: Connecting the dig- ital and physical worlds with our first multimodal model. https://x.ai/blog/grok-1.5v, 2024. 5, 16

    2024

  48. [2022]

    OpenReview.net, 2022. 2

    2022

  49. [2024]

    OpenReview.net, 2024. 3

    2024

  50. [2279]

    Association for Computational Linguistics, 2022. 5, 15

    2022

  51. [8763]

    1, 5, 13

    PMLR, 2021. 1, 5, 13

    2021

This paper was first reviewed by deepseek-v4-flash on August 3, 2026.