Pith. sign in

REVIEW 3 major objections 7 minor 1 cited by

Global Semantic-Guided Sub-image Feature Weight Allocation in High-Resolution Large Vision-Language Models

T0 review · 3 major / 7 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read GSWA, a learnable module that weights image tiles by their global semantic relevance, improves high-resolution vision-language understanding.

desk verdict A plausible lightweight weighting module for sub-image LVLMs with a genuinely clean motivating experiment, but the train/test overlap on AI2D and DocVQA must be disclosed before the headline gains can be trusted. read the letter →

arxiv 2501.14276 v1 pith:SGFQDOLZ submitted 2025-01-24 cs.CV cs.AI

Yuxuan Liang , Xu Li , Xiaolei Chen , Haotian Chen , Yi Zheng , Chenghang Lai , Bin Li , Xiangyang Xue This is my paper · ORCID
classification cs.CVcs.AI
keywords high-resolutionvision-languagemodelssub-imagepartitioningglobalsemanticguidanceper-tileweightallocationhumanvisualattentionGSWAmoduleSleighVLmultimodalunderstanding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Sub-image partitioning lets high-resolution vision-language models (LVLMs) work within fixed-resolution vision encoders, but standard pipelines treat every tile equally. This paper claims that tiles whose content is more semantically aligned with the whole image carry more of the information needed for understanding, and that a lightweight module can learn to up-weight those tiles. The proposed Global Semantic-guided Weight Allocator (GSWA) reads only the summary tokens of the tiles and the global thumbnail, computes per-tile weights from the global token's attention, and rescales tile features before they reach the language model. Inserted into the InternVL2-2B framework, the resulting SleighVL model improves on the baseline across general, real-world, text-rich, and hallucination benchmarks while staying competitive with models several times larger. If the claim holds, adaptive tile weighting is a cheap, general upgrade for any sub-image-partitioning LVLM.

What carries the argument

The GSWA module is the central object: a lightweight stack of four transformer blocks followed by a multi-head self-attention layer that operates only on the <cls> tokens extracted from the $N$ sub-images and the global thumbnail after pixel shuffle. Its output is the weight vector $w$ in Equation 7, where the global token's average attention over the sub-image tokens is normalized to sum to one; Equation 8 then scales each sub-image's patch embeddings by its weight. The design converts the paper's semantic-relevance hypothesis into a differentiable, task-tuned weighting mechanism, and because it touches only <cls> tokens, it adds a small parameter count and can be inserted into any sub-image-partitioning LVLM.

What would settle it

Take high-resolution images where the task-relevant content is deliberately placed in a low-salience region, such as a small sign at the image edge, run SleighVL, and compare its per-tile weights with an independent ranking of tiles by how much the answer depends on them; if the lowest-weighted tiles turn out to be the ones whose removal hurts accuracy most, the central claim is wrong.

Watch

Extended reading notes

Core claim

The paper's central claim is that in high-resolution LVLMs that partition an image into tiles, the tiles whose <cls> token is most similar to the <cls> token of the whole-image thumbnail are the ones that preserve the model's visual understanding, and that this signal can be turned into a learnable per-tile weight. It supports the first part with a removal experiment: deleting the three most globally similar tiles from InternVL2-2B drops MME from 1876 to 1734 and OCRBench from 784 to 440, whereas deleting the three least similar tiles leaves scores almost unchanged. It then builds GSWA, which passes the <cls> tokens of all tiles plus the thumbnail through transformer blocks and uses the thumbnail token's average attention over tile tokens as a normalized weight vector; every tile's pixel-shuffled patch embeddings are multiplied by its weight before projection into the LLM. The resulting model, SleighVL, reports higher scores than InternVL2-2B on every benchmark where the baseline is listed in the comparison tables, including MME (1913), OCRBench (803), and TextVQA (75.9), and it matches or beats several 4B-13B models on real-world and text-rich tasks.

Load-bearing premise

Everything rests on the assumption that the thumbnail's summary token, after self-attention, reliably represents the image's global semantics and that its attention to each tile's summary token tracks how much task-relevant information that tile actually contains.

Editorial extensions

If this is right

  • Any sub-image-partitioning LVLM can adopt GSWA as a plug-in, since it consumes only the <cls> tokens that such encoders already produce and leaves the vision encoder untouched.
  • The ablation results imply that the learnable attention interaction matters: replacing it with fixed cosine similarity costs about 2.39% average performance, and removing the module costs about 3.49% relative to the fine-tuned baseline.
  • The same weight signal could later be reused as a pruning criterion: tiles with low learned weight are candidates for token dropping, which would offset the extra visual tokens that partitioning introduces.
  • The reported gains are largest on OCRBench, where the score moves from 784 to 803, suggesting that document-heavy and text-rich tasks are where the semantic-relevance weighting pays off most.
  • Because GSWA sits between the vision encoder and the projector, it can be trained or fine-tuned without changing the rest of the model's weights.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A text-conditioned version of the weight allocator, which the paper lists as future work, would likely outperform the image-only GSWA on question answering, since tile relevance is often question-dependent; this is an extension the authors did not test.
  • The weight vector could double as a saliency map for interpretability: inspecting which tiles receive high weight in SleighVL would give a post-hoc explanation of where the model looks.
  • If the semantic-relevance hypothesis transfers across encoders, the same weighting idea could apply to other tiled domains such as medical or satellite imagery, where informative regions are sparse.
  • Because the paper's removal experiment shows that low-relevance tiles can be dropped with little loss, the weighting signal may also serve as a token-compression schedule, reducing inference cost without retraining.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper proposes Global Semantic-guided Weight Allocator (GSWA), a module inserted between the vision encoder and projector of a sub-image-partitioning LVLM. GSWA takes the <cls> tokens of sub-images and a global thumbnail, passes them through transformer blocks, and uses the global token's attention over sub-image tokens to compute per-sub-image weights, which are then multiplied onto the patch features. The module is integrated into InternVL2-2B to form SleighVL. The authors first provide observational evidence (Table I) that removing sub-images with high global semantic similarity degrades performance more than removing low-similarity sub-images. They then evaluate SleighVL on 21 benchmarks, report gains over the original InternVL2-2B and competitive results with larger models, and include an ablation (Table VI) comparing GSWA to cross-attention, cosine-similarity weighting, and a fine-tuned baseline without GSWA.

Significance. If the reported results hold, GSWA provides a simple, modular way to make sub-image partitioning in high-resolution LVLMs adaptive, with a modest parameter overhead. The preliminary observation in Table I is a useful empirical study of how sub-image importance varies. The ablation in Table VI consistently shows a benefit of the learned attention-based weighting over uniform fine-tuning on eight benchmarks, which is the central evidence for the proposed mechanism. The paper also benefits from evaluation across a wide range of benchmarks. However, the lack of explicit train/test split disclosure for AI2D and DocVQA, which appear in both the training mixture and the evaluation tables, undermines the specific claims of competitiveness on those benchmarks, and the absence of variance estimates makes the small reported improvements difficult to interpret. These issues are addressable but require additional experiments and disclosure.

major comments (3)
  1. [V-A, Tables IV and V] The training data list in Section V-A includes AI2D and DocVQA, and the same datasets are used in the evaluation tables (Table IV: DocVQA Test; Table V: AI2D). The paper never states whether the training and evaluation splits are disjoint. Because the primary comparisons in these tables are against the original InternVL2-2B checkpoint, which was not fine-tuned on this mixture, the reported gains on DocVQA (87.1 vs 86.9) and AI2D (74.6 vs 74.1) could be partly due to exposure to training data rather than to GSWA. DocVQA in particular has known cases where test images share document sources with training images. The authors must either verify and explicitly state that the official test splits are disjoint from the training samples (including image-level and question-level deduplication) or, failing that, report the fine-tuned baseline (w/o GSWA) on these benchmarks. Without this, the claims of competitiveness on DocVQA and AI2D are not verifiable.
  2. [V-F, Table VI] The ablation study is the cleanest support for the central claim because it compares GSWA against a baseline fine-tuned on the same data. However, the ablation table omits DocVQA and AI2D, which are exactly the benchmarks where training/evaluation overlap is a concern. Please add these two benchmarks to Table VI so that the effect of GSWA on the contested sets can be assessed independently of fine-tuning. This would directly address whether the module itself helps on these datasets or only the added training data.
  3. [V-F, Table VI] No error bars, confidence intervals, or multiple-seed results are reported anywhere in the paper. Several improvements in Table VI are small in absolute terms (e.g., RealWorldQA 57.8 vs 56.6; HallusionBench 39.8 vs 37.5), and the benchmark scores are averages over test sets. Without an estimate of variance, it is not possible to determine whether these differences are statistically reliable. The authors should either provide multiple runs with standard deviations or use a significance test, at least for the central ablation table.
minor comments (7)
  1. [V-B] The text says SleighVL 'surpasses several larger models, such as Monkey (72.4/65.1), Cambrian-8B (75.9/64.9), and IDEFICS2-8B (76.8/65.1)' on MMB_EN and MMB_CN_V11. On MMB_EN, SleighVL (73.4) is actually lower than both Cambrian-8B (75.9) and IDEFICS2-8B (76.8); the statement is only correct for MMB_CN_V11. Please rephrase to avoid overstating the result.
  2. [V-E] In the discussion of Table V, IDEFICS2-8B is quoted as (39.1/86.2/88.7) when comparing HallusionBench/POPE/AI2D, but the table lists IDEFICS2-8B's AI2D score as 72.3 and its SQA-I score as 88.7. The 88.7 appears to be the SQA-I value, not AI2D. Please correct the text to match the table.
  3. [V-F] The text describing the cosine-similarity ablation says 'the MME score decreasing from 1907 to 1868', but Table VI reports MME=1902 for the cross-attention variant and MME=1868 for the cosine variant. The value 1907 does not appear in the table. Please reconcile the text with the table.
  4. [V-F] The term 'Average Decline' in Table VI is not defined. Please specify how it is computed (e.g., average relative decrease across the listed benchmarks).
  5. [IV-B] Equation (7) is hard to parse: the notation {·}_global is introduced but not rigorously defined, and it is unclear whether the softmax is taken over all sub-image tokens for each head before averaging. Please clarify the exact tensor dimensions and the normalization steps.
  6. [V-A and VI] The paper describes SleighVL as 'lightweight' but does not report the parameter increase or inference-time overhead introduced by the GSWA module. Section VI mentions additional computational cost qualitatively; please provide concrete numbers (e.g., parameter count, FLOPs or latency) to justify the lightweight claim.
  7. [II-A and V-E] There are a few typos: 'employe' in Section V-E and 'dierctly' in Section V-F. Please copy-edit the manuscript.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: GSWA weights are computed by a learned attention mechanism and validated on downstream benchmarks, not reduced to the paper's own inputs.

full rationale

The paper's derivation chain is empirical rather than definitional. The motivating hypothesis—that sub-images more semantically aligned with the full image carry more useful visual information—is supported by the removal experiments in Table I, where the intervention is defined by cosine similarity between <cls> tokens. That operation is distinct from the GSWA module's learned multi-head attention weights in Eq. (7), so the method does not define 'information density' as the same quantity it then predicts. The GSWA weights are not fitted to benchmark scores nor renamed benchmark outcomes; they are computed from learnable attention projections and applied to sub-image features in Eq. (8), with downstream task scores serving as independent evidence. No load-bearing self-citation appears: references [2] and [11] are external works (InternVL/InternViT), and no uniqueness theorem or ansatz is smuggled in from the authors' own prior work. The acknowledged limitation that GSWA ignores language input is an honest scope restriction, not a circular step. The overlap between the training-data list (AI2D, DocVQA) and the evaluation list is a potential data-contamination concern, but the paper does not assert that training samples equal test samples, and no equation or fitted value reduces to a benchmark score; under the hard rules, that is not a demonstrated circularity. Overall, the central claim is self-contained and externally validated, so the circularity score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on a few design choices and assumptions: the <cls> token proxy, the correlation between similarity and information density, the learnability of the attention weights, and the validity of evaluating on benchmarks that may share data with the training set. No new physical or conceptual entities are introduced; GSWA is a neural module.

free parameters (4)
  • Number of GSWA transformer blocks = 4
    Chosen by hand for the model design; no sensitivity analysis provided.
  • GSWA hidden dimension = 1024
    Chosen by hand; fixed during experiments.
  • Number of attention heads in GSWA = 4
    Chosen by hand; no ablation across this choice.
  • Maximum sub-images for dynamic cropping = 8
    Set to 8 following InternVL2 configuration; affects how many sub-images are weighted.
assumptions (4)
  • domain assumption The <cls> token of each sub-image summarizes its global semantic content.
    Used in Sections III and IV-B to compute semantic similarity and attention weights; not proven for this encoder.
  • domain assumption Semantic similarity between a sub-image and the full image correlates with information density.
    Stated as a hypothesis in Section III and supported only by two qualitative cases and a removal experiment.
  • domain assumption The global <cls> token's attention weights after self-attention can be trained to reflect information density.
    Core to GSWA in Equation 7; relies on the model learning useful weights during fine-tuning.
  • ad hoc to paper Evaluating on AI2D and DocVQA after including them in the training mixture is valid.
    The paper does not state whether train and test splits are disjoint, so this assumption is unverified and potentially circular.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Global Semantic-Guided Sub-image Feature Weight Allocation in High-Resolution Large Vision-Language Models." pith.science (2026). https://pith.science/paper/SGFQDOLZ

@misc{pith2026250114276,
  author       = {Pith},
  title        = {Pith review of: Global Semantic-Guided Sub-image Feature Weight Allocation in High-Resolution Large Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SGFQDOLZ}},
  note         = {Machine review of arXiv:2501.14276}
}
read the original abstract

As the demand for high-resolution image processing in Large Vision-Language Models (LVLMs) grows, sub-image partitioning has become a popular approach for mitigating visual information loss associated with fixed-resolution processing. However, existing partitioning methods uniformly process sub-images, resulting in suboptimal image understanding. In this work, we reveal that the sub-images with higher semantic relevance to the entire image encapsulate richer visual information for preserving the model's visual understanding ability. Therefore, we propose the Global Semantic-guided Weight Allocator (GSWA) module, which dynamically allocates weights to sub-images based on their relative information density, emulating human visual attention mechanisms. This approach enables the model to focus on more informative regions, overcoming the limitations of uniform treatment. We integrate GSWA into the InternVL2-2B framework to create SleighVL, a lightweight yet high-performing model. Extensive experiments demonstrate that SleighVL outperforms models with comparable parameters and remains competitive with larger models. Our work provides a promising direction for more efficient and contextually aware high-resolution image processing in LVLMs, advancing multimodal system development.

Figures

Figures reproduced from arXiv: 2501.14276 by the authors.

Figure 1
Figure 1. An example of human visual saliency analysis highlighting the areas of interest. (a) illustrates an image of a horse-drawn sled carrying four individuals in a snowy landscape, and (b) presents the human visual saliency map generated by SalGAN [6], which highlights the regions of the image with high saliency and information density that attract human visual attention. proaches include training encoders to process lar… view at source ↗
Figure 2
Figure 2. Radar chart comparing our model with existing popular LVLMs of similar parameter scales across ten benchmarks. in high-resolution images. Our work enables the capabilities of LVLMs to process high-resolution images. By leveraging the fine-grained details inherent in such images, the model is able to achieve improved visual perception and a more comprehensive understanding of visual content. B. High-Resolution Proces… view at source ↗
Figure 3
Figure 3. Two case studies to examine the semantic similarity distribution between each sub-image and the global image. (a) illustrates the degree of similarity between each sub-image and the global image semantics ”the player and football” in a football scene, and (b) illustrates the degree of similarity between each sub-image and the global image semantics ”the basketball players and court” in a basketball scene. TABLE I PR… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: The workflow of the proposed method. (a) describes the overall framework of SleighVL. (b) shows the design details of the global semantic-guided weight allocator. formance drops when sub-images with the highest global semantic similarity are removed. For instance, on t…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. HRVVS: A High-resolution Video Vasculature Segmentation Network via Hierarchical Autoregressive Residual Priors

    cs.CV 2025-07 conditional novelty 6.0 of 10

    A new dataset and network for segmenting hepatic vasculature in high-resolution hepatectomy videos, reporting the best scores on the new benchmark.

Reference graph

Works this paper leans on

48 extracted references · 34 canonical work pages · cited by 1 Pith paper

  1. [1]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Proceedings of the International Conference on Neural Information Processing Systems (NIPS) , pages 34892–34916, 2023

    work page 2023

  2. [2]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR) , pages 24185–24198, 2024

    work page 2024

  3. [3]

    Sharegpt4v: Improving large multi- modal models with better captions

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi- modal models with better captions. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 370–387, 2025

    work page 2025

  4. [4]

    Monkey: Image resolution and text label are important things for large multi-modal models

    Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Monkey: Image resolution and text label are important things for large multi-modal models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR) , pages 26763–26773, 2024

    work page 2024

  5. [5]

    InternLM-XComposer2-4KHD: A pioneering large vision-language model handling resolutions from 336 pixels to 4k HD

    Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Songyang Zhang, Haodong Duan, Wenwei Zhang, Yining Li, Hang Yan, Yang Gao, Zhe Chen, xinyue zhang, Wei Li, Li Jingwen, Wenhai Wang, Kai Chen, Conghui He, Xingcheng ZHANG, Jifeng Dai, Yu Qiao, Dahua Lin, and Jiaqi Wang. InternLM-XComposer2-4KHD: A pioneering large vision-language mode...

    work page 2024

  6. [6]

    Salgan: Visual saliency prediction with adversarial networks

    Junting Pan, Elisa Sayrol, Xavier Giro-i Nieto, Cristian Canton Ferrer, Jordi Torres, Kevin McGuinness, and Noel E OConnor. Salgan: Visual saliency prediction with adversarial networks. In CVPR scene understanding workshop (SUNw) , 2017

    work page 2017

  7. [7]

    PaLI: A jointly-scaled multilingual language-image model

    Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish V Thapliyal, James Bradbury, Weicheng Kuo, Mojtaba Seyed- hosseini, Chao Jia, Burcu Karagol Ayan, ...

    work page 2023

  8. [8]

    Deepseek- vl: towards real-world vision-language understanding

    Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, et al. Deepseek- vl: towards real-world vision-language understanding. arXiv preprint arXiv:2403.05525, 2024

    arXiv 2024

Show all 48 references
  1. [9]

    Cambrian-1: A fully open, vision-centric exploration of multimodal LLMs

    Shengbang Tong, Ellis L Brown II, Penghao Wu, Sanghyun Woo, ADITHY A JAIRAM IYER, Sai Charitha Akula, Shusheng Yang, Jihan Yang, Manoj Middepogu, Ziteng Wang, Xichen Pan, Rob Fergus, Yann LeCun, and Saining Xie. Cambrian-1: A fully open, vision-centric exploration of multimoda...

    2024

  2. [10]

    Llava-next: Improved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, 2024

    2024

  3. [11]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. Science China Information Sciences , 67(12):220101, 2024

    2024

  4. [12]

    Instruct- blip: Towards general-purpose vision-language models with instruction tuning

    Wenliang Dai, Junnan Li, DONGXU LI, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instruct- blip: Towards general-purpose vision-language models with instruction tuning. In Proceedings of the International Conference on Neural Information...

    2023

  5. [13]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Proceedings of the International conference on m...

    2021

  6. [14]

    Dual modality prompt tuning for vision-language pre-trained model

    Yinghui Xing, Qirui Wu, De Cheng, Shizhou Zhang, Guoqiang Liang, Peng Wang, and Yanning Zhang. Dual modality prompt tuning for vision-language pre-trained model. IEEE Transactions on Multimedia , 26:2056–2068, 2024

    2024

  7. [15]

    Sgva-clip: Semantic-guided visual adapting of vision-language mod- els for few-shot image classification

    Fang Peng, Xiaoshan Yang, Linhui Xiao, Yaowei Wang, and Changsheng Xu. Sgva-clip: Semantic-guided visual adapting of vision-language mod- els for few-shot image classification. IEEE Transactions on Multimedia , 26:3469–3480, 2024

    2024

  8. [16]

    Gpt4ego: Unleashing the potential of pre-trained models for zero- shot egocentric action recognition

    Guangzhao Dai, Xiangbo Shu, Wenhao Wu, Rui Yan, and Jiachao Zhang. Gpt4ego: Unleashing the potential of pre-trained models for zero- shot egocentric action recognition. IEEE Transactions on Multimedia , 27:401–413, 2025

    2025

  9. [17]

    mplug-docowl2: High-resolution compressing for ocr-free multi-page document understanding

    Anwen Hu, Haiyang Xu, Liang Zhang, Jiabo Ye, Ming Yan, Ji Zhang, Qin Jin, Fei Huang, and Jingren Zhou. mplug-docowl2: High-resolution compressing for ocr-free multi-page document understanding. arXiv preprint arXiv:2409.03420, 2024

    2024 arXiv

  10. [18]

    Internlm2 technical report

    Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. Internlm2 technical report. arXiv preprint arXiv:2403.17297 , 2024

    2024 arXiv

  11. [19]

    Mme: A comprehensive evaluation benchmark for multimodal large language models

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394 , 2023

    2023 arXiv

  12. [20]

    Ocrbench: on the hidden mystery of ocr in large multimodal models

    Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-Lin Liu, Lianwen Jin, and Xiang Bai. Ocrbench: on the hidden mystery of ocr in large multimodal models. Science China Information Sciences , 67(12):220102, 2024

    2024

  13. [21]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR) , pages 8317–8326, 2019

    2019

  14. [22]

    Mmbench: Is your multi-modal model an all-around player? In Proceed- ings of the European Conference on Computer Vision (ECCV) , pages 216–233, 2025

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In Proceed- ings of the European Conference on Computer Vision (ECCV) , pages 216–233, 2025

    2025

  15. [23]

    A diagram is worth a dozen images

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Han- naneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 235–251, 2016

    2016

  16. [24]

    Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models

    Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In Proce...

    2024

  17. [25]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR) , pages 26296–26306, 2024

    2024

  18. [26]

    Decoupled weight decay regular- ization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regular- ization. In Proceedings of the International Conference on Learning Representations (ICLR), 2019

    2019

  19. [27]

    Dvqa: Understanding data visualizations via question answering

    Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. Dvqa: Understanding data visualizations via question answering. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), pages 5648–5656, 2018

    2018

  20. [28]

    ChartQA: A benchmark for question answering about charts with visual and logical reasoning

    Ahmed Masry, Do Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In Proceedings of the annual meeting on the Association for Computational Linguistics (ACL) , pages 2263–2279, 2022

    2022

  21. [29]

    Docvqa: A dataset for vqa on document images

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR) , pages 2200–2209, 2021

    2021

  22. [30]

    An augmented benchmark dataset for geometric question answering through dual parallel text encoding

    Jie Cao and Jing Xiao. An augmented benchmark dataset for geometric question answering through dual parallel text encoding. In Proceedings of the International Conference on Computational Linguistics (COL- ING), pages 1511–1520, 2022

    2022

  23. [31]

    Ocr-free document understanding trans- former

    Geewook Kim, Teakgyu Hong, Moonbin Yim, JeongYeon Nam, Jiny- oung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. Ocr-free document understanding trans- former. In Proceedings of the European Conference on Computer Vision (ECCV), page 498–517, 2022

    2022

  24. [32]

    Are we on the right way for evaluating large vision-language models? In Proceedings of the International Conference on Neural Information Processing Systems (NIPS) , 2024

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and Feng Zhao. Are we on the right way for evaluating large vision-language models? In Proceedings of the International Conference on Neural Information Process...

    2024

  25. [33]

    Mm-vet: evaluating large multimodal models for integrated capabilities

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: evaluating large multimodal models for integrated capabilities. In Proceedings of the International conference on machine learning (ICML) , pages 57730– 57754, 2025

    2025

  26. [34]

    Seed-bench: Benchmarking multimodal large language models

    Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Benchmarking multimodal large language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR) , pages 13299–13308, 2024

    2024

  27. [35]

    Grok-1.5 vision preview

    xAI. Grok-1.5 vision preview. Technical report, 2024

    2024

  28. [36]

    Mme-realworld: Could your multimodal llm challenge high- resolution real-world scenarios that are difficult for humans? arXiv preprint arXiv:2408.13257, 2024

    Yi-Fan Zhang, Huanyu Zhang, Haochen Tian, Chaoyou Fu, Shuangqing Zhang, Junfei Wu, Feng Li, Kun Wang, Qingsong Wen, Zhang Zhang, et al. Mme-realworld: Could your multimodal llm challenge high- resolution real-world scenarios that are difficult for humans? arXiv preprint arXiv:...

    2024 arXiv

  29. [37]

    Divide, conquer and combine: A training-free framework for high-resolution image perception in multimodal large language models

    Wenbin Wang, Liang Ding, Minyan Zeng, Xiabin Zhou, Li Shen, Yong Luo, and Dacheng Tao. Divide, conquer and combine: A training-free framework for high-resolution image perception in multimodal large language models. arXiv preprint arXiv:2408.15556 , 2024

    2024 arXiv

  30. [38]

    Infographicvqa

    Minesh Mathew, Viraj Bagal, Rub `en Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographicvqa. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 1697–1706, 2022

    2022

  31. [39]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2023

    2023

  32. [40]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song- Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In Proceedings of the International Conference on Neural Inform...

    2022

  33. [41]

    What matters when building vision-language models? In Proceedings of the International Conference on Neural Information Processing Systems (NIPS), 2024

    Hugo Laurenc ¸on, Leo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models? In Proceedings of the International Conference on Neural Information Processing Systems (NIPS), 2024

    2024

  34. [42]

    Vila: On pre-training for visual language models

    Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR) , pages 26689–26699, 2024

    2024

  35. [43]

    Minicpm-v: A gpt-4v level mllm on your phone

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800 , 2024

    2024 arXiv

  36. [44]

    Internlm-xcomposer2: Mastering free-form text-image composition and comprehension in vision-language large model

    Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, et al. Internlm-xcomposer2: Mastering free-form text-image composition and comprehension in vision-language large model. arXiv preprint arXiv:2401.16420, 2024

    2024 arXiv

  37. [45]

    Paligemma: A versatile 3b vlm for transfer

    Lucas Beyer, Andreas Steiner, Andr ´e Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alab- dulmohsin, Michael Tschannen, Emanuele Bugliarello, et al. Paligemma: A versatile 3b vlm for transfer. arXiv preprint arXiv:2407.07726 , 2024

    2024 arXiv

  38. [46]

    Llava- onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava- onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326 , 2024

    2024 arXiv

  39. [47]

    Mini-internvl: a flexible-transfer pocket multi-modal model with 5% parameters and 90% performance

    Zhangwei Gao, Zhe Chen, Erfei Cui, Yiming Ren, Weiyun Wang, Jinguo Zhu, Hao Tian, Shenglong Ye, Junjun He, Xizhou Zhu, et al. Mini-internvl: a flexible-transfer pocket multi-modal model with 5% parameters and 90% performance. Visual Intelligence, 2(1):1–17, 2024

    2024

  40. [48]

    Ovis: Structural embedding alignment for multimodal large language model

    Shiyin Lu, Yang Li, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Han-Jia Ye. Ovis: Structural embedding alignment for multimodal large language model. arXiv preprint arXiv:2405.20797 , 2024

    2024 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.